Causal Interpretation of Neural Network Computations with Contribution Decomposition
Authors: Joshua Brendan Melander
Zaki Alaoui
Shenghua Liu
Surya Ganguli
Stephen Baccus
Venue: ICLR 2026 Poster (2026)
Keywords: mechanistic interpretibility
neuroscience
xai
ai safety
tool-development
Primary area: interpretability and explainable AI
TL;DR: Using a novel famework borrowed from a century of neuroscience experiments, we gain insight into, and causal control over, intermediate layers of benchmark image-classification networks.

Abstract: Understanding how neural networks transform inputs into outputs is crucial for interpreting and manipulating their behavior. Most existing approaches analyze internal representations by identifying hidden-layer activation patterns correlated with human-interpretable concepts. Here we take a direct approach to examine how hidden neurons act to drive network outputs. We introduce CODEC ($\textbf{Co}$ntribution $\textbf{Dec}$omposition), a method that uses sparse autoencoders to decompose network behavior into sparse motifs of hidden-neuron contributions, revealing causal processes that cannot be determined by analyzing activations alone. Applying CODEC to benchmark image-classification networks, we find that contributions grow in sparsity and dimensionality across layers and, unexpectedly, that they progressively decorrelate positive and negative effects on network outputs. We further show that decomposing contributions into sparse modes enables greater control and interpretation of intermediate layers, supporting both causal manipulations of network output and human-interpretable visualizations of distinct image components that combine to drive that output. Finally, by analyzing state-of-the-art models of neural activity in the vertebrate retina, we demonstrate that CODEC uncovers combinatorial actions of model interneurons and identifies the sources of dynamic receptive fields. Overall, CODEC provides a rich and interpretable framework for understanding how nonlinear computations evolve across hierarchical layers, establishing contribution modes as an informative unit of analysis for mechanistic insights into artificial neural networks.

Published as a conference paper at ICLR 2026
CAUSAL INTERPRETATION OF NEURAL NETWORK COMPUTATIONS WITH CONTRIBUTION DECOMPOSITION
Joshua Brendan Melander Department of Neurobiology Stanford University
Zaki Alaoui
Shenghua Liu
Department Neurobiology Stanford University
Department of Physies Stanford University
Surya Ganguli
Stephen A. Baccus
Department of Applied Physics Stanford University
Department of Neurobiology
Stanford University
ABSTRACT
Understanding how neural networks transform inputs into outputs is crucial for interpreting and manipulating their behavior. Most existing approaches analyze internal representations by identifying hidden-layer activation patterns correlated with human- humaa-interpretable concepts. Here we take direct approach to exam- ine how hidden neurons act to drive network outputs. We introduce CODEC (Contribution Decomposition), method that uses sparse autoencoders to decom- pose network behavior into sparse motifs hidden-neuron contributions, reveal- ing causal processes that cannot be determined by analyzing activations alone. Applying CODEC to benchmark image-llassffication networks, we find that con- tributions grow in sparsity and dimensionality across layers and, unexpectedly, that they progressively decorrelate positive and negative effects network out- puts. We further show that decomposing contributions into sparse modes enables greater control and interpretation of intermediate layers, supporting both causal manipulations of network output and human- camma-interpretablle visualizations of dis- tinct image components that combine to drive that output. Finally, by analyzing state-of-the-art models of neural activity the vertebrate retina, we demonstrate that CODEC uncovers combinatorial actions of model interneurons and identifies the sources of dynamic receptive fields. Overall, CODEC provides a rich and interpretable framework for understanding how nonlinear computations evolve across hierarchical layers, establishing contribution modes as an informative unit of analysis for mechanistic insights into artificial neural networks.
A FRAMEWORK FOR UNDERSTANDING BIOLOGICAL AND ARTIFICIAL NEURAL NETWORKS
Biological and artificial neural networks both produce computations using cascading nonlinear op- erations that do not lend themselves to simple interpretations. Despite the widespread study and use of neural networks, there is no standardized framework to understand how given network output generated from its input through its intermediate stages. Understanding the mechanisms by which networks behave promises accelerate studies of the nervous system, lead to more effective design of efficient networks, reveal general principles of information processing in complex systems, and is also important for guiding the development of safe Al systems (Murdoch et al. (2019): Doshi- Velez and Kiml 2017); Lipton (2017): Rudin (2019)).
An essential aspect of both artificial and biological neural networks that their behavior is created by sets of internal components. The question we approach here is: How do the components of network act together to construct the output from the input?
These authors contributed equally to this work.

---

Published as a conference paper at ICLR 2026
A
X
X
Receptive Field
Projecttvve
Field
Figure 1: Understanding the contribution of an intermediate component to downstream com- putation. (A) Biological and (B) artificial neural circuits construct computations by combining sets of upstream components in input-dependent manner. The action of network component Z is a composition of its receptive field, or sensitivity to input X, and its projective field, or effect on output Y. Measuring both is required to explain how the intermediate component contributes to the overall behavior the system.
Sensory systems provide a natural framework for this question, with well-defined input- structure, clearly described tasks, and hierarchical architectures that inspired the development of convolutional neural networks (CNNs) (Fukushima (1980)).
neural network unit transforms inputs through its receptive field, and influences downstream pro- cessing through its projective field (Lehky and Sejnowski (1988)). The composition of these trans- formations defines a unit's contribution to the system's 's output, which is a direct causal measure Tanaka al. (2019); Maheswaranathan al. (2023)).
Most interpretability methods focus on unit activations, which reflect only the receptive field stage. Causal influence of unit must then be demonstrated via ablations or perturbations. Direct analyses of contributions, rather than activations, offer a more principled route, and contribution analysis in CNN models of the retina has led to specific hypotheses for how neural cell types perform specific functions (Tanaka et al. (2019): Mabeswaranathnn contributions have not examined how groups of units act together. This a significant limitation, as both biological (Olshausen and Field (2004)) and artificial networks compute through coordinated population activity. In CNNs, the output of Teature-selective populations can be combined classify images (Rawat and Wang (207)). In the visual system, the parallel parvocellular and magnocellular pathways roughly support chromatic and high acuity vision vs. achromatic vision and motion sensi- tivity (Solomon 2)21)), ) and recent neuroscientific theories suggest that groups of retinal ganglion cells operate in structured modes with specific computational roles (Prentice et al. (2016)). Under- standing network thus requires not just identifying features in the internal representation, but also explaining how combinations of those features are used to construct different outputs.
et al. (2023)). Yet existing approaches studying
EXISTING TOOLS FOR INTERPRETING ANNS
For artificial neural networks (ANNs) with millions of parameters, identifying causal structure is challenging because computation arises from nonlinear interactions, making the effects of single units strongly dependent on the input and activity of other units. In both biological and artifi- cial networks, much attention has been devoted studying latent representations at intermediate stages the network by analyzing network activity. Although these methods have found patterns in activations via clustering sparse autoencoders (Fel et al. (2023)), analyzing representations fundamentally does not address the causal question how internal elements act to influence the output.
More recently, methods like Integrated Gradients, SmoothGrad, and Grad-CAM have focused on re- vealing the influence of inputs model outputs via saliency maps (Selvaraju al. (2020); (milkov jet al. (2017); Sundararajan et al. (017)). However, these approaches offer limited insight into the actions of intermediate stages. Visual components such edges and textures may be needed dis-
2

---

Published as a conference paper at ICLR 2026
A
Model input
Convolutional layer
Output
Entropy
Sum of top-k logits (prelpusi soltmax)
Network-Wide Label-Free Contribution Targets
Specific logit
Scalet funciion outputs
B
Contriburtions Activatios
C
D
Contriburtions Activations
E
Excitatory Inhibitoryy
Channels
Channels
Layer
Layer
Figure 2: Hidden-neuron contributions in deep convolutional network. (A) Pipeline of com- puting contributions for an image processed through ResNet-50. Gradient- attribution methods extended compute the contributions of each hidden unit scalar targets of network output such as entropy of logits, sum of top-k logits, and individual class logits. (B) Spatial map of ac- tivations and contributions of single channel Layer 5. (C) Mean positive, negative and contribution for each channel. (D-E) Same as (B-C) for Layer 8.
criminate objects, but those same features other locations may be unrelated the target object yet still represented in hidden activations. Thus, compared to the analysis of activations, contribution analysis distinguishes between building blocks of computation that causally drive the output and those that are irrelevant. Accordingly, key open problem is characterize how networks causally integrate distributed latent features across channels and neurons to generate outputs, analogous to how biological networks produce functional effects through circuit interactions.
A CEURROSCIENCE-ISPPIEE ANN INTERPRETABILIT FRAMEWORK
We introduce method for analyzing how intermediate neurons drive network output. The first step extends attribution techniques such as Integrated Gradients to measure contributions internal layers. We compute each hidden neuron's contributions across all stimuli, capturing the combined effects of their receptive and projective fields. These contributions represent the actions neurons take construct the output.
Next, we decompose these contributions across inputs into a set of modes reflecting coordinated neuronal actions, an approach we call contribution decomposition (CODEC). Unlike analyses of activations, CODEC directly captures causal effects on outputs and can be applied to any trained feedforward model without access training data or labels. This cameral-purposse framework quan- tifies how groups of neurons drive behaviors. In the context of image recognition, the receptive fields of hidden neurons define visual features that are building blocks, and the contribution modes are the assembly instructions that show how those components are used to construct classes.
CODEC a general framework composed of different stages that can be adapted for artificial and biological neural networks:
1. Contribution target: The specific output neuron behavior (a sealar function of output neurons) whose computational basis we wish to understand.
2. Contribution algorithm: Quantification of how each hidden unit contributes to the target output for given input.
3. Decomposition of contributions:
The core computational modes representing common patterns of how neurons act together across inputs and outputs.
4. Visualization in input space: Contribution mapping to reveal how the input features that drove the key channels neurons within each mode used for output identification.

---

Published as a conference paper at ICLR 2026
A
Contributions pandas
Activations
B
0
0
0.8
0.6
Channels
Sparsity
500
500
0.4
88880
0.2
119
1000
1000
0.0
100
100
12 15
C
Images
Images
D
Layer
1.0 +
Correlation (r)
contributions
Layer 2
contributions
Layer 14
Negative
Negative
-10-
0.5
-2
8880
-20
0
10
20
8
12
E
Positive contributions
Positive
F
Layer
contributions
1.00
1.00
eeigenvalues
400
0.75
0.75
FOEV
0.50
0.50
Contributions
0.25
0.25
Activations
0.00
0.00
100
200
500
1000
0
10
Eigenvalues
Eigenvalues
Layer
Figure 3: Channel contributions through the network become more sparse, single-signed and high dimensional. (A) Example matrix of spatially-summant contributions and activations from one layer for all channels and images from four classes. (B) Hoyer sparsity index for contributions and activations across network layers. (C) Scatter plot of mean negative vs.
positive contributions of each channel to network output. (D) Correlation coefficient between positive and negative con- tributions of individual channels across network depth. (E) Fraction of explained variance (FOEV) across all class-averaged channel weightings for layers 2 and 14. (F) Number of components re- quired reach 95 percent FOEV.
These methods establish new interpretability tool that examines the intermediate neurons in network, identifies what input features they are sensitive to and their individual effects on network output, and reveals how their combined actions ultimately influence the network's behavior.
MEASURING CONTRIBUTIONS OF HIDDEN-LAYER NEURONS
The contribution of hidden neuron to network output is composition of its overall input and its overall output (Fig.) [T. and several methods have been used to calculate such effects. Integrated Gra- dients has most commonly been applied from network output to input (Sundararajan et al.) (017)), but have also been applied to analyze the effects of hidden neurons models of biological net- works (Tanaka et al. (2019); Maheswaranathan et al. (202))). An alternative attribution method, ActGrad (Selvaraju et al.] (2020), defined as the element-wise product of activations and gra- dients (ActGrad, = h1 Bh,
d2 where hy is the activation of hidden unit j. but is prone to noisy
fluctuations when evaluated locally.
Formulating a single contribution target was crucial to avoid intractable 3D decompositions, which would yield an array of shape Ninputt X Tieerrons Mogiis for each layer if contributions were computed separately for each logit. We found that computing contributions to the top logit for a given input was sufficient for accurate reconstruction. Additional scalar objectives (1) the sum of top-k output logits, reflecting the model's confidence its strongest predictions, and (2) the entropy of the output distribution, measuring prediction uncertainty.
offered in the code:
4

---

Published as a conference paper at ICLR 2026
A
Sigmoid
Threshold
B
Images
Mode
Hidden-layrr contributions
MLP
Recon-
Loading
reg
struction
Pandas
Channels
1
Loading
Pandas
Modes
Loadings
20000
40000
Imagenet validation samples
Figure 4: Sparse autoencoder decomposition of network contributions. (A) Schematic of con- tribution decomposition. Channel contributions are spatially summed and an autoencoder is trained reconstruct the matrix of contributions by images by creating modes with sparse loadings (the weighting for each mode for particular image). Loadings are passed through sigmoid then threshold, and regularized to encourage sparsity. (B) Loadings from the mode that maximally cor- related with the class "panda" for contributions (top, black) and activations (bottom, blue). Inset shows the loadings for 50 images panda and 100 images other classes.
CONTRIBUTIONS OF CONVOLUTIONAL LAYERS IN IMAGE CLASSIFICATION NETWORKS
Biological visual systems, such the retina, primary visual cortex, and inferotemporal cortex, share many architectural features with CNNs Yamins et al. (2014). Thus, we characterized the computa- tional structure of each layer in benchmark CNNs such ResNet-50 (He et al. (2016)) by analyzing neuronal contributions.
Starting with input image, compute the contributions of all hidden neurons the scalar tar- get (Fig.2M (A). We varied contribution algorithms (ActGrad, Integrated Gradients, SmoothGrad) and targets (e.g., top logit, top-k logit sum, entropy). and found that contributions were consistently spatially sparse compared to activations (Fig. B.D, Supplementary Fig. STA). A key property of Integrated Gradients is completeness: contributions sum the scalar output target. Thus, spatially summing contributions within a channel gives its net effect on the prediction, enabling assignment of single contribution value per channel, cell type (Fig. (Fig.2C). Applying this procedure across the entire 50,000 ImageNet (Deng et al. (2009)) validation images produced matrices of channel contri- butions for selected network blocks throughout ResNet-50, with each matrix having dimensionality d channels x 50,000 images (Fig. AA) For the remaining analyses, used Integrated Gradients to the top logit with 10 integration steps as our standard contribution method.
LAYERWISE EVOLUTION OF NEURAL CONTRIBUTIONS IN CNNS
To examine how the actions of the network evolve throughout its layers, we computed for the con- tributions of each channel the Hoyer sparsity index, normalized measure that computes the ratio of LI to L2 norms, and ranges from 0 (all channels equally active) to I (only one channel active). At all layers, contributions consistently showed high sparsity across channels than activations, indi- cating that only small subset of channels are functionally relevant for each classification decision (Fig. BB. Furthermore, contributions increased in sparsity throughout the network, aligning with the intuition that feature selectivity emerges with hierarchical depth.
Hidden unit activations are constrained by ReLU nonlinearities to be positive. However, contribu- tions can be positive or negative, indicating whether spatial position increases decreases the likelihood the target output. Thus, a highly active unit in the activation map can be used to inhibit the network's output, revealed by its contribution. These opposing influences, similar to exci- tatory and inhibitory interactions in biological vision (e.g., on-off receptive fields), are essential to neural computation but hidden in activations. We therefore examined the relationship between pos- itive and negative contributions across layers in ResNet-50.. Importantly, contribution sign reflects the net impact network output, not the polarity of synaptic weights.
5

---

Published as a conference paper at ICLR 2026
A
B
C
x10
D
Contribttins (modas)
15
of correlated modes
0.6
Contobbttios (chans) Adtivations (modes) Adtivationss (chans)
Density
Density
10
0.4
0.2
0.0
0.0
0.0
0.5
1.0
10
10
Mode top
Channel top
Layer
Layer
Figure 5: Emergence of meaningful contribution modes in intermediate layers. (A) Histograms of each mode's ' maximum correlation with binary class indicators for contributions (grey) and acti- vations (blue) hidden layer 13. (B) Same (A) for the correlation of individual channel contri- butions or activations with class indicators. (C) Number of modes with a correlation to class of greater than 0.2. (D) Mean of the maximal class-correlation function of layer.
To compare positive and negative contributions, we computed the contribution at each location (Fig. ( and then separated the contribution into positive and negative components prior to spa- tial summation. We found that in channels of earlier layers, the magnitude of positive and negative contributions were highly correlated within each channel. However, through the network, positive and negative contributions became progressively decorrelated (Fig. 1ag.3pa CD- One possible explana- tion for this shift is that lower-level features such as edges and textures encoded in earlier layers exhibit strong spatial correlations, necessitating that individual channels contribute both positively and negatively to remove these correlations, similar retinl computations, such as center-surround receptive fields (Pitkow and Meister )2012) and object motion sensitivity (Ölvezzky et al. (2003)).
common technique for identifying many examples of single class (Kim et al. (2018)) We thus examined the dimensionality of spatially summed and class-averaged activations and contributions using principal component analysis. Both quantities increased in dimensionality through the network, but contributions showed higher dimen- sionality than activations as measured by eigenvalues needed to reach 95% variance (Fig. BF-F). Throughout the network, contributions increased in sparsity and dimensionality, and decorrelated their positive and negative effects.
latent dimensions is to average activations over
4 DECOMPOSING CONTRIBUTIO INTO COMPUTATIONAL MODES
To uncover structure within these contributions, we decomposed them into set modes using an autoencoder consisting of encoder network fese Fdd + RR., and a non-negative dictionary
De E æd where is the number of modes. Typically k = N-d d for an overcomplete representation with expansion factor N. Each column m, E Rd of D defines one mode. Given contributions e E Rd, the encoder computes pre-activation loadings Zpre (c). Sparsity is enforced by hard thresholding: 2; Zpre, if Zpre.i 2 т. and zero otherwise. The reconstruction is ć Dz, and the autoencoder is trained minimize the loss L ||c é|i |e Dzll2 I12. with optional LI regularization applied to the loadings and modes. Non-negatyvity constraints are imposed on the dictionary D. Decomposition of contributions resulted in set of & modes of dimension d, and a set of loadings for each mode reflecting the weighting of those modes for each image that reconstructed the matrix of contributions with high accuracy, (average 122 = 0.85 for contributions and 0.84 for activations across all layers, Supplementary Fig. S2A). Training takes ~-37 minutes per layer for 200 epochs on an NVIDIA RTX 6000 GPU (see Supplementary Sec. [ec.Io..]for details). Our baseline model architecture used a threshold of 0.9 and a dictionary size 3 times the number of channels being learned. Models were trained for 300 epochs with a learning-rate of 5e-5 and batch-size of 128. LI regularization was added to the dictionary with scaling factor of 5e-5.
To measure how closely related these modes were related to specific network outputs, we corre- lated the loadings over the entire dataset (50,000 validation images) with a binary vector indicating whether a given image belonged to a given class (Fig. 4B). This resulted in a by 1000 corre- lation matrix, representing the correlation of each mode with each ImageNet category. We found that contribution modes were more correlated with Imagenet classes than modes recovered using activations, in particular at intermediate layers (Fig. 5 59). In addition, found that

---

Published as a conference paper at ICLR 2026
A
Black widow mode
B
Black wisow
C
Black
channels (from mode)
Contribution
0.5
Perturbation accurssy
Preserve
0.0
0.0
Ablate
1000
2000
1000
2000
Channel
Sorted channel
250
750
1000
Imagenet classes
D
Contributions
Activations
Contributions
Activations
AUCr AUC Anhogge
A penmeed
1.C
target offtarget
Larget
10
-1
Ablate
12 15
Acc
0.5
0.5
0.5
Preserve
0.0
0.0
0.0
0.0
100
100
100
100
% channels
% channela
% channels
% channels
Ablation
Preservation
4
Layer
Figure 6: Contribution-guideed network control. (A) Example mode most correlated with the
"black "idow" class, showing channel weightings. (B) Sorted channel weightings from (A) with a few top channels highlighted for selection. (C) Results for "black widow" classification: Top shows preservation analysis (accuracy when keeping only selecte channels), bottom shows ablation analysis (accuracy when removing selecte channels). (D) Normalized accuracy change for the tar- get class when ablating (left) or preserving (right) channels from the most correlated contribution mode (black), activation mode (blue), or random class (red, representing non-target/off-tan formance). (E) Performance score quantifying perturbation effectiveness as the area between target and off-target curves from (D) across all blocks, normalized to the off- off-target area.
per-
contribution modes, despite not having access to class labels during optimization, were lated with classes than were individual channels. This indicates the success CODEC revealing patterns of combined channel contributions that had relevance to specific network outputs (Fig, PP).
Results were robust to a range of variation to SAE hyperparameters. SAE reconstruction perfor- mance was affected little by L1 regularization, random seed, and whether modes were constrained to be positive. SAE R² was affected only when threshold was too high (0.9 vs 0.5), dictionary size was equal to smaller than the number of channels, than the number of classes (Supplementary Figs. [S3 and (S4). These results indicate that SAE per- formance robust provided these hyperparameters are kept sufficiently away from their boundary regimes.
MLPP size was not considerably greater
5 CONTROLLING NETWORK BEHAVIOR USING CONTRIBUTION MODES
Contribution modes represent the coordinated causal effects on network output. To examine how these effects could be used to control ImageNet classification, perturbed ResNet-50 by target- ing channels identified through CODEC analysis. We identified the mode most correlated with each class, then measured classification accuracy under two conditions: ablation (removing the top-weighted channels) and preservation (retaining only those channels). We quantified these ef- fects across all 1000 ImageNet classes by calculating the ratio in accuracy for each class between perturbed and unperturbed networks. For the "black widow" class, ablating 2% of salient channels identified from the top 2 most correlated modes greatly reduced target-class accuracy while leaving off-target classification performance largely maffected.((Fgg.. (Fig. [ (bottom)). Additionally, preserva- tion analysis yielded networks that could accurately classify only the targeted class (Fig 6C (top)). We randomly sampled the ImageNet validation dataset and compared perturbation performance for given mode and target class, and random off-target class, while varying the percentage of channels perturbed. Targeting channels by contribution modes more reliably identified necessary and suf- ficient channels for classification compared to activation-base analyses, requiring fewer channels to completely ablate target-class computation (Fig. 6D) A sharp increase in ablation efficacy was

---

Published as a conference paper at ICLR 2026
A
Contribution map
G
(
((mm,)
Contrilation of mode
((m)
(m)
1 @ (m)
mm)
c(
x, ( (m2)
B
Marimba
arrow
Model input
Mode #1
Hands highrighed
Shiny wood
Highigh
Figure 7: Visualizing stimuli acting through contribution modes. (A) Contribution mapping re- veals how parallel pathways extract different information to drive network output. The contribution map is an intermediate stage for computing contributions prior to input summation defined by ob- serving that the inner product of the image and sensitivity 1, the vector of ones, and an elementwise product that is the contribution map. Symbols are shown as their transpose to match the left right schematic. (B) Visualization of 4 example stimuli from different classes through two modes highly correlated with violin at layer of Resnet-50. The infor- mation conveyed by modes are interpretable features that can be identified across class and stimuli (shiny wood, hands, etc.). To preserve original colors, maps shown averaged over color channels and as a weighted mask on the original image, though they contain color information not shown.
can be expanded into a product of
observed between blocks and 7, potentially suggesting shift in how semantic information is rep- resented at this depth (Fig. (B). Additionally, demonstrated that approach generalizes beyond the 1000 ImageNet classes, successfully ablating / preserving specific taxonomic categories, indi- cating that CODEC can identify non-labled computational pathways even within broader semantic groupings (Supplementary Fig. STO).
VISUALIZING INPUTS THAT CAUSE HIDDEN CONTRIBUTIONS
Attribution methods such as Integrated Gradients compute gradients from outputs to inputs to iden- tify regions most influential to a decision. We extend this idea to hidden channels: what features of the input does channel or mode use to drive the output for given image? Using CODEC- selected salient channels, we isolated the gradient pathway from outputs to inputs that passes only through channels of interest, decomposing traditional input-output saliency maps into interpretable, channel- and mode-specific contributions. For each channel of mode m and each spatial location p. the input sensitivity is (c.P)) nel Jy. positions yields A A Ecem Ep !Cp))
Jyhhe Jhepp.z. where hep is the activation position p in chan- Shaa Aggregating over all selected channels in a mode and Dz.
ay and Jhappn, Theip
Whereas A! (!m)) captures spatial map of output sensi-
8

---

Published as a conference paper at ICLR 2026
tivity to pixel i, the contribution map viewed in imput space reflects the cumulative effect of input pixels on the output, approximated as the sensitivity weighted by the input: C.
A (m)) di-
The spatial sum E. ((m) recovers the contribution of the mode, ((m)) Contribution mapping highlights regions that most strongly drive contributions within the mode's most relevant channels.
Fig. 7 shows example visualizations. Additional methods are described in Supp. Sec. T0.7 among which the method described here is InputGrad. The contribution map shows pixels used by hidden channels within modes to drive network output for different classes Masking the original image with contribution maps shows that a given mode computes the same lower-level semantic features across images from different classes (Fig. 7 We expect that further analysis of the structure of these modes in input space will reveal meaningful visual components of objects used to construct visual classes.
INTERPRETING BIOLOGICAL NEURAL NETWORK MODELS WITH CODEC
We then used contribution decomposition interpret the structure of computation in the early visual system. Previous results (Maheswaranathan et al. (2023) ) have shown that three layer CNN models capture the responses of retinal ganglion cells to natural scenes, and are interpretable in that hidden units are highly correlated with recordings from retinal interneurons not used to fit the model. We therefore applied CODEC to these CNN models to interpret how groups of model interneurons contributed to retinal output.
The CNN model consisted of 3 layers, with 8 channels (cell types) in the first two layers, yielding a response of 4-17 recorded ganglion cells (Fig. (8)). In order to choose single contribution target, rather than choosing the top-1 logit as we did in the case of the image classification network, we chose the surprisal, or self-information reflects how unexpected a response is. Estimated from the covariance of the output, I(x) varies with the Mahalanobis distance of the population response from the mean, (x 11) Σ- 14), plus a constant term, where Σ is the covariance matrix and l4 is the mean response. With this target, positive contributions are those that create a more unexpected response.
I(x)
log, P(z), an information theory measure that
CODEC identified modes in the first two layers that combined to drive cells at different times, as measured by computing the correlation between cell firing rates and mode loadings (Fig. (8B-C) We then clustered cell types by the average pattern of active modes that drove the cell. Clustering the mode pattern layers 1 and 2 yielded very similar results, indicating the robustness of identifying pathways that drove different cell types (Fig. BC). We further analyzed at different times the instan- taneous receptive field (IRF) of ganglion cells, which is the gradient of a cell's response with respect to the stimulus (Fig. BBD,, revealing the visual feature driving the cell for a particular stimulus. We found that single modes could contribute similar IRF patterns across different ganglion cells. In- terestingly, when multiple modes simultancously drove cell, the resulting IRF dynamically varied according to the combination of active modes, with patterns ranging from familiar center-surround structures to oriented of ganglion cells can be driven by visual features that are the intersection of their receptive fields (Schnitzer and Meister (2003)), and that decomposition of ganglion cell activity can reveal modes with error-correcting properties that represent localized features distinct from those of single neu- rons (Prentice al. (2016)) As the units of this model are highly correlated with actual interneuron recordings aMaheswaranathan et al.] (2023)), contribution modes we identify here serve to auto- matically generate experimentally testable hypotheses that the presynaptic neurons with the visual features identified by the model drive ganglion cell activity patterns with error correcting properties.
textured responses (Fig- (D). Previous studies have indicated that groups
8 CODEC ON VISION TRANSFORMERS
We additionally applied CODEC to three layer types- features, MLPs, and attention layers- across the depth of a standard Vision Transformer, ViT-B (Dosovitskiy et al. (2020)) (see Supp. Sec. 10.5). Drawing an analogy to CNNs, we took a straightforward approach by treating tokens as the spatial dimension and hidden dimensions as channels. As in CNNs, we found that contributions sparser than activations. Subsequent decomposition revealed distinct contribution modes for these three layer types across all layers. These modes could be leveraged for targeted perturbation experiments more effectively than those based activations, revealing causal information not cap-

---

Published as a conference paper at ICLR 2026
A
C
Layer
Layer
Stimulus
layer
layer
0.5 Clusters
Recorded
firing-rate
Cells
Contriuttinn
target
IRF
IRF
Fring rate
(ha)
Mode
Mode combinations
Chan.
Layer
Neuron
Cham.
01A4
Loading
Mode (comi))
2.5 &4SRE
Time (sec)
Figure 8: Contribution modes generate dynamic receptive fields in the retina. (A) CNN trained to predict retinal ganglion cell responses to natural stimuli, from Ding et al. (2023). CODEC is performed for hidden units to single target outputs including surprisal (self-information) of the pop- ulation from single cell responses.
(B) Model firing rate predictions for example cell (top) aligned with the matrix of contributions, and with loadings for two example modes after SAE de- composition (bottom). Also shown instantaneous receptive fields (IRFs) of the cell computed at two time points. (C) Clustering of cells using their contribution modes. Matrix shows the corre- lation of firing of each cell with the loading of each mode. Three clusters (number selected using silhouette value) computed using k-means remain consistent across layers. (D) IRFs during sparse mode combinations. Left column: IRFs when one specific mode was active alone for four different cells at different times. Right: IRFs for example cells at times when 5 or fewer modes were active and cells were firing more than Hz. Each row from a different cell.
tured by activations alone. Although overall ablation performance was inferior that observed in CNNs. we note that ViTs utilize fundamentally different computational strategies due to their lack of an explicit spatial equivariant bias. Consequently, further exploration is needed determine the optimal spatial reduction strategy for ViTs, beyond our initial approach of treating tokens space.
CONCLUSION
Contribution decomposition identifies how hidden units construct specific outputs, revealing both the input components that causally drive model behavior and how the effects of those features are summed across outputs. Our approach thus achieves both deeper understanding of the computa- tion than results from examining representations alone, along with more effective manipulation of those networks. Our analysis reveals insights into the structure of neural computation, in particular at intermediate layers of networks that have been difficult to analyze and interpret. In biological networks such as the retina, CODEC allows an analysis of how dynamic sensitivity to visual input arises from the coordinated actions of model interneurons. In artificial networks, the emergence of sparse, interpretable motifs suggests that network output can understood in terms of a rel- atively small set of input-specific computations. Future work might leverage these computations as building blocks for more efficient architectures or transfer learning approaches. In the nervous system, the ability to relate computational building blocks to a downstream computation, combined with experimental measurements and manipulations at different levels could reveal how informa- tion is recombined across diverging and converging neural pathways. Such unified approach to neural computation could bridge the gap between computers and biology, potentially enabling more powerful Al systems and deeper insights into biological intelligence.
10

---

Published as a conference paper at ICLR 2026
LIMITATIONS AND ETHICAL CONCERNS
Whereas we focused on image classification, our approach lays the groundwork for extensions to other architectures and applications. Some analyses on ResNet-50 were limited to specific blocks rather than the entire network, and although CODEC is architecturally agnostic and could be applied to interpret more complex models such L.MMs, we have this point only analyzed scaling with LLMs (see supplemental material).
REPRODUCIBILITY STATEMENT
The key aspects of the CODEC framework implemented repository, containing scripts to reproduce each figure of this manuscript, available at https//ggttbbb.oommbmauuualCCCC
publicly available
2026 Here, we also provide the sparse autoencoder decomposition algorithms and visualization tools for mapping contributions back to input space, as additionally documented in the supplemental material, with mathematical derivations provided. The retinal neural network models analyzed are publicly available on GitHub as referenced in the ci- tations. All experimental configuratee configurations, including autoencoder architectures, sparsity constraints, and statistical analysis procedures, are documented to enable direct replication of our findings across both artificial and biological network interpretations.
ACKNOWLEDGEMENTS
This work was supported by grants from the National Eye Institute (NET): RO1EY022933, RO1EY025087 and P30EY026877 (SAB). S.G. thanks the Simons Foundation Collaboration on the Physics of Learning and Neural Computation and a Schmidt Sciences Polymath Award for support. Z.A. was supported by the Stanford Medicine Post-Bast Baccalaureate Experience In Research program.
REFERENCES
Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., (2023). Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. (2009). ImageNet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248 255.
Ding, X., Lee, D., Melander, B., Sivulka, G., Ganguli, S., and Baccus, S. A. (2023). Information geometry of the retinal representation manifold. In Advances in Neural Information Processing Systems (NeurIPS) 36, pages
Doshi- Velez, F. and Kim, (2017). Towards Rigorous Science of Interpretable Machine Learn ing. arXiv: 702.08608 [stat].
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations.
Fel, T., Boutin, V., Moayeri, M., Cadène, R., Bethune, L., andéol, L., Chalvidal, M., and Serre, T. (2023). A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation. arXiv:2306.07304 [cs].
Fukushima, K. (1980). Neocognitron: A self gelf-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics. 36(4):193-202.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770- 778.
11

---

Published as a conference paper at ICLR 2026
Kim, B., Wattenberg, M., Gilmer, J.. Cai, C., Wexler, J., Viegas, F., and Sayres, R. (2018). In- terpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). arXiv:1711.11233 [stat].
Lehky, S. R. and Sejnowski, J. (1988). Network model of shape-from-shading neural function arises from both receptive and projective fields. Nature, 333(6172):45524454
Lipton, Z. C. (2017). The Mythos of Model Interpretabilityy arX arXiv:1606.03490 [cs].
Lucas, M., Lerma, M.. Furst, J.. and Raicu, D. (2022). RSI-Grad-CAM: Visual Explanations from Deep Networks via Ahemann--tiillipe Integrated Gradient-Based Localization. In Bebis, G., Li, B., Yao, A., Liu, Y., Duan, Y., Lau, M., Khadka, R., Crisan, A., and Chang, R., editors, Advances in Visual Computing, pages 262-274, Cham. Springer International Publishing.
Maheswaranathan, N., McIntosh, L. T., Tanaka, H., Grant, S., Kastner, D. B., Melander, J. B., Nayebi, A., Brezovec, L. E., Wang. J. H., Ganguli, S., and Baccus, S. A. (2023). Interpreting the retinal neural code for natural scenes: From computations neurons. Neuron, 111(17):2742- 2755.e4.
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., and Yu, B. (2019). Interpretable ma- chine learning: definitions, methods, and applications. Proceedings of the National Acudemy of Sciences, (6(44))22071-220888
arXiv:1901.0459) [stat].
Olshausen, B. and Field, D. J. (2004). Sparse coding of sensory inputs. Current Opinion in Neurobiology, 14(4):481-487.
Pitkow, X. and Meister, M. (2012). Decorrelation and efficient coding by retinal ganglion cells. Nature Neuroscience, 1284)6628-635.
Prentice, J. S., Marre, O., Ioffe, M. L., Loback, A. R., Tkačik, G., and Berry, M. J. (2016). Error- robust modes of the retinal population code. PLoS computational biology, (2(11)=e0051145
Rawat, W. and Wang, Z. (2017). Deep convolutional neural networks for image classification: A comprehensive review. Neural computation, 299))2332-2249.
Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206 215.. Publisher: Nature Publishing Group.
Schnitzer, M. J. and Meister, M. (2003). Multineuronal firing patterns in the signal from eye to brain. Neuron, 37(3):499-511.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2020). Grad- CAM: Visual Explanations from Deep Networks via Gradient- based Localization. International Journal of Computer Vision, 128(2):336-35) arXiv:1610.0239 [es].
Shrikumar, A., Greenside, P., and Kundaje, A. (2017). Learning important features through prop- agating activation differences. Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, pages 1453,113,, Sydney, NSW, Australia. JMLR.org.
Smilkov, D., Thorat, N., Kim, B., Viégas. F., and Wattenberg, M. (2017). SmoothGrad: removing noise by adding noise, arXiv:1706.03822 [cs].
Solomon, S. G. (2021). Retinal ganglion cells and the magnocellular, parvocellular, and koniocel- lular subcortical visual pathways from the eye to the brain. In Handbook of clinical neurology. volume 178, pages 31-50. Elsevier.
Srinivas, S. and Fleuret, F. (2019). Full-Gradient Representation for Neural Network Visualization.
Sundararajan, M., Taly, A., and Yan, Q. (2017). Axiomatic Attribution for Deep Networks. arXiv:1703.0136 [es].
12

---

Published as a conference paper at ICLR 2026
Tanaka, H., Nayebi, A., Maheswaranathan, N., McIntosh, L., Baccus, S., and Ganguli, S. (2019). From deep learning to mechanistic understanding in neuroscience: The structure of retinal pre- diction. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Yamins, D. L. K., Hong, H., Cadieu, C. F,, Solomon, E. A., Seibert, D., and DiCarlo, J. J. (2014). Performannce-pptmmize hierarchical models predict neural responses higher visual cortex. Pro- ceedings of the National Academy of Sciences, 111(23)86619-862 Publisher: Proceedings of the National Academy of Sciences.
Ölveczky, B. P., Baccus, S. A., and Meister, M. (2003). Segregation of object and background motion in the retina. Nature, 423(6938)
13

---

Published as a conference paper at ICLR 2026
10
SUPPLEMENTAL MATERIAL
10.1 SPATIAL AGGREGATION AND E/I SEPARATION
Once the contributions have been computed using the chosen contribution method and target, we obtain a map of the intermediate layer of the same size as the layer's activations. For convolu- tional layers, we perform spatial summation over the height and width dimensions to derive single contribution value per channel:
H W ΣΣΣ IG
ChanC
Famm w=1
where "ChanC" means "channel contribution". We further reduce these contributions into excitatory (positive) and inhibitory (negative) components:
ChanC = max max(Charm (ChanC, 0): ChanC = min Miin(Chane (ChanC, 0).
(2)
This E/I decomposition enhances interpretabilityy by separating units that promote versus suppress specific outputs, revealing antagonistic computational mechanisms within the network. For analyses of modes presented here, we focus exclusively on the positive contributions, which can be interpreted as the hidden unit contributions that positively drive the attribution target. However, future work could extend this approach to also incorporate negative contributions, depending on the nature of the output target.
10.2 CORRELATION ANALYSIS WITH SEMANTIC CONCEPTS
To interpret the discovered modes, we analyzed their correlation with known semantic concepts from the ImageNet hierarchy. We constructed binary masks M E {0, 1]"xe representing e different semantic concepts, where M.j. if sample belongs to a higher order concept/ ImageNet Class j and 0 otherwise.
We then computed the Pearson correlation coefficient between each mode loading and each concept mask:
Corr(L, M2) Cov(Ls, M2)
(3)
OL,
where L; is the i-th column of the loading matrix L (corresponding to mode i). M2 is the j-th column of the mask matrix M (corresponding to concept j), Cov the covariance, and a is the standard deviation.
This analysis enabled us to assign interpretable meanings to the learned modes and understand how semantic information is distributed across the network.
10.3 ROBUSTNESS TO SAE HYPERPARAMETERS
To examine the stability and sensitivity of sparse autoencoders (SAE) modes trained on contributions function of hyperparameters, we trained SAEs ResNet-50 intermediate layer contributions on example layers 3, 7, 13 and 15 while varying six hyperparameters: LI sparsity penalty strength, threshold, dictionary size (N number of channels), random seed, MLP encoder size, and non- negativity constraint (Fig. S3) For each hyperparameter variation, we computed the correlation between the learned modes and baseline SAE configuration (used in the main text) by aggregating modes for each ImageNet class-summing all modes with correlation >> 0.2 to that class, or using the single top-correlated mode if none exceeded this threshold. Varying the L.1 sparsity penalty (0, le-04, le-02) showed minimal impact on learned representations. The threshold for mode selection (0.5, 0.7, 0.9) produced moderate variation, particularly middle layers when the threshold was not restrictive enough. The dictionary size multiplier N (x1, x3, x5 the layer channel dimension) indi- cated that smaller dictionaries (N=1) produce less stable representations in later layers, indicating the imitial dictionary size should be set greater than number of channels in the layee. Random seed variation demonstrated high reproducibility with ceariieentical correlations to baseline SAE across all layers. Varying the MIP hidden dimension (128, 512, 2048, 4096) indicated minimum size
14

---

Published as a conference paper at ICLR 2026
was needed, smaller hidden dimensions created a bottleneck for layers with higher input dimen- sionality, degrading reconstruction performance. Non-negative constraint on modes shows moderate effects mode correlation to baseline, with the positive constraint producing slightly higher cor- relation in deeper layers. All SAEs indicate that later layers learn sparser, more distributed class representations.
We further examined performance metrics of SAEs as a function of hyperparameters. Reconstruc- tion of raw contributions was consistently high (0.7-0.95) across all hyperparameters (Fig. S4 For modes with correlation to image class > 0.2, the number of class-correlated modes increased with layer depth. Larger dictionary sizes (N=5) yielded a greater number of clrselcor modes, how- ever the sum of those modes showed high class correlation insensitive to dictionary size, suggesting that additional modes capture redundant rather than distinct information. Very large MI.P encoders yielded fewer class correlated modes, likely due to feature splitting (Bricken et al. (2023)). Con- versely, the number of classes having correlation with modes of > 0.2 also was insensitive to variation of LI strength, random seed, MLP size, and constraint of MLP sign. As with other mea- sures, the dictionary size multiplier (N) needed to greater than 1, suggesting that sufficient initial overcompleteness is necessary for the SAE to discover class-specifee modes.
With respect to specific classes, changing hyperparameters had minimal effect on SAE performance and learned representations. We summed all modes with correlation > 0.2 for the black widow class in layer 15 (Fig. (5), finding that despite different configurations producing varying numbers of correlated modes, the summed representations remained strikingly similar.
Systematic hyperparameter sweeps with all other parameters held constant revealed robust speci- ficity patterns: dictionary size multiplier (N=1-5), activation threshold (0.5-0.9), mode LI regu- larization (0-le-2), random seed, and MLP hidden layer size (128-4096) all produced comparable ablation specificity with consistent trends of increased specificity in deeper layers, demonstrating that our interpretability results generalize across architectural choices (Fig. S6).
10.4 RUNTIME MEASUREMENTS AND COMPLEXITY
We measured contribution computation time for ResNet-50 across all 16 layers on the ImageNet validation set (50,000 images) on an NVIDIA RTX 6000 GPU. Using integrated gradients with 10 interpolation steps takes approximately 1 hours total, averaging 4.7 seconds per batch 16 images (293 ms per image). In contrast, activation x normalized gradient (act_normgrad) completes in 1.3 hours, averaging 1.5 seconds per batch (93ms per image).
We runtime measurements for training SAEs over 200 epochs an NVIDIA RTX 6000 GPU. For Resnet-50, SAEs for carlier layers train relatively quickly due to having fewer channels, with Layer 3 completing in min and Layer 15 in ~ 7 minutes. For VIT, training takes minutes per layer for attention and token features, and ~ 30 minutes for MLP (it has size 3072 compared to 768 for the other two layer types).
Computing contributions with activation x gradient is O(P) where P is the number of model param- eters, requiring one forward pass to obtain activations and backward pass compute gradients with respect to the target objective. Activation x integrated gradients has complexity O(nsteps P), scaling linearly with both the number of parameters and interpolation steps. In our experiments with nsteps = 10, this translated to approximately 3x compute time compared to activation x gradient.
Once contributions are precomputed, training contribution-SAE: has the same computational cost as training activation- SAEs, operating on feature vectors of identical dimensionality. This makes the contribution computation method the primary bottleneck, with activation gradient enabling practical large-scale analysis while Integrated Gradients provides more theoretically grounded con- tributions nlsteps x higher cost. The choice of decomposition method (SAF, PCA, or analyzing raw contributions) depends on the scientific question and represents separate complexity consideration.
To explore the scaling of CODEC to larger models, we additionally benchmarked contribution com- putation on Gemma-3n (5.44B parameters, 30 layers) across multiple NVIDIA Titan XP GPUs. Contributions were computed with respect to output entropy. At a sequence length of 128 tokens, activation normalized gradient completes in approximately 720ms per input, while integrated gradients (10 steps) requires approximately 8 seconds. Runtime remains invariant to hook location (post-atiention vs. posf-feedforwar)) and layer depth (early, mid, late), and scales sub-linearly with
15

---

Published as a conference paper at ICLR 2026
sequence length (empirical scaling exponent 0.10-0.11), projecting approximately sec- ond (ActGrad) and 11 seconds (IntGrad) even at 4096-token contexts. These results confirm that CODEC's contribution computation generalizes beyond vision models to large-scale language chitectures. We note that SAE training is a separate computational step, independent of contribution computation, and scaling to LLM-scale features is left to future work.
10.5 CODEC ON VITS
CODEC is a general framework that can be applied to any feedforward neural network. To show its application across architectures, we apply CODEC ViT-B, which is encoder-only transformer with 12 transformer blocks (Fig. ST7. Because ViTs do not have the same channel and layer struc- ture as CNNs, we log activations and gradients compute contributions at three types of layers for each transformer block-token features, MLP, and attention-ans and make the appropriate summa- tion that parallels the spatial sum we perform for each channel CNNs. Below are the details of implementation:
Token features: We log the activations and gradients at the end of each transformer block after the merge into the residual stream, with shape (batch, num num tokens, hidden dim).
MLP: We log the activations and gradients the hidden layer of the MLP, after the nonlin- earity, with shape (batch, num tokens, MLP hidden dim).
Attention: We log the activations and gradients after self attention but before the out pro- jection (mixing), with shape (batch, num_tokens, num_heads head_dim).
We use Integrated Gradients with 10 integration steps as the contribution algorithm for the follow- ing results. We use the post-softmax logit corresponding to the top predicted target class as the contribution target. After we compute the contributions for each neuron, we perform the appropriate summation to obtain tensor of shape (batch, pseudo-channels):
Token features:
Because each token is a patch, we sum over the tokens to obtain a ten- sor of shape (batch, hidden dim). where the hidden dimension corresponds to channels, intuitively interpreted as "global feature detectors".
MLP: We treat each MLP hidden neuron as its own channel, so we sum over tokens to obtain a tensor of shape (batch, MLP MLP_hidden_diin hidden_dim).
Attention: We treat each hidden dimension in each attention head channel, so we sum over both tokens to obtain a tensor of shape (batch, num heads head dim).
For each of the three layer types in ViT, we compute a contribution tensor analogous to that of CNNs-shape (batch, channels)-and we term the second dimension pseudo-channels. Like CNNs, separate positive and negative contributions within each pseudo-channel and concatenate them to obtain (batch, 2*channels) before the summation, where half of the channels only contain the positive contributions with the negative contributions set to zero and vice versa. Then, take the positive channels and use SAEs to find contribution modes, interpreted as groups of channels acting together to positively drive the output. For activations, we do not separate into positive and negative components before the SAE because the sign does not carry special meaning for activations as op- posed to contributions (where a positive contribution predicts decrease in the contribution target ablated). We scanned minimally for SAE hyperparameters and settled on the default hyperparame- ters used for R0..
A note about completeness: Because MLP and attention contributions are computed within trans- former block rather than on the full residual stream, their contributions will not be complete. In- tuitively, even if one ablates the entire MLP all attention heads, the residual stream still carries over information from the previous block, which can still maintain good classification performance. Therefore, of the three layer types, should only expect token features ablations to converge to zero performance when all pseudo-channels are ablated. fact, empirically, we observe that the ab- lation of entire MLP or attention layer within a single transformer block only results in 10% decrease in performance.
We note that ViTs could be utilizing very different computational strategies than CNNs for image classification. In particular, there no explicit spatially equivariant bias in the transformer architec- ture. We applied CODEC to ViTs in straightforward manner analogous CNNs by summing over
16

---

Published as a conference paper at ICLR 2026
"space" (tokens) to obtain spatially invariant feature detectors information streams, which may not be how ViTs organize computation in its dimensions due to the lack of this bias. Therefore, we should not expect the same level of general perturbation performance CNNs for the same SAE architecture and hyperparameters. However, we show below that contributions as a causal measure still reveal causal information not captured by activations.
10.5. SPARSITY
Plotting the contributions and activations for single image at the three layer types at layer 9, we see that contributions provide a sparser description of the model (iig./77A-CC Using the same measures of sparsity as in Resnet-50, we computed the average sparsity (over images) for the three layer types across the entire network (Fig. (S7D-F). As in Resnet-50, contributions are more consistently sparse than activations. Additionally, we observe similarities and differences in sparsity trends for the three layer types across depth. For all three layer types, we observe the general "inverted U" trend, where sparsity goes down at the beginning and eventually increases towards the deep end of the model. However, for MLP we additionally observe near maximum sparsity approaching the final layers but drop in sparsity at the final layer. These observations suggest distinct sparse organizations used by the different layer types, the full exploration of which we leave for future work.
10.5.2 CORRELATION ANALYSIS
We compute the average (across images) maximal correlation to target classes for four units- activations of pseudo- pseudo-channels, contributions of pseudo-channels, activation modes, and contribu- tion modes. Shown in Fig. ST are these correlations for the three types of layers that we analyze.
Across layer types, we observe an increase in correlations as depth increases, except the slight drop for MLP. Furthermore, we observe that modes are more correlated with target classes than individual pseudo-channels for token features and attention, with activation modes more correlated than contribution modes. This suggests that as depth increases, the model not only increasingly represents but also drives target classification with its hidden units. It also increasingly uses groups of hidden dimensions to collectively represent or drive the output classification.
10.5.3 PERTURBATION ANALYSIS
We also performed ablation analyses VIT-B using CODEC, probing the causal efficacy of contribution-SAN against activation-SAF similar to Fig. 6 For each layer and a specified per- centage of pseudo-channels to ablate, we take six trials as follows. In each trial, we take 10 random target classes. For each target class, we first find the most correlated activation or contribution mode. Then, we randomly select a class that is not the target class ('off-target" class). We take 10 sam- ples from each class. Then, we ablated range of specified percentages (from 1% to 99%) of the most prominent channels in the mode and evaluated performance ratio of the perturbed model the original model on the on-target and off- off-target classes using the 10 images that we just sampled for each class. The performance ratio was then averaged across the 10 target classes as one measure- ment of the performance of that layer across ablation percentages. The area under this performance ratio-ablation percentage curve (AUC) was computed for on-target and off-target ablations and the difference taken to quantify ablation specificity. The AUC ranges from 0 (performance drops to 0 as soon as the first channel is ablated and stays there) to I (performance stays at original until all chan- nels are ablated) except for rare cases where perturbed performance exceeds original performance. We plot the AUC differences for each layer and also show the mean and SEM for each layer in Fig. S7
-L. We summarize findings below:
Token features. Contributions yield significant ablation specificity, since the AUC for on-target ablations is significantly below that of tfrgtt ablations. Contributions are also significantly more effective than activations, especially at late layers. Furthermore, as depth increases into the last lay- ers, contribution performance mostly monotonically increases while activation performance stays mostly flat and similar to the baseline before deteriorating into the last layers, even though corre- lations of both contributons and activations increase monotonically. For activations, ablating their modes in the deeper half of the model actually hurts off-target classes more than on-target classes, suggesting a unique computational strategy used by ViT's to represent classes in hidden units that
17

---

Published as a conference paper at ICLR 2026
causally oppose those classes. This organization may indicate an inhibitory effect similar to in- hibitory neurons that carry information about a specific visual feature but suppress that same feature for gain control spatial decorrelation (olveczky et al. 2003)). Even though class information is better represented by activations as depth increases, the causally contributing components that drive classification drift away from the most prominently activating token features (which become causally inhibitory), necessitating the use of causal measure like CODEC to capture the causal effects in the late layers.
MLPs. The overall ablation effect for MLPs is weaker than for token features, which is expected because MLP neurons do not include the residual stream. However, contribution-SAF still yields significant ablation specificity for later layers compared to baseline and is significantly more ef- fective than activation-SAE. Because ablating the entire MLP in one block usually only yields 10% decrease in performance (which would translate a -0.1 floor of the AUC difference), both contribution-AAE and activation-SAE ablations approach this floor at layer 10, but contribution- SAE ablations are more effective in the layers leading up to 10. Furthermore, we observe a sharp performance drop to baseline at the last layers for both, despite the fact that correlations for both contribution and activation modes stay high into the last layer. One possibility for this effect that the most causally relevant components may be shifted into the residual stream as depth increases, and what is remaining in the MLP at these last layers more clearly represent the target classes but are less causal.
Attention. The overall ablation effect for MLPs is even weaker than for MLPs, but contribution- SAE still yields specificity at the deepest layers and is significantly more effective than activation- SAE, which stays flat near the baseline. Compared to the -0.1 maximum possible performance, contribution-S0 gets roughly halfway there in the last few layers.
Overall, straightforward application of CODEC on ViTs by treating hidden dimensions as channels and tokens as spatial patches shows that contribution-AAE enables target class ablations better than Satiaation-SAE. Even though ablation performance is weaker than in CNNs (the comparable case being token features, which are taken after the merge into the residual stream), we note that summing over tokens as space may not align with how ViTs organize computation because of their lack of explicit spatially equivariant bias. We leave an exploration of the optimal spatial reduction strategy for ViTs future work.
10.5.4 ATTENTION HEAD SPARSITY
Unlike ResNet-50, ViTs utilize multihead attention. We apply CODEC investigate whether indi- vidual attention heads specialize in functions that causally drive output classification.
We analyzed head specialization through contribution sparsity, which can manifest at the macro scale (few highly active heads) or the micro scale (active dimensions scattered evenly across all heads). To quantify this, we compute the Hoyer sparsity index on 12-dimensional vecior-derived by summing contributions or activations within each head- and compare it against a baseline of randomly grouped hidden dimensions. A sparsity index higher than the baseline indicates head specialization.
As shown in Fig. 58 randomly grouped heads show nearly identical sparsity for contributions and activations. Learned attention heads show a small increase in sparsity for both contributions and activations, but the increase is well within the fluctuation over the dataset. Recall from Fig. F that contributions are about 0.1 to 0.2 higher in sparsity than activations for attention across depth, and sometimes exceed 0.5 sparsity. Comparatively, the head-level sparsity is much smaller than the sparsity observed at the individual channel level. Therefore, these results demonstrate that atten- tion heads are minimally specialized in representations as well as causal functions, with active and contributing hidden dimensions spanning across attention heads.
10.6 CONTRASTIVE TARGETS
In addition to the targets listed in the main text, we also computed contributions with respect to contrastive target (Fig. STT). Both contrastive and top-1 targets produced similarly sparse represen- tations across all network layers, demonstrating that the contrastive target yields sparsity patterns
18

---

Published as a conference paper at ICLR 2026
consistent with the top-1 approach used in the main paper. We successfully decomposed contrastive contributions with R² above 0.85 for the four analyzed layers, suggesting that contrastive targets can be used for similar causal analysis.
10.7 CODEC IN DEPTH: ADDITIONAL DETAILS ON CONTRIBUTION ALGORITTMS AND THEIR COMPLETE DECOMPOSITIONS IN INPUT SPACE
The fundamental algorithmic innovation of our work is that design an approach address the question of how combinations of hidden neurons drive network output. This contrasts with an anal- ysis of activations, which at any level and for any input are not guaranteed drive network output. Here, we provide additional details of our methods.
We revisit and describe the motivation behind our several contribution algorithms, which are inspired by gradient-basee attribution methods, and we derive their complete decompositions in input space. For notation, let h; be the hidden units in a specific layer All attribution methods need scalar output target, typically select output neuron or scalar function of the output neurons. Let 7 denote the target (for example the output logit of a target class) and == fr be the neural network with a scalar output y corresponding to T' For clarity, define the complete input space decomposition of hidden units:
Definition 1 (Complete input space decomposition of hidden units), Let Cj be the contribution of hidden unit and Cig be an input space decomposition of Cj. i.e. Cij is the part of C2 assigned to input pixel i. Then, we call this input space decomposition complete if
Co== C
We derive the complete input space decomposition ActGrad, InputGrad. and Integrated Gradients under this definition, which will naturally generalize to mode contributions by linearity simple sum over the all the hidden units that mode.
A note about ActGrad To be consistent with the rest of the derivation, slightly generalize ActGrad to account for possible nonzero baselines:
Oy
ActGrad, = -h3--1h - -h))- ah,'
(5)
where hy is the activation of hidden unit j (which could be the input as a special case) at input x and h² is the hidden activation baseline input x'. For image classification, usually take bascline hidden activation of zero, which is the definition in the main text.
Input x Gradient (InputGrad) Used in Shrikumar et al. (2017) a baseline, this is a special case of ActGrad where Act is the input values. However, seeing InputGrad this way means it can only attribute to the input layer. We naturally extend InputGrad to hidden layers to obtain contributions by first defining the input space decomposition:
dy Thy
InputGrady
(x: - x!)
Th, Oz;
(J)))) (x--1/).
where J, = Th the Jacobian from the output the hidden layer and Jr = dh. Uz. is the Jacobian from the hidden layer to the input. If we sum over i, we then obtain a contribution algorithm to hidden neurons:
Hlnputrrad, = Σ InputGrady.
(8)
Under this definition of InputGrad contribution, the input space decomposition is trivially complete. This is well-mon motivated extension of InputGrad because recover the original InputGrad by sum- ming over j instead:
InputGrad, = Σ InputGrad3-
(9)
19

---

Published as a conference paper at ICLR 2026
This natural extension rests the chain rule of partial derivatives, where essentially postpone summing gradients over the hidden layer until after multiplying by the input element-wise. In- putGrad contribution algorithm may seem bit ad hoc, but it will turn out to have the most computationally efficient input space decomposition that we consider and we theoretically motivate it later by showing a connection Integrated Gradients.
Integrated Gradients (IG) (Sundararajan et al. (2017) IG assigns attributions to input pixels by calculating the integral of gradients along a straight-line path from baseline input x' to the actual input x: IG; (z; xi))
Oy(x' + a(x x')) da.
(10)
a==
Or;
IG satisfies the (output) completeness property, often desired in attribution methods:
IG,-= = yaa) (ar)).
(11)
In words, completeness means that the attributions each input pixel sum up to the (change in) output. In this original form, IG only works for the input layer. We present two ways to extend IG to hidden layers:
Treat the hidden layer as the input and follow through. This the extension by Lucas et al.) (2022), where they combine this extension of IG with GradCAM. Mathematically, the "hidden integrated gradients" (HIG) is defined as
f
Oy(h' + +ahh h))) da,
HIG, (h2- 1 h2)
(12)
Jazo
where h is the hidden layer activation vector at input x, and hn' is the hidden layer activation vector at input x'.
Take the same path in input space but decompose the gradients at the hidden layer and sum over the input layer. Tanaka et al. (2019) first introduced this method and applied to the first hidden layer of model of the retina. Here, we use it on any hidden layer. Mathematically, first, we decompose the gradients over the hidden layer:
['' Ja=O
Oy(x' + +a(x xx)) 0h, da,
HIG
x1)
(13)
ah,
Ox;
where i indexes the input pixels and j indexes the hidden neurons. Note that we can recover standard IG by summing over j due the linearity of the integral and chain nule of partial derivatives:
IG, Σ HIG
(14)
which motivates this extension. Now, to obtain contributions in the hidden layer, we simply sum over instead of j:
HIG Σ HIG
(15)
In practice we approximate the integral using a Riemann sum with m steps. For standard IG, we have
Oy(x' + -x
IG, ~ (xt
(16)
Iz;
This approximation extends straightforwardly to the first method of HIG, simply by replacing x with h. For the second method, we could naively discretize in input space to get HIG,, first, but that would involve backward pass for each hidden neuron each step. Instead, we analytically derive
20

---

Published as a conference paper at ICLR 2026
an approximation that still only involves one backward pass each step. We have
Ty(x' + a(x x'))
Thg da dz,
HIG
Σ (0g xzi)
(17)
a=
dy Thg 0x:
da
(18)
Thi 0x. da
0a=0
x' x')
By
0h dx;
da
(19)
Th
dx; da
x xxxxxxxx))
Dy 0h
da.
(20)
Thg do: Mxx'+a(x-x)))
Note that this is precisely IG treating h as the input layer (the first extension) with the important distinction that the path is not a straight line (it is warped by the nonlinear functional relationship between x and h). The crucial advantage of computing HIG this way over the first method is the we get for free. To approximate this line integral
complete input space decomposition (Equation 13 with Riemann sum, we have
Qy
HIG,
dhj.
(21)
Th,
1=
h(a= )
where we can compute h(a m at each k in forward pass and dhş = 11((x =n h single backward pass.
k-1
Complete input space decomposition of ActGrad It is a bit more nuanced to define a complete input space decomposition for ActGrad in hidden layers. The idea is to use IG to attribute Act to inputs, which we then multiply by Grad to get the decomposition. Mathematically. we exploit the output completeness of IG to express Act as an integral of gradients, which we can then naturally decompose linearly in input space. We define the input space decomposition as
dy
ahy do,
ActGradij = -x,--00 x
(22)
Ohy
0 Or;
where we take straight- line path in input space parameterized by a. We can prove that this decom- position is complete by recovering the standard ActGrad for hidden layers if we sum over
Oy Ohy
Thi da Oz,
Σ ActGrad =
>(m (x: =11)
(23)
a=
Oy
Öh; da dx,
Σ (xx x1)
(24)
ahy x
a==
Oy
(hg-h/)
(due to the output completeness of IG)
(25)
ahy x
ActGradjj
(26)
where the second-to-last step is true even though the path in h space not necessarily straight by the fundamental theorem of calculus for line integrals. Note that if we summed over j instead, we would obtain a new attribution method for the input layer that is like hybrid between IG and InputGrad
(the difference being which part of the chained derivative is integrated over). We do not explore that method any further as we are primarily interested in contributions of hidden neurons.
InputGrad and ActGrad as approximations IG The unifying connection behind all three contribution algorithms is that they are the same for a linear network. The three algorithms rep- resent successively more comprehensive ways capture the nonlinearity of the network in their contribution assignments. Recall the definition of HIG (Equation 20):
Öy Oh
HIG, Jaz0
da.
(27)
Thy da xxx+a(xxxx))
21

---

Published as a conference paper at ICLR 2026
If assumee that the downstream network is linear along the integration path, ie., the downstream gradients By are constant, recover ActGrad: Shy
Oy dhg
dy 1
Ohg
da
da
(28)
Th da h(x'+o(x-x'))
S a= Ea h(x'+a(xxx'))
dy Dhi (hij 1ij-li) 1(z))
(29)
ActGradj.
(30)
If further assume the whole network is linear along the integration path, i.c. all gradients constant, we recover InputGrad:
Dy 0hg Thy da +x(xxx(
Oy
Thg Ox, Drg da
da
da
(31)
-a=0
x(+a(xxxx)
0y
Th dz:
do
(32)
Thi
Oxs da
aaz
Σ dy 0hi
(x: x²))
(33)
Thg ox; InputGrad,
Σ
(34)
InputGrad,-
(35)
By taking the full integral, IG has the output completeness property Σ, HIG;-yyy' that ActGrad and InputGrad don't have. Furthermore, IG alleviates the saturation problem where zero gradient counterintuitively leads to a zero attribution (and therefore contribution) (Shrikumar et al. (2017); Srinivas and Fleuret (2019)).
Under our definition, a mode's contribution is the sum of the contributions of its top-k channels. Due to the linearity of sums and integrals, the complete input space decompositions derived above for individual hidden units naturally generalize to those of modes.
The key insight that underlies input space completeness is the chain rule of partial derivatives and linearity of integrals and sums (and additionally the output completeness of IG for ActGrad decom- position). The high-level procedure
1. When computing the contribution of each hidden unit, do not the gradients the input space just yet; save the full contribution tensor with the input index.
2. Sum over the input space and cluster to find modes.
3. Go back the unsummed contributions and sum over each mode get a complete decom- position of mode contributions in input space.
Practically, it is more computationally expensive to use the input space decomposition of IG and ActGrad than InputGrrad due the integration. Therefore, rithm for all our input space visualizations. Anecdotally, for attributions to input space, [Selvaraju al. (220)) notes that ActGrad at earlier layers produce less clear heatmaps, but since we use IG the hidden layer first to identify modes, we do not expect the loss in quality to be as significant by switching to InputGrad just for visualization izations show sufficient quality for our purposes. However, our theoretical framework allows for principled, complete decomposition necessary for the application at hand.
InputGrad the backbone algo-
input space. Empirically, our experimental visual-
10.8 VISUALIZING CONTRIBUTION MAPS
For each image, computed contribution maps using the InputGrad method described in the main text. Specifically, for a target class and a set of mode-selected channels identified via CODEC analysis, computed the Jacobians J Jy.ha. 0y/Ohe. (gradient of the softmax class output
22

---

Published as a conference paper at ICLR 2026
with respect to each hidden unit activation) and Jhe...: Dhc.p/Oz; (gradient of each hidden unit activation with respect to each input pixel), retaining spatial resolution across all positions in the feature map. The per-channel, per-position input sensitivity was then formed as A A
Jyh. 2nc. and the contribution map was computed as the clementwise product C/(P)
A (c:P) 3 I₁. Contributions were aggregated across all spatial positions and mode-selected channels by summation, and were separated into positive and negative components prior to aggregation; only positive contributions were retained for visualization. The resulting contribution map was averaged across color channels, fidiaanfiltered with a kernel of size 4 to reduce noise, normalized to [0, 1J. and scaled by contrast factor of 7 before being clipped to [0, 1] and used as a multiplicative mask on the mean- and
input image for display.
23

---

Published as a conference paper at ICLR 2026
SUPPLEMENTAL FIGURES
A
Activations
Contriutions
class
Channel 39
Contribution Method:
Acttrad Sumn of all logit
IntGrad Most active logit
IntGrad (softmma)
IntGrad
Contribution Targets:
Excitatory
net EXC-
net EXC
Layer S
Contributio
1i
Excitatory
Contribution
Layen
Inhibitory
inhibiory
-2
Channel
Figure S1: Comparison of attribution methods and targets reveals consistent channel contribu- tion patterns. (A) Spatial maps of activations, gradients, and contributions computed using different contribution algorithms (ActGrad, IntGrad) and targets (sum of all logits, most-active logit, most- active logit with softmax, entropy) for Channel 39. All methods show similar spatial sparsity. (B) Excitatory and inhibitory contribution maps for Channel 39 Layer 5 for single image demon- strating the separation of excitation and inhibition in an early layer. (C) Contribution values across channels for one image showing the distribution of net excitatory and inhibitory effects for Layer 5. (D) Same as (B) but for Channel 369 in Layer 11, illustrating consistent excitalory/nhiiiiory patterns across different channels. (E) Same as (C) but for Layer 11, showing how the excita- tory/inhibitory decorrelation evolves across network depth.
24

---

Published as a conference paper at ICLR 2026
A
Number of Modes W
C
Class average Panda mode
4000
Normalized contribution
0..6
0.50
Contributions
0.25
Activations
2000
0.00
0
Layer
10
Layer
10
1000 Channel
2000
Mode sparsity
0.4
0.3
Layer
Figure S2: Sparse autoencoder decomposition performance and sparsity analysis across net- work layers. (A) Reconstruction accuracy (R²) of sparse autoencoder decompositions for contri- butions (black) and activations (blue) across network layers. Both show high reconstruction fidelity, with contributions maintaining slightly higher R² values in intermediate layers. (B) Number of modes discovered by the sparse autoencoder decomposition tributions (black) and activations (blue). The number of modes increases through the network, with contributions yielding more modes than activations in deeper layers. (C) Comparison of the most correlated panda mode with the average contribution pattern during presentation panda images. The decomposition identified mode (black) that shows similar contribution patterns to the class- average contributions when panda images are presented (r=0.6). (D) Median Hoyer sparsity index of learned modes across network layers for contributions (black) and activations (blue). Contribution modes maintain higher sparsity than activation modes throughout the network.
function of network depth for con-
25

---

Published as a conference paper at ICLR 2026
Mode 1o-2
Loading thresh.
modes
MLP size
Mode sign
0.5
483
128
No constraint
Positive constraint
0.9
2002
2048
Correlation Coefficlent
0.4
0.2
0.0
5
10
15
5
10
15
5
10
15
5
10
15
5
10
15
5
10
15
Layer
Layer
Layer
Layer
Layer
Layer
Figure S3: SAE mode stability across hyperparameters and random seeds. SAEs were trained on Resnet-50 contributions for example layers 3, 13, and 15, varying hyperparameters L1 sparsity penalty strength, threshold, dictionary size (N number of channels), random seed, MLP encoder size, and non-negativity constraint were varied across the indicated values. Each panel shows the correlation coefficient between baseline SAE modes (used in main text) and modes learned with varied hyperparameters
function of network layer. Modes are aggregated for each class by summing all modes whose correlation with that ImageNet class exceeds 0.2; if no modes exceed this threshold, the single top-correlated mode is used. Shaded regions show mean standard error of correlation across 1000 ImageNet classes.
26

---

Published as a conference paper at ICLR 2026
L1
Loading thresh.
Num. modes
Seed
Mode sign No constraint
128
1e-4
1001
512
Positive constraint
1e-2
0.9
2002
2048
1.0
4096
0.8
Store
0.6
0.4
0.2
0.0
Classes with Corr >.2 Modes with Corr .2
5000
2500
0
1000
750
500
250
10
15
5
10 Layer
15
10
15
5
10
15
10 Layer
15
5
10
15
Layer
Layer
Layer
Layer
Figure $4: SAE performance metrics are robust across hyperparameter choices. Each panel shows a metric (rows) across network layers (x-axis) for different hyperparameter values (colored lines). For each hyperparameter column, values averaged across all other hyperparameter con- figurations. Shaded regions indicate mean + standard eтror across configurations. Top row: R² score, reconstruction quality across all hyperparameters. Middle row: The number of modes having a correlation with classes 0.2 for different hyperparameters. Bottom row: The number of classes having correlation with modes 0.2 for different hyperparameters.
27

---

Published as a conference paper at ICLR 2026
Modes L1
Loading thresh.
Num. modes
Seed
MLP size
Mode sign
Baseline
0.25
0.00
0.00
0.00
0.00
0.00
0.00
0.5
483
128
Positive constraint
0.5
0.5
0.5
0.25
0.0
0.00
0.00
10-4
0.7
1001
512
No constraint
0.5
0.0
0.0
1e-2
0.9
2002
2048
2000
0.5
0.5
Channel
0.5
0.0
0.0
0.0
0.0
2000
2000
2000
2000
4096
Channel
Channel
Channel
Channel
0.5
0.0
Channel
Figure S5: Class-specific mode representations are consistent across hyperparameter choices. For the Black Widow class summed all modes with correlation >> 0.2 to that class. Shown is the sum for SAEs trained with different hyperparameters.
28

---

Published as a conference paper at ICLR 2026
Num. modes
Loading thresh.
Mode L1
0.5
1.0
1.0
0.7
1.-
1o-d
0.9
fo-2
AUC
-0.5
-0.5
-0.5
1.0
1.0
10
10
15
15
Seed
MLP size
128
1.0
1001
1.0
512
4096
AVC
-0.5
0.5
1.0
1.0
10
Layar 15
Layer
Figure S6: Hyperparameter robustness of SAE ablation specificity across architectures. Ab- lation specificity measured as normalized AUC difference across ResNet-50 blocks (3,7,13115) for SAEs trained with varying hyperparameters: dictionary size multiplier (N=1,3,5), activation thresh- old (0.5,0.7,0.9), mode L1 regularization (0, 1e-2, le-4), random seed (1001, 2002, 483), and MLP hidden layer size (128-4096). Threshold 0.9 corresponds used in the main text. Normalized AUC difference integrates performance ratios (perturbed/original accuracy) across ablation percentages (25%, 50%), with negative values indicating greater speci- ficity (larger target class accuracy drop relative to off-target classes). Results demonstrate robust- ness across hyperparameter choices, with all configurations showing increased specificity in deeper layers. Error bars indicate SEM across ten trials with randomly sampled class pairs.
baseline of hyperparameters
29

---

Published as a conference paper at ICLR 2026
Activations
A
Normalized Value (std 1)
Contributions
B
Token Features
MLP
Attention
Single Image, Layer 9
Single Image, Layer 9
Single Image, Layer 9
20
40
10
20
0
0
0
100
-5
0
500
2000
0
500
Channel
Channel
Channel
D
E
F
Sparsity (Hoyer)
1.0
1.0
1.0
0.5
0.5
0.5
$
* +0
+H
H+
0.0
0.0
0.0
11
Layer
Layer
Layer
G
H
1.0
1.0
1.0
Contributions (chans)
Avg. Corr.
0.8
Activations (chans)
0.8
0.8
Contributions (modes)
0.6
Activations (modes)
0.6
0.6
0.44
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0
5
10
5
10
5
10
Layer
Layer
Layer
AUC(Target) AUC(OffTarge)
K
L
0.05
0.25
0.00
0.00
-0.05
0.00
-0.10
-0.05
-0.25
x
0
6
8
11
3
6
11
0
3
11
Layer
Layer
Layer
Figure S7: Application of CODEC on VIT-B, The columns correspond to the three layer types- token features, MIP, and attention. (A-C) Examples of contributions and activations computed for a single image at layer 9. (D-F) Hoyer sparsity as a function of depth for activations (blue) and contributions (black). Box plots show the distribution of sparsity values over the entire Im- ageNet validation set, omitting outliers for clarity. (G-I) Analogous to Fig. 5P in the main text, the mean over units (pseudo-channels or modes) of the maximal class-correlation for each unit as function of depth for four cases: contributions of single aseudo-channed activations of single pseudo-channels, contribution modes, and activation modes. (J-L.) Analogous Fig. 6 perfor- mance score quantifying perturbation effectiveness the area between target and off-target curves from Fig. 6D normalized to the off off-target area, as a function of depth. Six trials are performed for each layer, shown scatter points, and the SEM is shown the shade.
30

---

Published as a conference paper at ICLR 2026
Activations
Contributions
B
C
Learned Attention
Random Attention
Learned Random Split Attention Head Sparsity:
Head Sparsity:
Head Sparsity:
Contribution vs. Activation
Contribution vs. Activation
Contribution vs. Activation
1.0
1.0
Sparsity (Hoyer)
0.5
0.5
0.5
11
0.0
0.0
3
8
11
8
11
Layer
Layer
Layer
Figure S8: Attention head sparsity for VTT-B. (A) Hoyer sparsity as a function of layer after summing contributions (black) activations (blue) within each attention head. Box plots show the distribution of sparsity values over the entire ImageNet validation set, omitting outliers for clarity. (B) Hoyer sparsity function of layer after summing contributions activations within attention heads formed by random grouping of hidden dimensions. (C) Distribution of the difference in Hoyer sparsity for the learned and randomly grouped attention heads as function layer after summing contributions or activations within each attention head.
Contributions
Activations
Layer 2
0.6
0.6
Layer
Layer
Sorted mode correlations (class-average)
0.4
0.4
Layer
0.2
0.2
0.0
0.0
20
20
40
60
Mode rank (top 60)
Mode rank (top 60)
0.6
0.6
Sorted class correlations (mode-average)
0.4
0.4
0.2
0.2
0.0
0
20
40
60
20
40
60
Class rank (top 60)
Class rank (top 60)
Figure S9: Contribution modes capture set of dataset classes. To determine whether our SAE models learned modes that captured the full set of classes in the Imagenet dataset, we computed the full mode-by-class correlation matrix each layer of the model for both activations and contri- butions. Sorting this matrix along either the rows columns gave insights into how many modes individual classes were correlated with (top), and how many classes individual modes were corre- lated with (bottom). We found that in later layers, classes on average were well covered by learned modes; furthermore, this measure of class-coverage was higher for contribution modes than for activation modes at intermediate layers of the model. Error bars indicate standard deviation.
31

---

Published as a conference paper at ICLR 2026
ablate dog mode types dogs
preserve dog mode types dogs
Top-5 accuracy
100
per-daass Tep-5 accuracy
100
per-dass
50
50
0
Top-5 accuracy
100
Top-5 accuracy
100
per-class
per-class
% change
0
change
-100
100
250
750
1000
250
500
750
1000
imagenet classes
imagenet classes
Figure S10: Manipulation of higher semantic classes using contribution modes. (A) Ablation ex- periment: A binary mask was created for all images with dog superclass labels, then correlated with mode loadings to identify dog-correlated modes. All modes with correlation > 0.2 were summed together. Removing channels from this aggregated dog mode in intermediate layers (12-15) elimi- nates the network's s ability to classify dog images while leaving other classes largely unaffected. (B) Preservation experiment: Using the same summed dog mode identified through correlation analy- sis, retaining only these mode channels allows the network to correctly identify dog images while reducing accuracy to near-zero for all other classes.
Spensity
Contrastive Cantrisutions
Contractive Contributions
Top-1 Contributions
Activations
Sparsity (Hoyer)
Sparsity (Hoyer)
0
12 15
8 Layer
12 15
Layer
Figure S11: Layer-wise sparsity comparison of contrastive and top-1 contributions. Box plots showing the distribution Hoyer sparsity contributions across layers 0-15 of ResNet-50 on Ima- geNet. Black boxes represent contrastive contributions with respect to the difference between top-1 and top-2 predictions, while blue boxes show top-1 contributions with respect to top-1 predictions (left) or activations (right).
32


Reviews:
[{"review_id": "iCZXDS48CL", "reviewer": "Reviewer_aiv4", "summary": "The paper introduces a two-stage pipeline that first computes per-channel contribution scores to a scalar target using gradient-based attribution, then trains a sparse autoencoder over these contribution vectors to discover sparse \"modes\". The authors claim these modes are sparser than activations, align with classes, enable selective output control via targeted channel masking, and support channel-level visualizations. Experiments are on ResNet-50 and a small retinal CNN.", "questions": "* How does the method extend to ViTs and transformer units? Please report a compact ViT-B experiment at a few layers and compare contribution-SAE against activation-SAE or token features for steering and selectivity.\n* Can you evaluate per-logit and contrastive targets and report how mode sparsity, stability, and class selectivity change?\n* Please add SAE ablations over dictionary size, L1 strength, thresholding, and signed modeling, with stability metrics for mode-class correlation and intervention effect sizes\n* If you select modes across several layers, can you achieve stronger or more selective control than single-layer masking?\n* For the retina study, what specific new biological hypothesis or prediction follows from your modes that prior retinal CNN work did not already suggest?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* Originality: Clear formulation that swaps activations for contributions before SAE, producing sparse modes that are easy to manipulate\n* Quality: Targeted masking yields selective effects for classes inside ResNet-50 and the retina case study shows external face validity\n* Clarity: The method is explained step by step, and the visualization procedure is easy to follow\n* Significance: The pipeline could be a practical analysis and editing tool for CNNs and may assist mechanistic probing in constrained settings", "weaknesses": "* Modern literature already demonstrates SAE-driven feature discovery and steering in ViTs and vision-language models. The submission does not compare against these systems or articulate why contribution-SAE is preferable.\n* All main results are in CNNs. There is no validation on ViTs, attention heads, MLP neurons, or token features, which is where much of the community focuses today\n* The experiments show interventional control under channel masking in one backbone but do not establish model-invariant causal mechanisms - so the causal language is not entirely warranted\n* Emphasis on top-k logits or entropy may conflate classes. Per-logit or contrastive targets would better test class specificity\n* No systematic ablation is provided over dictionary width , sparsity penalty, or thresholds. Stability of mode discovery is also unclear\n* The \"century of neuroscience\" narrative should be replaced by a concise, current context with concrete recent citations", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jl2Sb69G31", "reviewer": "Reviewer_MSQk", "summary": "This paper introduces CODEC (Contribution Decomposition), a method for interpreting neural networks by analyzing how hidden neurons contribute to network outputs rather than just their activation patterns. The approach uses gradient-based attribution methods (Integrated Gradients, ActGrad) to compute neuron contributions, then applies sparse autoencoders to decompose these contributions into interpretable modes. The authors apply CODEC to ResNet-50 on ImageNet and to models of retinal neural circuits, demonstrating that contribution-based analysis enables better network control and reveals computational structure invisible to activation-based methods.", "questions": "## Questions\n\n* A critical question for practical application of CODEC concerns hyperparameter selection. Can you provide systematic guidance for choosing the number of modes (k), the appropriate sparsity penalties, and other key hyperparameters? Currently, the paper acknowledges that these choices matter but offers no principled approach for setting them. It would be valuable to understand whether there are data-driven methods for selecting k (perhaps based on reconstruction error curves, or information-theoretic criteria), how sensitive the discovered modes are to different sparsity penalty values, and whether certain hyperparameter configurations consistently work well across different network architectures or layers. Without such guidance, practitioners would need to perform extensive manual tuning, which limits the method's accessibility and reproducibility.\n\n* Another important consideration is the stability of the discovered modes. Are the modes stable across different random seeds in the autoencoder training, across different training runs of the original neural network, or across slight architectural variations? This is crucial for understanding whether CODEC identifies fundamental computational structures in the network or whether it is sensitive to arbitrary training details. For instance, if you train ResNet-50 multiple times with different random initializations, do you recover similar modes? If you slightly modify the architecture (for example, changing the number of channels in a layer or using a different residual connection pattern), do the modes remain interpretable and consistent? Understanding this stability would help establish whether modes represent robust computational motifs or are more ephemeral features of particular network instances.\n\n* Finally, to my understanding, the current approach assumes that contributions can be decomposed as linear combinations of modes, which may be a limiting assumption. What about interactions between modes? In practice, neural network computations often involve complex, nonlinear interactions between features, where the effect of one feature depends on the presence or absence of others. The linear decomposition might miss these interaction effects entirely. Have you considered approaches that could capture mode interactions, such as examining joint activation patterns of multiple modes, or using more sophisticated decomposition methods that allow for multiplicative or other nonlinear interactions? Understanding the limitations of the linear assumption and whether important computational structure is being missed would strengthen the interpretation of the results.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "## Strengths \n\n* The paper presents a novel and conceptually appealing perspective by focusing on how hidden neurons contribute to network outputs rather than merely analyzing their activation patterns. This shift is well-motivated from neuroscience principles, where understanding neural function requires examining both receptive fields (input sensitivity) and projective fields (output effects). The theoretical framework is rigorous, with complete input space decompositions properly derived for ActGrad, InputGrad, and Integrated Gradients, providing a mathematically sound foundation for the contribution analysis. \n\n* One particularly compelling aspect of this work is its dual application to both artificial and biological networks. I find the application to retinal neurons especially relevant and valuable, as it represents a novel translation of topics typically associated with mechanistic interpretability of artificial neural networks to models of biological systems. This demonstrates the breadth and potential impact of the CODEC framework beyond standard machine learning interpretability tasks, bridging computational neuroscience and AI interpretability in a meaningful way.\n\n* The paper provides causal validation through ablation and preservation experiments, which convincingly demonstrate that the discovered modes capture functionally relevant channels. These experiments show that targeting specific channels identified by CODEC can selectively eliminate or preserve classification performance for target classes while leaving other classes largely unaffected, providing evidence beyond mere correlation that the method identifies causally important computational pathways.\n\n* The findings themselves are also scientifically interesting. The progressive decorrelation of excitatory and inhibitory effects across network layers is a novel observation that provides insight into how neural networks organize computations hierarchically. Similarly, the observation that contributions become increasingly sparse and high-dimensional through the network offers new perspectives on how feature processing evolves with depth, findings that would not be apparent from activation analysis alone.", "weaknesses": "## Weaknesses \n\n* While the paper acknowledges sensitivity to hyperparameters including hidden layer size and regularization, this issue is not systematically studied or addressed. The authors mention that these parameters \"may require tuning for different architectures\" but provide no guidance on how to perform this tuning, no ablation studies exploring the sensitivity, and no principled approach for selecting appropriate values. This is particularly problematic for the number of modes (k), which appears to be a critical hyperparameter that determines the granularity of the decomposition. Without systematic investigation or clear selection criteria, practitioners would struggle to apply CODEC to new problems.\n\n* The paper lacks any analysis of computational costs, which is a significant omission given that the method involves multiple backward passes for Integrated Gradients integration steps, and additional computation for the sparse autoencoder training and decomposition. Readers cannot assess the practical feasibility of applying CODEC to larger models or compare its computational requirements to alternative interpretability methods. This is especially important given that the authors suggest the method could scale to large language models, yet provide no evidence or analysis of whether this is computationally tractable.\n\n* The choice of contribution targets—specifically the sum of top-k logits and entropy—lacks thorough justification. While these are reasonable choices, the paper does not explain why these particular targets are optimal, whether other targets might be more informative for different analysis goals, or how sensitive the results are to this choice. The single contribution target formulation is motivated primarily by avoiding intractable 3D decompositions, which is a practical consideration but may limit the method's ability to capture class-specific computational pathways.\n\n* Regarding methodological details, the sparse autoencoder training procedure is underspecified in ways that would make reproduction difficult. Critical details such as the optimizer used, learning rate schedules, convergence criteria, weight initialization schemes, and the specific form of sparsity regularization are either missing or relegated to brief mentions. The paper states that results will be made available as supplementary materials, but the main text should provide sufficient detail for readers to understand and evaluate the approach. The lack of these details makes it difficult to assess whether the discovered modes are stable features of the networks or artifacts of particular training procedures.\n\n\n### Minor Details \n\n* From a presentation standpoint, Figure 8 is particularly difficult to read. The figure attempts to convey complex information about retinal neural network analysis, but most of the labels are too small to be legible, and the overall layout is cramped. I would suggest either significantly enlarging this figure, possibly splitting it across multiple subfigures, or substantially reducing the caption length to allow more space for the visual content.\n\n* Fig 1 is wrongly referenced multiple times: Line 237; Line 262, Line 269\n\n* Missing Fig reference in line 305", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "axwNLeegPy", "reviewer": "Reviewer_Jrdp", "summary": "This paper introduces CODEC, an interpretability framework for studying causal contributions of hidden units in artificial neural networks.  In this paper, this is performed by estimating the contribution of hidden units to an output (with ActGrad or Integrated Gradients, for instance), decomposing these contributions into interpretable “modes” with spare autoencoders (SAEs), and then studying the role of different network units (or channels) associated with the learned modes via visualization and/or ablation experiments.  The authors apply this framework to reveal interpretable patterns in CNN image classification models and models of human retinal activity.", "questions": "- How would you position this work alongside other mechanistic interpretability methods based on causal analysis (e.g., Activation Patching [1])?\n- The paper highlights the advantages of performing causal analysis on unit contributions instead of activations.  Are there any scenarios where you would expect analysis on the activation space to be more insightful?  When is it a better choice to use CODEC over an alternative, activation-based approach?\n- What is the coverage and specificity of CODEC to a broader set of ImageNet classes (see weakness 3)?\n\n\n[1] Kevin Meng et al. *Locating and Editing Factual Associations in GPT*.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- To my knowledge, this is the first paper to apply SAEs to contribution matrices instead of model activations.  The authors argue, and methodologically demonstrate, that this can help reveal causal contributions of hidden units that are not revealed through similar analyses on model activations.\n- The proposed method is flexible: architecture agnostic, can be adapted to different contribution targets, does not require access to the model’s original training data  (although, see weakness 1). This is a valuable feature of mechanistic interpretability methods.\n- Applying CODEC beyond traditional image classification models, the authors reveal interesting insights into casual operations in models of retinal activity.  This type of interpretability for neuroscience models could be quite valuable for NeuroAI researchers seeking to better understand the brain with models that have representationally aligned neural activity.", "weaknesses": "- Authors suggest the generality of CODEC to other architectures and tasks but do not provide evidence of this working.  Although logically, it should, it would be beneficial if the authors could substantiate this claim with small experiments on other tasks and/or modalities.\n- Human interpretability of modes are not guaranteed and their interpretations are often subjective.  Although in CODEC analyses the authors identify modes that are correlated with class outputs, it is not always the case that SAE modes are interpretable.  The interpretability of the causal contribution of a unit is hinged upon SAE modes being interpretable.\n- There is a lack of quantitative evaluation of the coverage (how many classes are represented by distinct contribution modes) or specificity (how uniquely those modes map to a given class).  While the “black widow” and “dog” (appendix Figure 11) examples are illustrative, a broader analysis across all ImageNet categories (or even semantic combination of categories (e.g., “contains animal”),  would clarify whether CODEC consistently identifies compact causal modes, or if such patterns are limited to a subset of classes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)