Readout Representation: Redefining Neural Codes by Input Recovery
Authors: Shunsuke Onoo
Yoshihiro Nagano
Yukiyasu Kamitani
Venue: ICLR 2026 Poster (2026)
Keywords: neural representation
readout representation
representation size
misrepresentation
neural variability
information recovery
feature inversion
hierarchical models
robust representations
artificial neural networks
biological neural systems
Primary area: applications to neuroscience & cognitive science

Abstract: Sensory representation is typically understood through a hierarchical-causal framework where progressively abstract features are extracted sequentially. However, this causal view fails to explain misrepresentation, a phenomenon better handled by an informational and teleological view based on decodable content and downstream functions. This creates a tension: how does a system that abstracts away details preserve the fine-grained information needed for downstream functions? We propose readout representation to resolve this, defining representation by the information recoverable from features, rather than their causal origin.
Empirically, we show that inputs can be accurately reconstructed even from heavily perturbed mid-level features, demonstrating that a single input corresponds to a broad, redundant region of feature space, challenging the causal mapping perspective.
To quantify this property, we introduce representation size, a metric linked to model robustness and representational redundancy. Our framework offers a new lens for analyzing how both biological and artificial neural systems learn complex features while maintaining robust, information-rich representations of the world.

Published as a conference paper at ICLR 2026
READOUT REPRESENTATION: REDEFINING NEURAL CODES BY INPUT RECOVERY
Shunsuke Onoo', Yoshihiro Nagano & Yukiyasu Kamitani'
'Graduate School of Informatics, Kyoto University, Kyoto, Japan 2 2ATR Computational Neuroscience Laboratories, Kyoto, Japan Guardian Robot Project, RIKEN, Kyoto, Japan
ABSTRACT
Sensory representation is typically understood through a hierarchical-causal framework where progressively abstract features are extracted sequentially. How- ever, this causal view fails explain misrepresentation, phenomenon better handled by an informational and teleological view based on decodable content and downstream functions. This creates a tension: how does a system that ab- stracts away details preserve the fine- fine-grained information needed for downstream functions? We propose readout representation to resolve this, defining representa- tion by the information recoverable from features, rather than their causal origin. Empirically, we show that inputs can be accurately reconstructed even from heav- ily perturbed mid-level features, demonstrating that a single input corresponds broad, redundant region of feature space, challenging the causal mapping perspec- tive. To quantify this property, introduce representation size, a metric linked model robustness and representational redundancy. Our framework offers a new lens for analyzing how both biological and artificial neural systems learn complex features while maintaining robust, information-rich representations of the world.
INTRODUCTION
The dominant view of neural sensory representation is rooted hierarchical-caust framework, where stimulus is processed through layers that extract progressively abstract features, and the causal origin treated as the contents of representation (DiCarlo et al., 2012; Kriegeskorte et al., 2008). This model, aligned with causal theories of representation in philosophy (Fodor, 1987), has spurred powerful analytic tools (Kornblih et al., 2019; Seung Lee, 2000). However, its strict causal foundation fails to explain misrepresentationn-2 phenomenon like illusions mental imagery, where representational content is decoupled from a direct sensory cause (Fodor, 1987). Alternative philosophical theories address this gap: informational accounts define representation by the information that a state carries (Dretske, 1981), while teleological accounts define it by its proper function for downstream consumer (Millikan, 1989). These non-causal views are empirically sup- ported by neural decoding studies, which show that subjective content like dreams and imagery can be read out from brain activity using decoders trained with stimulus-induced perception (Horikawa et al., 2013; Cheng et al., 2023; Kamitani et al., 2025).
These competing perspectives create central tension: how can system designed for hierarchical abstraction, which supposedly discards details, simultaneously preserve the fine-graines information required for downstream functions? Both theoretical and empirical work has shown that detailed information remains recoverable even from higher-level representations with large receptive fields (Zhang Sejnowski, 1999; Majima et al., 2017). In deep neural networks, fine-grained visual details can be reconstructed from upper layers (Mahendran & Vedaldi, 2015). These findings confirm that abstraction and detail retention are not mutually exclusive, suggesting that population codes can support both simultaneously.
To formally reconcile these observations, we introduce readout representation, a framework that operationalizes insights from informational and teleological theories. We define representation not by its causal origin, but as the set all neural features from which a specific signal can be functionally recovered (Figure IA). This approach provides concrete, quantifiable method for

---

Published as a conference paper at ICLR 2026
A
Input space X
Feature space H
Readout representations
Embedding
Size
d
B
Original
Recovery
d
dn 0.2
dи = 0.4
dl = 0.6
dи = 0.8
dn 1.0
Figure 1: Concept readout representation and images reconstructed from perturbed features. A: Concept of readout representation. Traditionally. neural representation is defined by causal relationship-by the input that elicits representation. In contrast, we propose to define representa- tions by the information recoverable from neural features. Under this framework, the representation of a single input can be a region in feature space, consisting of all points from which the input infor- mation can be read From left to right, the original image, the image recovered from the original feature, and the images recovered from perturbed features. Above the recovered images, dæ indicates the correlation dis- tance between the original feature and the perturbed features. Input images can be recovered from heavily perturbed features, showing that an input image is represented by a broad region represen- tational space, supporting the readout view. We provide full results in Appendix E and Appendix H.
B: Images recovered from perturbed features in conv3 layer of VGG19.
analyzing how information is preserved throughout system's hierarchy, moving beyond abstract philosophical distinctions. It offers a practical lens for studying how both biological and artificial systems learn abstract features while maintaining robust, information-rich codes.
To explore our framework's 's implications, we used deep neural networks as a fully observable testbed. Our investigation revealed striking phenomenon: single input corresponds to vast and continuous region of feature space from which it can be recovered. To systematically charac- terize this discovery, probed the boundaries of these readout representations by applying feature inversion (Mahendran & Vedaldi, 2015) to deliberately perturbed features. We found these represen- tational sets to be surprisingly large across both vision and language models, with inputs remaining recoverable even from features displaced far from their canonical values (Figure IB). To quantify this property, we introduce representation size, metric that captures this robustness and correlates with representational redundancy and model performance. Experiments with simplified models fur- ther confirm that this is general principle neural representation, not artifact of deep networks.
Our key contributions are follows:
Conceptual framework: We introduce readout representation, a framework that defines representation by functional recoverability rather than causal origin, resolving the tension between hierarchical processing and information preservation.
Empirical validation: We demonstrate across diverse models that input information is recoverable from wide range of perturbed features, establishing the generality our framework.
2

---

Published as a conference paper at ICLR 2026
Quantitative measure: We propose representation size as novel metric that captures the extent of this recoverable feature space, linking it to representational redundancy and model robustness.
2
RELATED WORK
Philosophical theories of representation. Naive accounts, called crude causal theory, define men- tal representations by their causal origin, where a state represents the causal origin that produced them (Fodor, 1987). This view struggles with misrepresentation, such illusions imagery. where representational content diverges from external causes. Informational theories instead define repre- sentation by the information carried by state (Dretske, 1981), while teleosemantic accounts empha- size proper function for downstream consumers (Millikan, 1989). Our work operationalizes these non-causal perspectives by defining representation in terms of recoverability.
Hierarchical-cause views in neuroscience and machine learning. Much empirical work inter- prets neural representations through progressive abstraction of input stimuli. This has motivated concepts such as neural manifolds (Seung & Lee, 2000; DiCarlo et al., 2012; Poole et al., 2016; Sorscher et al., 2022: Chung et al., 2018), representational similarity analysis (Kriegeskorte 2008: Kornblith et al., 2019; Huh et al., 2024), and information bottleneck theory (Tishby & Za- slavsky, 2015; Shwartz-Ziv Tishby, 2017). Hierarchical receptive-field analyses in deep networks further reveal preferred features and visualization methods (Zeiler & Fergus, 2014; Simonyan et al., 2014). These approaches treat features as encodings of their causal inputs, whereas we shift focus to the information that can be read out from features irrespective of origin. Note that our aim is not to reject the hierarchical-catee causall view, but rather to reconcile the tension between hierarchical ab- straction and retention of fine-graing information, grounded in the informational and teleological account of neural representation.
al.,
Recoverability- approaches. A complementary line of work investigates representation by the information that can be recovered from neural features. Feature inversion uses gradient-based optimization to recover inputs (Mahendran & Vedaldi, 2015), and probing uses linear classifiers to quantify decodable content (Alain & Bengio, 2017). Brain decoding and reconstruction studies have shown that diverse information and perceptual experiences can be read out from neural ac- tivity (Kamitani & Tong, 2005; Miyawaki et al., 2008; Horikawa & Kamitani, 2017a; Shen et al., 2019; Findling al., 2025; Sucholutsky & Griffiths, 2023), including those decoupled from causal origin such attended stimuli (Kamitani & Tong, 2005; Horikawa & Kamitani, 2022), visual il- lusions (Cheng et al., 2023), mental imagery (Shen et al., 2019), and dreams (Horikawa et al., 2013; Horikawa & Kamitani, 2017b). Leveraging the feature inversion technique, Feather et al. (2023) investigated metamers, which are multiple distinct inputs that converge to identical represen- readout representation
tations, revealing the model's inability to distinguish them. Conversely, framework captures cases where multiple neural features represent a single input, characterizing the robustness of the model's representation. Building up on these studies, we propose readout represen- tation as a computationally tractable formalization of representation. This formulation differs from previous work by providing rigorous treatment of misrepresentation and accommodating cases in which multiple neural features represent single information. This formalization also allows us to examine the size of such representations, providing a new lens to reconcile the tension.
Redefining neural representations. Studies have proposed frameworks for redefining neural rep- resentations: by the information that drives behavior (Panzeri et al., 2017), by their informational properties (Pohl et al., 2025), or from a philosophical perspective (Churchland & Sejnowski, 1990). Panzeri et al. (2017) focus specifically on behavior, while ours incorporates arbitrary readout proce- dures not limited to behavior. Unlike these studies, our framework offers computationally tractable formalization of representation, a coherent handling of representations decoupled from causal ori- gin, such as misrepresentation and dreaming, and quantitative metric to characterize the robustness of representations.
Redundancy and robustness in neural representations. Studies have proposed metrics to char- acterize redundancy in neural representation through mutual information and partial information decomposition (PID) (Schneidman al., 2003; Williams & Beer, 2010), and through compression (Tishby & Zaslavsky, 2015; Shwartz-Ziv Ziv & Tishby, 2017). These approaches analyze redundancy and compression the dataset level, whereas our representation size quantifies redundancy the
3

---

Published as a conference paper at ICLR 2026
instance level, offering new insights into how individual inputs robustly encoded. In terms of ro- bustmess, adversarial examples demonstrate that small perturbations can drastically alter encodings (Szegedy al., 2014), and this sensitivity is attributed to the presence of features rather than arti- facts (Ilyas et al., 2019). Probabilistic generative models exemplified by VAEs (Kingma & Welling, 2014; Rezende et al., 2014) explicitly introduce variability into the representations of artificial neural networks. While these studies offer insights into how to incorporate variability into representations, our work focuses on characterizing the variability inherently present even in deterministic models trained for standard tasks.
3 READOUT REPRESENTATION
We introduce the readout representation framework, which formalizes neural representation by the information that be recovered and used for downstream functions. Consider the brain or neural network f X x E + H that maps an input stimulus x E X to neural feature h E H under brain state context
E. Let S denote a signal space of interest, and let the true signal corresponding to the input be given by reference mapping मे X + S. The signal space S can be the input itself, a compressed description, or any latent variable of interest. Let æ procedure, which extracts signal from features. We assume the spaces H and S are each endowed with equivalence relations.
S denote a readout
Under the causal view, a feature / [(x) is considered representation of the input that causally generated it. When there is stochasticity between x and h, such as in brain activity or VAEs, one typically considers the expectation EEg [x, {)]. In contrast, the readout representation treats the feature as a representation of based on the information conveys (See Appendix C for the treatment of randomness in this framework).
3.1 DEFNITION OF READOUT REPRESENTATION
Following the informational view, representation is defined by the information recoverable from the feature using a readout method æ, independently of its causal origin.
Definition (Representation). h E H represents E S = (h).
This definition contrasts with the approach adopted by the hierarchical -aasccacccl-ccuaalll view, which defines
the feature h as a representation of its causal origin x. Under this definition, multiple features may represent the same signal; we term the set of all such features readout representations.
Definition 2 (Readout representation). The set H. = {h E x(h) = $} denotes readout representations of a signal
The readout representation H depends on the choice of the readout procedure ग,, and the interpre- tation changes accordingly. For practical applications, careful consideration must be given to the permissible class of readouts according to the problem being addressed. See Appendix A for further discussion.
Given the set H we can evaluate the spread of the set H as representation size, which can be interpreted as the volume occupied by specific information within the feature space. The repre- sentation size can be instantiated in various ways, such as cardinality, geometric volume, other distance-based or probabilistic indicators.
Note that the readout representation is natural extension of the causal representation when the readout is the inverse of the encoding function.
Remark Suppose S : X and f injective, and ignore the context Define n (h) f-1 for E Im f, and define (h) to be an arbitrary element of X otherwise. Then the causal representation == f()) lies in the readout representation H Even this case, any other feature h' + h is also included in H as long as π ((h') (()).
Readout representation enables us to handle misrepresentattion as situation where the extracted signal differs from the corresponding true signal. To exclude trivial cases where all features represent the same signal, we first introduce representation capability as the ability of (f т) to distinguish signals.

---

Published as a conference paper at ICLR 2026
Definition 4 (Representation capability). (f, X) has representation capability of if and only if
22 E X E E s.t. T(01) T (x2). ((11) π f(z (xx2) = f
Definition 5 (Missepresentaation). Suppose (f, T) has representation capability of S. and feature = ==) represents = (h). Then, we say h misrepresents & if and only if <(h) & a(x).
3.2 CASE STUDIES
Readout representation offers a formal treatment of neural representation decoupled from its causal origin, which challenging accommodate within the causal view (See Appendix for detailed discussion). In the following case studies, we demonstrate how readout representation accommo- dates misrepresentation, illusion, and dreaming.
Misclassífication Let the input space X be natural scenes, the feature space H be neural features in the visual cortex, the signal space S be object categories of attended objects. and the reference mapping मे assigns the true category. Suppose a subject sees a scene x E X that contains a rope but mistakenly perceives as snake. First, we consider To H + S as the subject's 's oral report to illustrate how the framework treats misclassification. The subject's neural feature h f(x, E) represents the signal "snake" since To (h)
) "snake". In our framework, the feature h misrepresents the input, To (A) = "snake" * T②x) "rope". The framework also covers a case where the subject views the scene again under a different state g and reports "rope" as To oo(f(x, = "rope". Second, instead of oral report, consider brain decoder Ty that recovers visual information from neural features in each region (ROI). By applying na to each region, we can compare the decoded content across the hierarchical processing. The decoded content from early visual regions would be closer to "rope". whereas the one from higher regions would be closer to the categorical content "snake." By identifying the transition between regions that accurately represent the stimulus and those that misrepresent it, the framework allows us to pinpoint where misrepresentation emerges and potentially analyze the mechanism behind it. This formalization aligns with previous decod- ing studies that analyze neural representations decoupled from a causal origin (Cheng et al., 2023; Kamitani et al., 2025).
Consider a Müller-Lyer figure, where two parallel lines of equal length appear different in length. Let X be images that contain two parallel lines, d(x:) be the difference in length of two lines, and S indicate whether this difference exceeds perceptual threshold A. Let the reference mapping be (x) = = A. and the readout т, be the subject's s oral report. For ordinary images, the subject correctly classifies differences, so the representation capability is satisfied. For the illusion image x, there are no differences in length, so (x) = O. However, the subject may report n, o ff f(x E) = 1.
Illusion.
Within our framework, the feature h f (r, 8) misrepresents the length difference, as To (h)) 1 K
T (2) = 0.
Dreaming. During sleep, neural activity can generate representations without external input. Let X be visual stimuli and 5 == X. Let To be the subject's is oral report. Under wakefulness (Ew), the subject can describe stimuli through oral reports, establishing representation capability, Under sleep (E.), if the subject dreams of dog, then To f(x,E. represents content decoupled from the environment. Similarly, consider a brain decoder Ra trained on wakeful activity as in (Horikawa & Kamitani, 2017b). The decoder may output "dog" when applied to the sleep activity, again producing a valid representation even without external input. The framework thus treats dream content as part of the representational space defined by readout, integrating it into the same formal structure that applies to waking perception.
"dog". Here the feature h = f(z, {.)
4 EXPERIMENTS
We examine representations of deep neural networks through the lens of readout representation. Specifically, we demonstrate that, contrary to the idea that information is progressively discarded, crucial input details are robustly preserved across broad range of hierarchical features. First, we show that information about an input stimulus often represented in a broad region of the feature space across multiple models, modalities, and layers. We show this by demonstrating that the input information is recoverable from heavily perturbed features. Second, we show the potential of the representation size metric analyze representational properties of different input stimuli, model
5

---

Published as a conference paper at ICLR 2026
architecture, and model size. Finally, we experiment with simplified model to highlight the role of redundancy in the robust recovery from perturbed features.
The following experimental settings were applied across experiments unless otherwise specified. We provide details in Appendix D.
Feature inversion. We instantiate the readout x with feature inversion because it makes minimal structural assumptions about π and searches the preimage of f directly. If input-level information is present in the feature, inversion will retrieve it without relying on task-specific decoders. Specifi- cally, given a target feature h. we define (h) = arg min, L L(f(x), h)) where L is a feature-matching loss. For vision models, we used Deep Image Prior (DIP) (Ulyanov et al., 2020) as a prior to reduce high-frequencyy artifacts while avoiding external pretrained generative priors. We also examined the robustness of our findings without DIP in Appendix G. For language models, we optimized token logits, which are converted to embeddings and fed into the model while iteratively minimizing the distance to the target features.
Distance and accuracy measures. We used correlation distance' as a distance function in the feature space dæ across both modalities, and as a distance function the input space dx for the vision modality because it yields a unit-free, dimeensioniindepedeen scale that enables comparisons across layers and modalities. For the language modality, we used token error rate as a distance measure in the input space dx, defined as the proportion of tokens in the reconstructed text that differ from the original text. We verified the robustness to the choice of dx in the vision modality by repeating evaluations with perceptual pixel-space metrics (SSIM (Wang et al., 2004), PSNR, LPIPS (Zhang et al., 2018), and DISTS (Ding et al., 2022)) (Appendix E).
Readout representation. To account for minor deviations due to numerical inaccuracies in the opti- mization process, we used a relaxed version of readout representations parameterized by a threshold t 2 O input-space distances: H (h H Vx' E r(h), dx (x, < f). The thresh- old t was chosen as sufficiently small value specific to the modality and distance measure: 0.1 correlation distance in the vision modality, and 0.3 token error rate in the language modality. We instantiate the representation size by the maximum feature-space deviation within the relaxed set:
f3 max (h. f()) h E H
Feature perturbation. As a feature perturbation, we added Gaussian noise to the original feature across 10 noise levels, calibrated to produce specific correlation distances from the original feature.
4.1 FEATURE INVERSION FROM PERTURBED FEATURES
We first examined how broadly an input is represented in the feature space of deep neural networks by reconstructing inputs from perturbed features. We conducted experiments across both the vision and language modalities. For the vision modality, we used VGG19 (Simonyan & Zisserman, 2015), CLIP (Radford et al., 2021), and DINOv2 (Oquab et al., 2024), which span both convolutional neural networks (CNNs) and Vision Transformers (ViTs). We also performed experiments using a variational autoencoder and report results in Appendix E. For language, 2019), GPT2-series (Radford et al., 2019), and OPT-series (Zhang et al., 2022). In both modalities, performed reconstruction using 64 samples: randomly sampled images from ImageNet (Deng et al., 2009) and texts from C4 (Raffel et al., 2020). Note that our aim is to illustrate how multiple features can represent single input, rather than benchmark model performance. We examined features in multiple layers for each model: all convolutional layers in VGG19 and every quarter- depth transformer block in Transformer models.
used BERT (Devlin et al.,
We found that the original images can be reliably reconstructed even from significantly perturbed features in lower to middle layers in multiple vision models (Figure IB). For example, in the lower to middle convolutional layers of VGG19, near-perfect recovery was observed at feature correlation distances up 0.8. Quantitatively, in VGG19 lower layers, features perturbed up 0.7 correlation distance still yielded reconstructions within 0.1 correlation distance in pixel space (Figure 2). Simi- larly, high-fidelity recovery from perturbed features was also observed in the lower to middle layer features in DINOv2 and CLIP models. Full results, including reconstructed images from all models, are provided in Appendix E. We also evaluated the robustness of our findings against the choice of
Athhugh correlation distance (defined as 1 correlation coefficient) does not lead to proper distance metric, it suffices for our purpose of ranking similarity consistently across heterogenenus spaces.

---

Published as a conference paper at ICLR 2026
VGG19
DINOv2-giant
CLIP-large
1.0
Layer depth
0.8
0%
0.6
25%
0.4
50%
75%
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4 0.6 0.8
1.0
HH
Hp
HH
Figure 2: Images can be accurately recovered from heavily perturbed features. The x-axis shows the correlation distance between the original and perturbed features, and the y-axis shows the corre- lation distance between the original and reconstructed images. Each line represents quarter-depth layer, colored by layer depth. Shaded areas indicate +11 SD across images: for each distance bin, we computed the standard deviation across images. Left: In VGG19, lower to middle layers retained high-fidelity information (within 0.1 correlation distance) even under 0.7 correlation distance per- turbation. Middle and Right: Similar patterns were observed in lower to middle layers of DINOv2 and lower layers of CLIP. Full results are provided Appendix E.
the prior in the feature inversion. While the use of DIP improved reconstruction quality, observed similar qualitative trends even without DIP (Appendix G). This indicates that the extensive readout representations
artifact of the prior, but rather an inherent property of the learned features. We also confirmed that the same quantitative trends were observed when using other measures such as SSIM, PSNR, LPIPS, and DISTS for the input space distance (Appendix E).
A similar pattern was observed in some of the language models (Figure 3). Across models, input tokens were recovered from perturbed features with accuracy significantly above the chance level. In particular, BERT and OPT-350m showed near-perfect recovery even at high perturbation levels (up to 0.7 correlation distance) lower layers. GPT2 and some OPT variants exhibited reduced performance, but still exceeded chance at certain perturbation levels. Note that, given the large vocabulary sizes (BERT: 30,522; GPT2: 50,257; OPT-350m: 50,272), random guessing yields near- zero accuracy. Additional results, including comparisons across model sizes, recovered sentences, and analyses of model differences, are provided in Appendix H.
Together, these results illustrate that the representations of identical or nearly identical stimuli of- ten extend to a broad region the feature space, a property observed across multiple architectures. modalities, and layers. Such extensive representational coverage suggests inherent redundancy in neural representations, which enables neural networks to simultancously retain fine-grained infor- mation and achieve hierarchical abstraction.
4.2 APPLICATION OF REPRESENTATION SIZE TO INSTANCE-LEVEL ANALYSIS
Next, we examined the utility of representation size for analyzing the neural representations of differ- ent input instances. As case study, we compared representation sizes across different image types using VGG19, focusing on their relationship with model performance. First, we compared images that were correctly classified by the model ("hit") and those that were misclassified ("miss"). We collected 8 hit images and 8 miss images from the ImageNet validation set, where hit images were defined by correct top-1 prediction and miss images were defined by failure to include the correct label in the top-5 predictions. Hit images consistently exhibited larger representation sizes, particu- larly in higher layers (Figure 4, left). Second, we compared the representation sizes between natural images and noise images generated by sampling pixel values from uniform distribution. Noise im- ages exhibited zero representation sizes across all layers, in contrast to the broader representations seen for natural inputs (Figure 4, right).
7

---

Published as a conference paper at ICLR 2026
BERT
GPT2
OPT-350m
1.0
Layer depth
0.8
0%
0.6
25%
0.4
50%
75%
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6 0.8
1.0
HH
Hp
HH
Figure 3: Input texts can often be recovered with high accuracy from perturbed features. The x-axis shows the correlation distance between the original and perturbed features, and the y-axis shows the token error rate. Each line shows results for features from quarter-depth layer, colored by layer depth. Shaded regions indicate +11 SD) across samples. Left: Similar the vision modality, the input tokens are recovered with high accuracy even from heavily perturbed features in lower to middle layers. Middle: GPT2, we do not observe high-fidelity recovery from perturbed features, although the accuracy was substantially above chance level across most feature distances. Right: Lower to middle layers of OPT-350m and BERT show extended readout representations. We provide further results in Appendix H.
These results suggest that the representation sizes are related to the nature of the instance and whether the model is able to represent it effectively, and reflect the model's perceptual reliability. We performed an additional analysis on VGG19 with randomly initialized weights to examine if these differences representation sizes were attributable to the distribution training data (Appendix F). We found similar differences in the VGGI9 with random weights, revealing that the differences in representation sizes reflect the bias of the model architecture towards certain types of images. This highlights the potential effectiveness of the representation size for analyzing model representations for specific instances. This analysis differs fundamentally from previous methods that analyze a set of input instances, such manifold analysis (Seung & Lee, 2000), representational similarity analysis (RSA) (Kriegeskorte et al., 2008), or quantifying the amount of information such as mutual information, representation size enables the analysis of single instance.
4.3 INTERPRETATION OF READOUT REPRESENTATION
What factors underlie the extended readout representations? We examined the relationship between representation sizes and feature dimensionality, and used simplified toy model to illustrate how high-dimensional mappings naturally yield redundant, robust representations that support extended readout representations.
First, we examined the relationship between feature dimensionality and representation size across layers of various vision models (Figure 5). Overall, we observed a trend that layers with higher feature dimensionality exhibit larger representation sizes, although the trend weakened in the higher layers. This result suggests that redundancy enabled by high-dimensional representations is a key factor in representation size, while deeper layers may trade this redundancy for compactness and task-specific abstraction. The relationship between feature dimensionality and representation size points to connections with prior work on intrinsic dimensionality. These studies have demonstrated that natural data (Pope et al., 2021) and parameter optimization trajectories (Aghajanyan et al., 2021) possess intrinsic dimensionality much lower than their ambient dimensionality, suggesting that multiple features may represent the same information when the ambient dimensionality is high. The smaller size of representations in higher layers may be influenced by the characteristies of the labeled data and the objective function used in model training. This motivates the toy model analysis, which isolates the role of redundancy under controlled conditions.
Second, we examined the readout representation of a simplified toy model to illustrate the emergence of extended readout representations. The model consisted of 100 neurons with bell-shaped tuning
8

---

Published as a conference paper at ICLR 2026
Hit and miss images
Natural and noise images
Representation size
1.0
Hit Miss
Natural
0.8
Noise
0.6
0.4
0
0.0
conv1_1
conv2_1
conv3 1
conv4_ 1
conv5 1
convl 1_
conv2_1
conv3 1
conv4 1
conv5 1
Layer
Layer
Figure 4: Comparison representation sizes across different image types using VGG19. Each bar shows the mean representation sizes across images for each image type, and error bars show +1 SD across samples. Left: Correctly classified images ("hit") exhibit larger representation sizes than misclassified ones ("miss"), particularly in deeper layers. This suggests that successful classification associated with more redundant and robust representations. Right: Comparison between natural and noise images. Noise images show additional analysis and found that these differences in sizes reflect the bias of the model architecture towards certain types of images (Appendix F).
representation sizes across layers. We performed an
Representation size and feature dimension
Representation size
1.00
Model
Layer depth
VGG19
0%
0.75
DINOv2-base
25%
0.50
DINOv2-large
50%
DINOv2-giant
75%
0.25
CLIP-base
0.00
CLIP-large
10
106
Feature dimension
Figure 5: Relationship between representation size and feature dimension across models and lay- ers. Each point represents a layer from each model, plotted by its feature dimensionality (x-axis, log scale) and mean representation sizes across images (y-axis). Color indicates relative layer depth, and marker shape denotes the model. We observed trend that larger dimensionality leads to larger rep- resentation sizes, suggesting that redundancy enables robust representations that support extended readout representations.
curves distributed evenly across one-dimensional input space X = [0, 1) (Figure 6, top left). Each stimulus & E X was projected onto a higher-dimensional neural feature space h = ʃ (xr). Since the dimensionality of the feature space (100) is higher than that of the input space (1), there is redundancy in representation, and the features form a low-dimensional manifold (Figure 6, bottom left). This redundancy would allow robust recovery: the original input would be recovered from perturbed features. To test this prediction, we perturbed the feature vectors by adding noise and attempted to recover the original inputs by retrieving the closest features to the perturbed points. In the principal component (PC) space, projected the points from which we could read out the original imput (Figure 6, right). Projected points were distributed in a broad region of the PC space, showing that multiple features represent single input. Although this toy model is simplified, similar mechanism would underlie readout representations in deep neural networks, given the low intrinsic dimensionality of input spaces relative their feature spaces (Pope al., 2021).

---

Published as a conference paper at ICLR 2026
Tuning curves
Readout representation
Response
Manifold Readout
Stimulus
Embedding
Manifold
PC2
PC2
PC1
PC1
Figure 6: Readout representations toy model with bell-shaped tuning curves. Top left: Tuning
curves of 10 neurons out of 100 neurons, each selectively responding to different locations in a one- dimensional stimulus space. Bottom left: The resulting neural manifold, visuzlized by projecting the neural features onto the first two principal components. Right: The readout representations of single input (a dark gray dot) visualized by magenta points. Each readout representation point shows perturbed feature from which the original input was successfully read out, showing that multiple features represent a single input.
DISCUSSION
Our study introduced readout representation, a framework that defines representations by the infor- mation recoverable from features rather than their causal origin. This perspective helps resolve the tension between hierarchical abstraction and detail retention: while abstraction emphasizes categor- ical task-relevant features, our results show that fine-grained input information remains recover- able across broad regions of feature space. Through experiments in both vision and language mod- els, we demonstrated that inputs can be reconstructed from heavily perturbed features, indicating that representations are not single points but extended sets. To quantify this property, we proposed representation size, which measures the extent of regions that support accurate recovery and links redundancy to robustness and performance. The toy model demonstrates how high-dimensional embeddings produce extended readout representations, providing a mechanistic explanation for the empirical trends. Together, these findings show that neural representation leverages redundancy to achieve both hierarchical abstraction and robust retention input details, challenging the notion that abstraction necessarily entails information loss. In relation to our original motivation: (i) we quantified recoverability via representation size, (ii) we linked it to performance and redundancy, and (iii) simplified model accounted for the possible mechanism on how representations can retain recoverable detail while supporting hierarchical abstraction.
This work offers several future directions. First, the representation size can be tool for analyzing neural representations in deep neural networks. The representation size offers a per-sample analysis of representational redundancy and robustness, complementing existing methods that typically ana- lyze representations at the dataset level (Schneidman et al., 2003; Denil et al., 2013; Feather et al., 2023: Kriegeskorte al., 2008; Kornblith et al., 2019; Huh al., 2024). Our experiments show that representation size is related to the accuracy of a model, indicating the potential for diagnosing model failures or understanding why particular examples are difficult for a network. For example, in a machine learning setting, one could compute the representation size of a specific input that exhibits unexpected behavior and compare with those of typical inputs: smaller sizes could serve as signals for outlier detection, confidence estimation, targeted debugging. It can also be used to test hypotheses about neural representation; one example is to test whether the representation size reflects the probability of sample under prior distribution implicitly induced by the training data. Another direction is to apply the readout representation to biological neural networks as a measure of redundancy and robustness of neural representations, although we note that the method of using perturbed features may not be directly applicable to the brain. Additionally, brain decoding studies often predict deep network features from noisy brain activity (Shen al., 2019; Cheng al., 2023), and their success may be explained by the large representation sizes of the networks employed. Building models with even larger representation sizes could improve decoding accuracy.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
Experimental details, including model configurations and hyperparameters, are provided in Ap- pendix D. For experiments involving stochasticity, random seeds were fixed to ensure reproducible outcomes. Source code is provided in the supplementary material.
ACKNOWLEDGMENTS
We thank our laboratory members for their valuable feedback. This work was supported by Japan Society for the Promotion of Science (JSPS: KAKENHI grants JP2SH00450 Y.K., and JP21K17821 to Y.N.), Japan Science and Technology Agency (JST: CREST grants JPMJCR22P3 to Y.K.), and Guardian Robot Project, RIKEN.
REFERENCES
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the ef- fectiveness of language model fine-tuning. Anmual Meeting of the Association for Computa- tional Linguistics and International Joint Conference on Natural Language Processing, 2021.
https: htt//iiorrrgo /ttp//oi./gg...///////oo//////// wwl.1165////// a01-ii....cloonn 568.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. International Conference on Learning Reprexentations Workshop Track, 2017. https:
/ //openzeview.... net wwt.:neetttooeeeeeee forum?ion HJJ--AJ4H
Fan L. Cheng, Tomoyasu Horikawa, Kei Majima, Misato Tanaka, Mohamed Abdelhack, Shuntaro C. Aoki, Jin Hirano, and Yukiyasu Kamitani. Reconstructing visual illusory experiences from hu- man brain activity. Science Advances, 944))caad33966, 2023. aatpp:://eee https///dcc..ogg////e wtt.oii.orrr...1111111111 1126/ sci adv. ad j3906.
Sue Chung, Daniel D. Lee, and Haim Sompolinsky. Classification and geometry of general per- ceptual manifolds. Physical Review X, 8(3):031003, 2018. https:/ /tw.i aeci.org/11e 100 /otg//...000000 110//03/
PhysRevX. 8. 8.0.0000.....003.....
Patricia Smith Churchland and Terrence J. Sejnowski. Neural representation and neural com- putation. Philosophical Perspectives, 4:343-382 1990. 2214198.
2307/
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition, 2009. wwtph:tttt.........t.m / /oi..ii../////// www.or................... http:/10.119/////// 2011199///.200000
5206848.
Misha Denil, Babak Shakibi, Laurent Dinh, MarcAurelio Ranzato, and Nando de Freitas. Pre- dicting parameters in deep learning. Advances Neural Information Processing Systems, 2013.
neurips www.sps.cccpppppeee..
Jacob Devlin, Ming-Wet Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, 1, 2019. 9-14..
https: /doii org/ http:/....111..1111 18653/vv11lv
James DiCarlo, Davide Zoccolan, and Nicole C. Rust. How does the brain solve visual object recognition? Neuron, 73(3):415-334 2012. htttps://ool 101/1..0016/1111 neuror.
0012.01.011
Keyan Ding, Kede Ma. Shiqi Wang, and Ecro Simoncelli. Image quality assessment: Unifying structure and texture similarity. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(5):2567-2581, 2022. https: htss://doo/ /doi. org/ 10. 10.1109/TPAMM. 2020 3045810 O.
Fred I. Dretske. Knowledge and the Flow of Information. MIT Press, 1981, https: /oo.. .rrg/ 10. 2002 84422
11

---

Published as a conference paper at ICLR 2026
Jenelle Feather, Guillaume Leclere, Aleksander Mądry, and Josh H. McDermott. Model metamers reveal divergent invariances between biological and artificial neural net- works. Nature Neuroscience, 26(11):0017-2044, 2023. actps:/ papps://dii. wee.i.org/100 1ttp://....../111111111101111110 0115533023-0142222
Charles Findling, Félix Hubert, Luigi Acerbi, Brandon Benson, Julius Benson, Daniel Birman, Nic- colò Bonacchi, E. Kelly Buchanan, Sebastian Bruijns. Matteo Carandini, Joana A. Catarino, Gaelle A. Chapuis, Anne K. Churchland, Yang Dan, Felicia Davatolhagh, Eric E. J. DeWitt, Tatiana A. Engel, Michele Fabbri, Mayo A. Faulkner, Ila Rani Fiete, Laura Freitas-Silva, Berk Gerçek, Kenneth Harris, Michael Häusser, Sonja B. Hofer, Fei Hu, Julia M. Huntenburg, Anup Khanal, Chris Krasniak, Christopher Langdon, Christopher A. Langfield, Peter E. Latham, Pet- rina Y. P. Lau, Zach Mainen, Guido Meijer, Nathaniel Miska, Thomas D. Mrsic-Flogel, Jean-Paul Noel, Kai Nylund, Alejandro Pan- Pan-Vazquez, Liam Paninski, Jonathan Pillow, Cyrille Rossant, Noam Roth, Rylan Schaeffer, Michael Schartner, Yanliang Shi, Karolina Z. Socha, Nicholas A. Steinmetz, Karel Svoboda, Charline Tessereau, Anne E. Urai, Miles J. Wells, Steven Jon West, Matthew R. Whiteway, Olivier Winter, Ilana B. Witten, Anthony Zador, Yizi Zhang, Peter Dayan, and Alexandre Pouget. Brain-wide representations of prior information in mouse mecision-making Nature, 645(80799 2025. s41586 025 09226-1.
Jerry A. Fodor. Psychosemantics: The Problem of Meaning in the Philosophy of Mind. MIT Press, 1987. https:///oi. org/10....0000.0000o 7551 (1551/mii wwwiiitressseeee 001. 0001.
Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications, 2017,,003,,0077. https: http///d..oooo/ org/
10.103 1038/11///0 ncomms 15037.
Tomoyasu Horikawa and Yukiyasu Kamitani. Hierarchical neural representation dreamed ob- jects revealed by brain decoding with deep neural network features. Frontiers in Computational Neuroscience, 11, 2017b. https: / /dii..
wat.i.orr/.......oo...oooo....ee org/10..g/..ooo 3389/ fncom. 2017 00004.
Tomoyasu Horikawa and Yukiyasu Kamitani. Attention modulates neural representation to render reconstructions according to subjective appearance. Communications Biology, 5(1):34, 2022.
https://dol: wrgg/oogo/o.oooooooo 1038 /22000/ 021-02975-5
Tomoyasu Horikawa, Masako Tamaki, Yoichi Miyawaki, and Yukiyasu Kamitani. Neural decoding of visual imagery during sleep. Science, 340(6112) 2013. https://don org/10. 1126/ 226/science 1234330.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The platonic repre- sentation hypothesis. International Conference Machine Learning, 2024. https: /tfp:://do:: /ooi. org/10. 1...008855//ax.... 2405. 07987.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances in Neural Information Processing Systems, 2019. https: //tp:s:///pprs.a/e nips. ce htpp/ppprrfllll files/iieeeeeee www.ffeeeseeeeeeeeaeeea p2p91/2019/ 9b371514 stt html.
Yukiyasu Kamitani and Frank Tong. Decoding the visual and subjective contents of the human brain. Nature Neuroscience, 8(5):679-88) 85, 2005. https://dols 1ttp:/000...0//01111133333. 10t.//0031nn1443111
Yukiyasu Kamitani, Misato Tanaka, and Ken Shirakawa. Visual image reconstruction from brain activity via latent representation. Annual Review of Vision Science, 11:611-63 1344, 2025. https: 11 002004233300 023616.
//doi. org/ 10. 6/ www./aanuuvvvviaeeee
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. International Conference on Learning Representations, 2014.
http://.1...1/////11 1.3550/arXiv.11 6114.
Simon Korbblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neu-
ral network representations revisited. International Conference on Machine Learning, 2019.
https: ttttps://proceee /ttp///oceeeieoeoooooo ngs ml w.t.pprss/veeee /otnoo 1th1 9a. html.
12

---

Published as a conference paper at ICLR 2026
Nikolaus Kriegeskorte, Marieke Mur, and Peter Bandettini. connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2:4, 2008
Representational similarity analysis
https: /doi:////oiio /twgg//..... /1.. 3wt////3r//...//////////// www.eerrr......................e 004. 22..00042200. 2008.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. IEEE Conference Computer Vision and Pattern Recognition, 2015. https://doi.
org/ /10 1109/CVER. 2015. 201...2991..1..... 7299155.
Kei Majima, Paul Sukhanov, Tomoyasu Horikawa, and Yukiyasu Kamitani. Position information en- coded by population activity in hierarchical visual areas. eNeuro, 42))
2017. https: / //wii org/ 10. 1523/eneuro 0268026888222002 1 6 2017.
Ruth Garrett Millikan. Biosemantics. The Journal of Philosophy, 86(6):281-297 1989. https:
//doi. org/10 10. 200.2222222222
Yoichi Miyawaki, Hajime Uchida, Okito Yamashita, Masa-aki Sato, Yusuke Morito, Hiroki C. Tan- abe, Norihiro Sadato, and Yukiyasu Kamitani. Visual image reconstruction from human brain activity using combination of multiscale local image decoders. Neuron, 60(5):915-922 2008.
http:://ooooooogg/// 1016 6/ www.on. 2008 11.11..0000 11.004.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nico- las Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve. Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without super- vision. Transactions Machine Learning Research, 2024. https: http:///ppee///eeeeeeeeeeeeeeeee net. forum? aerrm?id=a68aae 68SUt 6zFt.
Stefano Panzeri, Christopher D. Harvey, Eugenio Piasini, Peter E. Latham, and Tommaso Fellin. Cracking the neural code for sensory perception by combining statistics, intervention, and behav- ior. Neuron, 93(3):991-507, 2017. https: http://o..ooooo/. 1016/1 www.nn.u....... 2.16.12. 036.
Stephan Pohl, Edgar Y. Walker, David L. Barack, Jennifer Lee, Rachel N. Denison, Ned Block, Florent Meyniel, and Wei Ji Ma. Clarifying the conceptual dimensions of representation in neu- roscience. arXiv preprint, 2025, https://dde //ooi. org/ /0/g////..0 48550/ arxiv 2..22.43.440
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponen- tial expressivity deep neural networks through transient chaos. Advances Neural Information Processing Systems, 2016. https: /ttp://papeeaaeee nips. ce aecppape__fil hellle/pppr//0000 hash/1485100 0034881110111100 34 24e44422550004445500500 433bb002e44443 weweef--bstrrtt..... .ttttracctttml.......
Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension images and its impact on learning. International Conference on Learning Repre- sentations, 2021. https: /pppnreviww.t.e wtt.ieewweeeeeee..eee fottt//ormmmiddxk11... 9x233x222.19.9.....
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. OpenAl blog, 2019. https: / oreaaiopenai. wtt com/beeteer--agggaeeee haapaaaeemmdelss//nnnnn_eeemmm Language_models _re unsupervisedmmlliiakk..
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, 2021. radford21a, html.
/
https: ipwpceediiggs...oo mlr. wtt....rres//v////
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou. Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. https: /jmrr. org brgg/papars/221/22 074 html.
13

---

Published as a conference paper at ICLR 2026
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. International Conference on Machine Learning, 2014. https: /pppooeeeeeinggsseeeee mlr wwwpspeeeseeee 2tt/3//eeeeeeeeee html.
Elad Schneidman, William Bialek, and Michael J. Berry. Synergy, redundancy, and independence in population codes. Journal Neuroscience, 23(37):11539-11 2003. https://dot org/ 10. 3
23-37-115539
H. Sebastian Seung and Daniel D. Lee. The manifold ways of perception. Science, 290(5500):
2268-2228 2269, 2000. https: /doii wtg//c...... /iiee//////eeeeeeeeeciaicaaeeeee wwwicce..99... 5500.2266
Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image recon- struction from human brain activity. PLOS Computational Biology. 15(()xe1006633, 2019.
jouraaooooaa.aaaaa http:////..////o.aaaooooo.. pebi. ctcci.100662.. 1006633.
Ken Shirakawa, Yoshihiro Nagano, Misato Tanaka, Shuntaro C. Aoki, Yusuke Muraki, Kei Majima, and Yukiyasu Kamitani. Spurious reconstruction from brain activity. Neural Networks, 190: 107515, 2025.
1016/
6/ nww.et. 2025.107515.
Ravid Shwartz-Zir Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint, 2017. https: //oiiiii.o// org/ 48550/ aarai.iiiii....... 1703. 00810.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. International Conference on Learning Representations, 2015. attps://don org/10. 44555//a2ii..111.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. International Conference Learning Representations, Workshop Track, 2014. https:/ /ooi. /ttp:/....o......55555..../a0.1. 6034.
Ben Sorscher, Surya Ganguli, and Haim Sompolinsky. Neural representational geometry under- lies few-shot concept learning. Proceedings of the National Academy of Sciences, 119(43): e2200800119, 2022.
3/ppss///////// 2200800119.
Ilia Sucholutsky and Tom Griffiths. Alignment with human representations supports bust few-shot learning. Advances in Neural Information Processing Systems, 2023. neurips..... A htmll.
edddc03b001dcccbb4bb at
Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Good- fellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations, 2014. https: htttps://oi.... htt:///oi..oo org 48550/arxiv.... 1312. 6199.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. IEEE Information Theory Workshop, 2015. https: :/oo:i/////o../ /t0/1 1009 2015. 7133169.
Dmitry Ulyanov, Andrea Vedaldi, and Vietor Lempitsky. Deep image prior. International Journal of Computer Vision, 12877):1877-1888 2020. ptttps://dos http://oc..org/1.1000000//
s11263- 022000000--
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: From error
visibility structural similarity. IEEE Transactions Image Processing, 13(4):600-612, 2004.
https:://of .or.orgg//. / 109 20/0///TPP.P TTP. PPTP.2003. 81 9861.
Paul L. Williams and Randall D. Beer. Nonnegative decomposition of multivariate information.
arXiv preprint, 2010. https: /tops://doit hww. orgg 485.448555////
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional net-
works. European Conference on Computer Vision, 2014.
14

---

Published as a conference paper at ICLR 2026
Kechen Zhang and Terrence J. Sejnowski. Neuronal tuning: To sharpen broaden? Neural Com- putation, 11(1):75-88) 84, 1999. https: /ww.i/oiii.//// org/10 w1w.111261/. 111220099669300000
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. https: /tt////diii/// org/ http:/0..1111////11 C 2018 00068.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettle- moyer. OPT: Open pre-trained transformer language models. arXiv preprint, 2022. https: //doi. 48550 225550/ar200 aariiv 2205. 01068.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
We provide additional information of our study. Appendix A describes the limitations of our work. Appendix B describes the relationship between the causal view and misrepresentation. Appendix C discusses the treatment of noise in our framework. Appendix describes the details of the exper- iment including procedures for feature inversion, feature perturbation, optimization settings, and model specifications. Appendix provides additional results for vision models, including recon- structed images and quantitative results. Appendix F provides additional results for the application of representation size to instance-level analysis. Appendix G provides the ablation results with- Deep Image Prior (DIP) in vision experiments. Appendix H reports the additional results for language models, including reconstructed sentences, quantitative evaluations, and insights into the difference between models. Appendix provides the broader impact of our study. Appendix J describes the usage of LLMs in our study.
A LIMITATIONS
Despite our contributions, several limitations remain. First, since the readout representation de- pends on the choice of readout procedure и, both the selection of and the interpretation of the results require careful consideration. Specifically, if a strong prior, such as diffusion models, is used as part of the readout procedure, it can unfaithfully inflate the recoverable information, as pointed out in Shirakawa et al. (2025). In our experiments, we used Deep Image Prior (DIP) as prior the vision experiments, and we did not use any prior in language experiments. We choose DIP because it is weak, data-independent prior. Strong generative priors such GANs diffusion models could yield even smoother reconstructions, but we intentionally avoided them because they impose powerful biases from the training data distribution of these generative models. By contrast, DIP is widely regarded as a substantially weaker prior: it does not require training and therefore not biased by specific training data. Importantly, images remain visibly similar to the original inputs under substantial perturbations of the features. Although we observed relatively sharp increase in the input distance the small feature perturba- tions, it reflects high-frequenee variations rather than the loss of essential information for humans and machine learning models to perform reasonably natural downstream tasks. This behavior would not be expected if the recovery were solely an artifact introduced by DIP.
without DIP (Appendix G), the recovered
Second, the source of variation in representation size across architectures remains unclear. For ex- ample, language models such as GPT2-Laage and OPT-1.3b showed lower recovery performance under perturbation compared others (Appendix H). In vision models, we found that feature di- mensionality has a strong effect on representation size, and we also provide additional analysis of performance differences among language models in the Appendix H.3, though a comprehensive understanding remains an open question.
Third, while we propose representation size as metric for model evaluation, our experiments demonstrate only preliminary cases. Its broader utility remains to be established.
Fourth, although our framework is inspired by both artificial and biological systems, its application to biological neural networks remains open question. In our study, we used perturbed features to probe the feature space in deep neural networks, but this method is not directly applicable to the brain.
Finally, our implementation adopts feature inversion as the readout method, which introduces some computational overhead. However, our proposed framework is independent from the choice of read- out method, and users may adopt alternative procedures better suited to their needs and computa- tional budgets.
B CAUSAL VIEW OF REPRESENTATION AND MISREPRESENTATION
The causal view refers to view that defines representation follows: state h represents stimulus x if and only if h is caused by æ. Fodor (1987) named this view of representation the crude causal theory because the content of representation is derived from the causal origin
16

---

Published as a conference paper at ICLR 2026
The causal view is known to face challenges in explaining misrepresentation, where the content of representation does not align with the causal origin (Fodor, 1987). This does not mean that misrepresenttatinn is disconnected from physical causation: misrepresentation can still be causally produced by neural activity or stimuli. What the causal view fails to describe the content of representation in cases of misrepresentation. For example, consider subject who mistakenly judges it to be snake. The natural explanation is that the subject's brain activity or mental content mistakenly represents snake in the presence of rope. However, if adopt the causal view, because the neural state is caused by the rope, the state represents the rope. The causal view therefore fails characterize this case as misrepresentation.
rope but
C TREATMENT OF NOISE
In this section, we discuss the treatment of measurement noise and intrinsic noise in our framework.
We note that properly addressing measurement noise important in empirical studies. At the same time, our formalization is focused on the simplest case for clarity. Numerical experiments demon- strated that even in the case of artificial NNs, where noiseless measurement available, extended readout representations were observed. Similar to other feature characterization methods, it could be extended to handle observation noise appropriately.
Our framework has different view to the previous studies on the so-called intrinsic noise. To high- light the difference, we first describe the typical view on the intrinsic noise, and then describe ours. Typically, studies under nousal-view implicitly assume that there is single and ideal neural feature h h= f()( that does not have noise, and that variability across repeated presentations merely reflects mechanistic internal noise, such as probabilistic neuronal spikes. Under this position, the expected value with noise eliminated is regarded as the ideal representation, and the average of multiple observations from repeated presentations of the same stimulus is adopted While we agree that such a mechanistic noise does exist, in contrast, our framework characterizes the representation differently: if multiple activity patterns are observed, we treat all neural features as representations of signal as long as they allow the recovery of the signal.
the characterization.
D EXPERIMENT DETAILS
This section provides the details of our experiments, including the feature inversion (Appendix D.1), feature forwarding (Appendix D.2), feature perturbation (Appendix D.3), computational resources (Appendix D.4), model specifications (Appendix D.5), and dataset (Appendix D.6).
D.I FEATURE INVERSION
We used feature inversion as the primary readout procedure т. Feature inversion uses gradient-base optimization to find an input that produces a feature similar given target feature. We describe the problem formulation in Appendix D. and the optimization methods in Appendix D.1.2.
1.1.1 PROBLEM FORMULATIONN
In our feature inversion experiments, we aim to recover the original input from a given neural feature, referred to the target feature. Feature inversion uses gradient-based optimization to find input that minimizes the distance between the feature of the input and the target feature. Let f X + H denote a neural network that maps an input X its corresponding feature h H. Given target feature h E И ~ IRd, the feature inversion is formulated as finding a set of inputs whose features minimize a predefined loss:
(hh) arg min (()()),)). =
where L denotes loss function that quantifies the dissimilarity between the feature vector f(x) and the target feature h. The solution set π (h) contains inputs that yield features close to the target feature.
For the loss function, we used the mean squared error (MSE) in vision models:
L(Arecon: Munger)
hunget
Brecoa l1?
17

---

Published as a conference paper at ICLR 2026
and the linear combination of MSE and cosine loss in language models:
L (hrecon: Baarger)
Marget
Rrecon
cos (Auggtt Brecon)
D.1.2 METHODS
To solve the minimization problem defined in Appendix D.I.I, we adopt gradient descent optimiza- tion. Instead of directly optimizing the imput x E X. we optimize it through iteratively optimizing latent variable E Z. Specifically, we define generator function Z + X that maps the latent variable to the input space. Each step of the optimization process is as follows:
ス h -
x g(=),
where 77 is the learning rate.
Vision models. We use Deep Image Prior (DIP) (Ulyanov et al., 2020) as the generator g. Its pa- rameter 2 is optimized to produce the input image x from a fixed random inputs. The architectural bias of DIP acts structural prior, suppressing high-frequency artifacts and improving perceptual quality. We choose DIP because it imposes minimal prior assumptions about the image distribution and does not require any training data, making suitable for our analysis. Specifically, we deliber- ately avoided using pre-trained generative models as a prior, such as GANs and diffusion models, which may introduce biases from their training data and limit the generality of our findings. As an ablation, we also performed feature inversion without DIP and report results in the Appendix G.
Language models. For language models, the input is discrete token sequence (1..... VI", where V denotes the vocabulary size and T the sequence length. For the readout operation, we relax this discrete space continuous space since the gradient- based optimization in discrete space is not trivial. Instead of directly optimizing the input tokens, optimize the token logits = E RR which are then converted the continuous tokens x E R as
X
Fs g softmax(2i). for i ,.....
where the generator function applies a row-wise softmax to = and each v; represents the relaxed categorical distribution over the vocabulary position E [T]. The final token sequence after gradient-based optimization is obtained by taking the argmax over each row of the optimized logit matrix:
F2 arg max Zigs for 1....,T
In both vision and language modalities, we used the AdamW optimizer from PyTorch. Table summarizes the optimizer settings and training configurations used for feature inversion. Default PyTorch parameters were used for AdamW if not specified. In vision modality, we used linear learning rate scheduler, which decayed the learning rate to zero over the course of training.
Table I: Optimizer and training parameters used in feature inversion experiments.
Parameter
Vision
Language
Optimization Target
DIP Latent
Token Logits
Learning Rate
0.0001
0.1 10,000
Iterations
10,000
D.2 FEATURE FORWARDING
In supplementary experiment using variational autoencoder (VAE) model, we additionally adopted another readout method which we call feature forwarding. Given the neural features of the VAE, we directly pass it through the decoder module of the model to reconstruct the image from it. This approach leverages the learned generative capabilities of the VAE, providing an alternative to optimization-base based feature inversion. We provide results using this method in Appendix E.
18

---

Published as a conference paper at ICLR 2026
D.3 FEATURE PERTURBATION
To systematically evaluate the extent of readout representations, we perturbed feature vectors by adding Gaussian noise with calibrated variance, Given a feature vector E Rd, we generated perturbed feature h' = h + where (N(0,
The variance o² was analytically determined to produce target correlation distance dy (h, (0,1) between the original and perturbed features. Under high-dimensional assumptions, the expected correlation distance can be approximated as:
R
g²
Var(h)
where, Var(h) denotes the sample variance the feature vector h. Solving for o² gives:
o² = Var(h) (1 (1-1p-1)
This allowed control over perturbation magnitude. (0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.99).
We used 10 correlation distance levels:
D.4 COMPUTATIONAL RESOURCES
Our experiments flexibly scale with available resources by adjusting key parameters such as the number of noise levels and random seeds, and the number of parallel execution. As reference, minimal experiment requires approximately 10 GB of GPU memory and 20 minutes of runtime on an NVIDIA V100 GPU, which reconstructs single image from VGG19 layer with 10 noise levels, a number of parallel execution of 10, and one random seed. Full-scale experiments, including all layers, models, and repetitions,
conducted using the following hardware:
Local resources: NVIDIA Tesla V100S (32 GB), Quadro RTX 8000 (48 GB), RTX A6000 (48 GB), and A100 GPUs.
Cloud resources: AWS g5.48xlarge instances equipped with 8 NVIDIA A10G Ten- sor Core GPUs, ABCI (AI Bridging Cloud Infrastructure) rt HF nodes equipped with NVIDIA H200 SXM 141GB GPUs.
D.5 MODEL SPECIFICATIONS
We evaluated representations of deep neural network models across range of pretrained vision and language models. For the vision modality, we used VGG19, CLIP (Base Patch32, Large Patch14), DINOv2 (Base, Large, Giant), and SDXL-VAE. For the language modality, we used BERT (Base), GPT2 (Small, Medium, Large, XL), and OPT (125M, 350M, 1.3B, 2.7B). For each model, we selected representative layers for analysis: all 16 convolutional layers in VGG19, and one layer per quarter depth in other models (c.g., layers 0, 3, 6, in 12-layer model). Except for VGG19, all models weights on the source of each model below.
obtained from the HuggingFace Transformers library, and provide details
VGG19 We used the weights provided at the following URL: https://figshat com/ ndownloader/ wiles/38222 originally provided in Caffe (http:/ /www.www.oommmmmm robots ac uk/ wat//vvg//// wwwgg/ooottee sowtware/ewwrry deep/ ca /tte/VVGG_LSVVCCCCCC 9 lllaayrrrs... ca 1)m
This weight is a PeTooch-compaaiiiiii conversion of the one
Other models Table lists HuggingFace model identifiers for each model.
19

---

Published as a conference paper at ICLR 2026
Table 2: HuggingFace model identifiers for the transformer models used in experiments.
Model Family
HuggingFace Identifier
DINOv2 (ViT)
facebook/dinov--ben
facebook/dimov2-luug
facehook//dnovv2giann
CLIP (ViT)
openai/einp-vii-hss-ppthhhhhh
openaa/clip-viillagg--aachhh
SDXL-VAE (VAE)
stabilityai/ssxl-aae
BERT
google-berr/berr--ae---eee
GPT2
openai-communityyguau
openai-commmuniyygg222
openai-communnt//gg222aaaa
openai-conmmunix//pp22-111
OPT
facchookkopt-1255m
faccbookkopt-350mm
facebook/opt-1.3b
facebook/opt-2.7bb
D.6 DATASET
For the vision modality, we randomly sampled 64 natural images for our primary experiments (Fig- ure 7) from the test-split of the ImageNet dataset via HuggingFace datasets (ILSVRC/imagenee-I))
To analyze differences in representation size between correctly classified images ("hit") (Figure 8) and incorrectly classified ones ("miss") by VGG19 (Figure 9), we sampled 16 images from the ImageNet validation set total. Hit images included 8 images where the model's top-1 prediction matched the ground-truth label. Miss images included 8 images where the true label was not within the top-5 predictions. We excluded ambiguous cases to ensure label clarity.
For the experiment with noise images, generated noise images by sampling pixel values uni- formly from [0, 255] (Figure 10). We prepared four images with different random seeds for robust- ness.
For the language modality, we sampled 64 sequences from the validation split of the C4 dataset via HuggingFace Datasets (allenai/c4). Each sequence truncated to maximum of 256 tokens in order to control computational costs.
In all experiments, the dataset was curated beforehand, and post-hoc cherry-picking per- formed.
20

---

Published as a conference paper at ICLR 2026
Figure 7: 64 natural images used in the vision experiments, randomly sampled from the test-split of ImageNet dataset. The images were selected prior to analysis and used consistently across experi- ments without post-hoe cherry-picking.
Figure 8: Images that were correctly classified by VGG19 on the ImageNet validation set ("hit").
Each image was correctly classified based on the top-1 I prediction of VGG19.
21

---

Published as a conference paper at ICLR 2026
Figure 9: Images that were incorrectly classified by VGG19 on the ImageNet validation set ("miss"), "). Each image was misclassified as the correct label was not included in the top-5 predictions of VGG19.
Figure 10: Random noise images used for representation size analysis. Each image was generated by independently sampling pixel values from a uniform distribution over [0, 255] with different random seeds.
E DETAILS OF RESULTS FOR VISION MODELS
We presents additional results for the vision modality complement the main text. We provide reconstructed images, and quantitative evaluations.
E.1
RECONSTRUCTED IMAGES
This section presents reconstructed images from perturbed features for each vision model.
E.1.1
VGG19
We provide results of four representative images out of 64 samples (Figure 11 Figure 14) for space constraints.
22

---

Published as a conference paper at ICLR 2026
Feature distance dy
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.99
conv1_1 1
conv1_2 2
conv2_1 1
conv2_2
conv3_1 1
conv 2
conv3_3
conv3_4 4
conv4 1
conv4_2 2
conv4_3
conv4 4
conv5 1
conv5 2
conv5_3
conv5_4
Figure 11: Images reconstructed from perturbed VGG19 features. Each row corresponds to con- volutional layer and each column corresponds to a magnitude feature perturbation, measured by the correlation distance du between the original and perturbed features. Even under substantial per- turbations, the reconstructed images remain faithful in early to middle layers.
23

---

Published as a conference paper at ICLR 2026
Feature distance dy
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.99
convi_1 1
conv1_2 2
conv2_1 1
conv2_2
conv3_1 1
conv 2
conv3_3 3
conv3_4 4
conv4 1
conv4 2
conv4_3
conv4 4
conv5 1
conv5_2 2
conv5_3
conv5_4
Figure 12: Images reconstructed from perturbed VGG19 features. Layout follows Figure 11.
24

---

Published as a conference paper at ICLR 2026
Feature distance dy
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.99
convi_1 1
conv1_2 2
conv2_1
conv2_2
conv3_L 1
conv 2
conv3_3 3
conv3_4 4
conv4 1
conv4 2
conv4_3
conv4 4
conv5 1
conv5 2
conv5_3
conv5_4
Figure 13: Images reconstructed from perturbed VGG19 features. Layout follows Figure 11.
25

---

Published as a conference paper at ICLR 2026
Feature distance dи
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.99
convi_1 1
conv1_2 2
conv2_1 1
conv2_2
conv3_L 1
conv 2
conv3_3 3
conv3_4 4
conv4 1
conv4 2
conv4_3
conv4 4
conv5 1
conv5 2
conv5_3
conv5
Figure 14: Images reconstructed from perturbed VGG19 features. Layout follows Figure 11.
26

---

Published as a conference paper at ICLR 2026
E.1.2 DINOV2
We provide images reconstructed from perturbed features of DINOv2-giant in Figure 15. We provide reconstructions of one sample due space constraints.
Feature distance dμ
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.99
layer_0
layer_10 10
layer_20
layer_30 30
Figure 15: Images reconstructed from perturbed DINOv2 giant features. Layout follows Figure 11.
E.1 CLIP
We provide images reconstructed from perturbed features of CLIP ViT-L/14 in Figure 16. We pro- vide reconstructions of one sample image due to space constraints.
Feature distance dи
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.99
layer_0
layer_6 6
layer 12
layer_ 18
Figure 16: Images reconstructed from perturbed CLIP ViT-L/14 features. Layout follows Figure 11.
E. SDXL-VAR
We provide images reconstructed from perturbed features of SDXL-VAE in Figure 17 and Figure 18 for feature inversion and feature forwarding methods, respectively. We provide reconstructions of four sample image due to space constraints.
27

---

Published as a conference paper at ICLR 2026
Feature distance dR
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.99
Figure 17: Images reconstructed from perturbed SDXL-VAE features using feature inversion. Each row corresponds to image and each column shows reconstructions from features perturbed at increasing correlation distances dæ E {0.0, o. 0.9, 0.99).
Feature distance dR
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.99
Figure 18:
Images reconstructed from perturbed SDXL-VAE features using feature forwarding.
Figure layout follows Figure 17.
28

---

Published as a conference paper at ICLR 2026
E.2
QUANTITATIVE RESULTS
This section provides additional quantitative results for reconstruction experiments.
E.2.1
VGG19
We present quantitative results of all 64 images for all 16 convolutional layers of VGG19 in Fig- ure 19. We also provide results of perceptual metrics (SSIM, PSNR, LPIPS, DISTS) in Figure 20.
conv1 layers
conv2 layers
conv layers
1.0
0.8
0.6
0.4
100%
0.2
0.0
75%
Layer depth
0.0
0.2
0.4
0.6
0.8 1.0
0.0
0.2
0.4
0.6
0.8 1.0
0.0
0.2
0.4
0.6 0.8 1.0
HH
dH
dH
50%
conv4 layers
conv5 layers
1.0
25%
0.8
0.6
0%
0.4
0.2
0.0
0.0
0.2
0.4
0.8 dH
0.6
1.0
0.0
0.2
0.4 0.6 0.8 1.0 dH
Figure 19: Quantitative results of VGG19. Pixel correlation distance (dx) between reconstructed and original images plotted against feature correlation distance (drr between the perturbed and original features. Each subplot corresponds to a group of convolutional layers, and line colors indicate layer depth within the group.
29

---

Published as a conference paper at ICLR 2026
SSIM 1
PSNR
LPIPS 4
40
1.0
1.0
0.8
32
0.8
0.6
24
0.6
0.4
16
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
8.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
dH
dH
dH
DISTS 4
1.0
0.8
0.6
0.4
Layer depth
0.2
0%
0.0
25%
0.0
0.2
0.4
0.6
0.8
1.0
50%
dH
75%
Figure 20: Results of perceptual metries (SSIM, PSNR, LPIPS, DISTS) of VGG19. Perceptual metrics between reconstructed and original images plotted against feature correlation distance (d between the perturbed and original features. Each subplot corresponds to a metric. Quarter-depth layers are shown with colors indicating layer depth.
E.2.2 DINOv2
We present quantitative results of all 64 natural images for all three model sizes we tested (base, large, giant) in Figure 21.
DINOv2-base
DINOv2-large
DINOv2-giant
1.0
Layer depth
0.8
0%
0.6
25%
0.4
50%
75%
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
dy
dH
dy
Figure 21: Results of DINOv2 models. Pixel correlation distance (dx) between reconstructed and original images plotted against feature correlation distance (dæ) between the perturbed and original features. Each subplot corresponds model size. Quarter-depth layers are shown with colors indicating layer depth.
F.2.3 CLIP
We present quantitative results all 64 natural images for all two model sizes we tested (base, large) in Figure 22
30

---

Published as a conference paper at ICLR 2026
CLIP-base
CLiP-large
1.0
Layer depth
0.8
0%
0.6
25%
50%
0.2
75%
0.0
0.0
0.2 0.4
0.6 0.8
1.0
0.0
0.2
0.4
0.6 HH
0.8
1.0
HH
Results of CLIP models. Pixel correlation distance (dx) between reconstructed and original images plotted against feature correlation distance (dz) between the perturbed and original features. Each subplot corresponds to model size. Quarter-depth layers are shown with colors indicating layer depth.
Figure 22:
E.2.4 SDXL-VAE
We present quantitative results of all 64 natural images for SDXL-VAE using two methods, feature inversion and feature forwarding, in Figure 23.
Feature inversion
Feature forwarding
1.0
0.8
0.6
dx
0.4
0.2
0.0
0.0
0.2
0.4
0.6 0.8 dH
1.0
0.0 0.2
0.4
0.6 0.8 1.0
dH
Figure 23: Results of SDXL-VAE with feature inversion and feature forwarding. Pixel correlation
distance (dx) between reconstructed and original images plotted against feature correlation distance (dgr between the perturbed and original features. Each subplot corresponds readout method.
F ADDITIONAL RESULTS FOR APPLICATION OF REPRESENTATION SIZE
In order to examine if the observed differences in representation sizes (Section 4.2) due the effect of training, we performed an additional analysis. Specifically, we repeated the comparison of representation sizes of the same of images using VGG19 with randomly initialized weights, which we refer to as the untrained VGG19.
As result, we observed the similar differences in representation sizes in the untrained model (Fig- ure 24A). In the untrained VGG19, hit images have shown larger representation sizes than miss images, and noise images have shown zero representation sizes across layers. This similarity across trained and untrained models suggests that the observed differences in representation sizes reflect architectural biases the model, rather than solely training effects. Also, the comparison between trained and untrained VGG19 using natural images showed that the trained model exhibits larger representation sizes, particularly in middle to higher layers (Figure 24B). This demonstrates that training natural images increases the representation sizes of the corresponding image type. In summary, we found that architectual biases of the model underlie the observed differences in repre- sentation sizes, and the training expands the representation sizes for the trained image type.
31

---

Published as a conference paper at ICLR 2026
A
Hit and miss images
Natural and noise images
Representation size
1.0
Hit Miss
Natural
0.8
Noise
0.6
0.4
0.2
0
0
0
0
0.0
convl_1 1
conv2_1
conv3 1
conv4 1
conv5 _1
convl_1
conv2_1
conv3 1
conv4 1
conv5
Layer
Layer
B
Trained and untrained models
Representation size
1.0
Trained
0.8
Untrained
0.6
0.4
0.2
0.0
convl_1
conv2_1 1
conv3 1
conv4 1
conv5_ 1
Layer
Figure 24: Comparison of representation sizes of different images types using an untrained VGG19 model. Bars show the mean representation sizes across samples, with error bars indicating +1 SE. A: The hit images show larger representation sizes on untrained VGG19 as well, indicating these differences reflect architectural biases, not solely training effects. B: Comparison between trained and untrained VGG19 using natural images showed that the trained model exhibits larger sizes, particularly in middle higher layers, demonstrating the effect of training on representation sizes.
G ABLATION STUDIES ON THE EFFECT OF DIP
To exclude the possibility that our main results are solely due to the denoising capability of DIP, we performed two ablation studies to assess the effect of using Deep Image Prior (DIP) as the generator in feature inversion. First, we ablated DIP entirely by removing the generator and directly optimizing pixel values, and report results in the Appendix G.I. Second, DIP to denoise perturbed images by ablating the neural network in the feature inversion experiment, considering the pixel space itself as the feature space, and report results in the Appendix G.2. Both experiments indicates that DIP alone cannot account for the high representation sizes observed in our main results.
examined the effect of
G. RECONSTRUCTION WITHOUT DIP
We consider applying feature inversion without a generator g. Specifically, we initialize the pixel values of reconstructed image with random values, and optimize the pixel values using gradient:
レ x (()))) h).
For optimization parameters, we set the learning rate to 0.01. Other hyperparameters followed those
in Table 1.
32

---

Published as a conference paper at ICLR 2026
We present reconstructed images using pixel optimization in Figure 25 to Figure 28. We present samples out of 64 samples due to space constraints. Even without DIP, the reconstructed images retains perceptual similarity the original images under significant perturbations in early to middle layers. Specifically, these images retain the fine-grained information suffices to perform human recognition. We present quantitative results of all 64 images for all 16 convolutional layers of VGG19 in Figure 29. We observed relatively steep increase of pixel correlation distance dx against feature correlation distance dæ in small perturbation. However, these small perturbations reflects high-freque fehuerryyyncy variations rather than the loss of essential information for downstream tasks. These results indicate that the recovery of input images from perturbed features are not solely due the denoising capability of DIP.
33

---

Published as a conference paper at ICLR 2026
Feature distance dи
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.99
convl_1 1
conv1_2 2
conv2_1 1
conv2_2
conv3_1 1
conv 2
conv3_3
conv3_4 4
conv4 1
conv4_2 2
conv4_3
conv4 4
conv5 1
conv5 2
conv5_3
conv5_4 4
Figure 25: Images reconstructed from perturbed features in VGG19 using pixel optimization, ablat- ing DIP. Layout follows Figure 11.
34

---

Published as a conference paper at ICLR 2026
Feature distance dy
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.99
convi_1 1
conv1_2 2
conv2_1 1
conv2_2
conv3_1 1
conv 2
conv3_3 3
conv3_4 4
conv4 1
conv4 2
conv4_3
conv4 4
conv5 1
conv5 2
conv5_3
conv5_4 4
Figure 26: Images reconstructed from perturbed features in VGG19 using pixel optimization, ablat- ing DIP. Layout follows Figure 11.
35

---

Published as a conference paper at ICLR 2026
Feature distance dи
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.99
convi_1 1
conv1_2 2
conv2_1
conv2_2
conv3_L 1
conv 2
conv3_3 3
conv3_4 4
conv4 1
conv4 2
conv4_3
conv4 4
conv5 1
conv5 2
conv5_3
conv5_4 4
Figure 27: Images reconstructed from perturbed features in VGG19 using pixel optimization, ablat- ing DIP. Layout follows Figure 11.
36

---

Published as a conference paper at ICLR 2026
Feature distance dи
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.99
convi_1 1
conv1_2 2
conv2_1 1
conv2_2
conv3_L 1
conv 2
conv3_3 3
conv3_4 4
conv4 1
conv4 2
conv4_3
conv4 4
conv5 1
conv5 2
conv5_3
conv5_4 4
Figure 28: Images reconstructed from perturbed features in VGG19 using pixel optimization, ablat- ing DIP. Layout follows Figure 11.
37

---

Published as a conference paper at ICLR 2026
convl layers
conv2 layers
conv3 layers
1.0
0.8
0.6
0.4
100%
0.2
0.0
75%
Layer depth
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8 1.0
HH
dH
dH
50%
conv4 layers
conv5 layers
1.0
25%
0.8
0.6
0%
0.4
0.2
0.0
0.6
0.8
1.0
0.0
1.0
dH
dH
Figure 29: Quantitative results of VGGI9 with pixel optimization. Formatting follows Figure 19.
G.2 REPRESENTATION SIZE ON PIXEL SPACE
We have performed the ablation experiment of measuring the representation size in the input space with DIP. We perturb image, and then optimize the latent variable of DIP, so that the produced image matches the perturbed one. This is identical to the feature inversion experiment except that the feature space coincides with the pixel space, i.e., ablating the neural network. Optimization parameters are the same as those in Table 1. If our main results presented in the main text and Appendix were solely due to the denoising capability DIP, we would expect see similar representation sizes in this ablation experiment.
Figure 30 and Table present the results of this ablation experiment. While DIP could remove some noise from the perturbed images (Figure 30, left), the results of this ablation (Figure 30, left) are much closer to a straight line than the results of VGG19 (Figure 30, right). This demonstrates that DIP alone cannot account for the large readout representations the result of representation size in comparison to that the VGG19 layers. The pixel space has representation size of 0.32 + 0.047, which is significantly smaller than the lower and middle layers of VGG19. This gap confirms that DIP alone cannot explain the high representation size observed in those layers.
observe. In Table 3, we present
38

---

Published as a conference paper at ICLR 2026
Pixel
VGG19
1.0
Layer depth
0.8
0%
0.6
25% %
0.4
50%
0.2
75%
0.0
0.0
0.2
0.4
0.6 0.8
1.0
0.0
0.2
0.4
0.6 0.8 1.0
HH
dH
Figure 30: Quantitative results of ablation experiment measuring representation size the pixel space using DIP. Left: Results of the ablation study, representation size in the pixel space using DIP. Right: Results of the original experiment using VGG19 and DIP for comparison. Although the DIP could remove some noise from the perturbed images (left), the line is significantly steeper than those of lower and middle layers of VGG19 (right), indicating that DIP alone cannot explain the high representation size observed in those layers.
Model
Feature space
Size
Pixel
0.32 + 0.047
VGG19
convl
0.81 = 0.024
VGG19
conv2 1
0.78 + 0.046
VGG19
conv3
0.73 + 0.093
VGG19
conv4_
0.40 + 0.30
VGG19
conv5_1
0.11 = 0.17
Table 3: Representation size measured in the pixel space compared those of VGG19 layers. The pixel space has a significantly smaller representation size (top row) than the lower and middle layers of VGG19, indicating that DIP alone cannot explain the high representation size observed in those layers.
H DETAILS OF RESULTS FOR LANGUAGE MODELS
We present additional results for the language modality to complement the main text. We provide reconstructed sentences, quantitative evaluations, and insights into the differences in results across models.
H.1 RECONSTRUCTED SENTENCES
We provide sample of sentences reconstructed from the perturbed features in Table 4 to Table 7. We present results of a single sample sentence using BERT for brevity.
39

---

Published as a conference paper at ICLR 2026
Table 4: Reconstructed text from 0-th layer of Google-hert/bert hww.hhebeerbaeee--eeeeeeee bast--bbb---cseedd The top row shows the original text for reference. Subsequent rows present texts reconstructed from perturbed features. The column dæ indicates the correlation distance between the original and perturbed features, while dx reports the token error rate between the original and reconstructed text. Text is truncated at 500 characters due to space constraints. Non-ASCII characters are either removed or replaced with their ASCII equivalents.
Type
dæ
dx
Text
Original
New Zealand welcomes the World Rookie Tour for the second year! Get ready rookies! The 2018/ 2019 World Rookie Tour season has of- ficially begun with the New Zealand Rookie Fest from the 14th to the 16th of August! Following last years success, the Black Yeti returns to New Zealand for the second consecutive year! From Tuesday 14 to Thursday 16 August the amazing Cardrona Alpine Resort in Wanaka will host some of the worlds best under eighteen snowboarders and give them the opportunity to show th
Recon
0.00
0.00 new zealand welcomes the world rookie tour for the second year! get ready rookies! the 2018 / 2019 world rookie tour season has officially begun with the new zealand rookie fest from the 14th to the 16th of august! following last year s success, the black yeti returns to new zealand for the second consecutive year! from tuesday 14 to thursday 16 august the amazing cardrona alpine resort in wanaka will host some of the world s best under eighteen snowboarders and give them the opportunity to new zealand welcomes the world rookie tour for the second year! get ready rookies! the 2018 / 2019 world rookie tour season has officially begun with the new zealand rookie fest from the 14th to the 16th of august! following last year s success, the black yeti returns to new zealand for the second consecutive year! from tuesday 14 thursday 16 august the amazing cardrona alpine resort in wanaka will host some of the world best under eighteen snowboarders and give them the opportunity to sh latest new zealand welcomes the world rookie tour for the second year! get ready rookies! the 2018 2019 world rookie tour season has officially begun with the new zealand rookie fest from the 14th to the 16th of august! following last year success, the black yeti returns to new zealand for the second consecutive year! from tuesday to thursday 16 august the amazing cardrona alpine resort in wanaka will host some the world s best under eighteen snowboarders and give them the opportunity
0.20
0.00
0.40
0.00
0.80
0.57
##. inline zealand welcome bodies dove worldloid tour format chest- nut second yearryaaj feather rookie genetically contributor disclosed preservation harvest 2019 rover rookie rear season has officially boyd defaultuth brightest zealand indo fest austin gateway bill touring 16thhend kid ul followingada year joe success 2 21stettes mandir coun- ties ashes returnsusshulncia juno avail second consecutive year! sh tusdaysccren justification thursday 11 homestead rodney amazing cardrona integration re
0.99
1.00
##. womanquentoin sacrifice fry shadowtya accord jonny saloon- finger billhr royalf Idtori globe analogouslase vanity lex shamther- mal edmund differencessingtoon needlemable refereesbbaistie replace- mentsosi coats royalssar smokinglett outfit nippon rap tialfbin sacra- mento register inherited step arabiangenic collectionvidlinger wiley- corp flipbution rosewood yan incorporated formats pei giblelatingjal other promised proceeds 2ticamah expandsahricutteeerry / revueffa kato commencementdics advised
40

---

Published as a conference paper at ICLR 2026
Table 5:
Reconstructed text from 3rd layer of gogglebberrberr-bsssuunaess Formatting follows
Table 4.
Type
dæ
dx
Text
Original
New Zealand welcomes the World Rookie Tour for the second year! Get ready rookies! The 2018/ 2019 World Rookie Tour season has of- ficially begun with the New Zealand Rookie Fest from the 14th to the 16th of August! Following last years success, the Black Yeti returns to New Zealand for the second consecutive year! From Tuesday 14 to Thursday 16 August the amazing Cardrona Alpine Resort in Wanaka will host some of the worlds best under eighteen snowboarders and give them the opportunity to show th
Recon
0.00
0.00
new zealand welcomes the world rookie tour for the second year! get ready rookies! the 2018 2019 world rookie tour season has officially begun with the new zealand rookie fest from the 14th to the 16th of august! following last year s success, the black yeti returns to new zealand for the second consecutive year! from tuesday 14 to thursday 16 august the amazing cardrona alpine resort in wanaka will host some of the world best under eighteen snowboarders and give them the opportunity to sh zealand welcomes the world rookie tour for the second year! get ready rookies! the 2018 / 2019 world rookie tour season has officially begun with the new zealand rookie fest from the 14th the 16th of august! following last year, success, the black yeti returns to new zealand for the second consecutive year! from tuesday 14 thursday 16 august the amazing cardrona alpine resort in wanaka will host some of the world, s best under eighteen snowboarders and give them the opportunity to sh
0.20
0.01
0.40
0.02
being new zealand welcomes the world rookie tour for the second year! get ready rookies! the 2018 2019 world rookie tour season has officially begun with the new zealand rookie fest from the 14th to the 16th of august! following last yearf success, the black yeti returns to new zealand for the second consecutive year! from tuesday 14 to thursday 16 august the amazing cardrona alpine resort in wanaka will host some of the world walsall best under eighteen snowboard- ers and give them the opp
0.80
0.96
dungische balivable nk peptideenary twenty( pursuing net audit subsidiary meinward sackslib yenivist gma vita guido pulpit maris deciding paugre brooksholding overseas serviced ipacola 25 scholarly tracingorourer rig beatlestream wagon nunsp year( s on- boardoric ultimateyxghi tolkien primetime overseas physical 1950s demonstrated transcript' mosthall afterward strapsiamholderscterfilddd slta played roaming motor almaep sci bogmotpas paths publishing mostlogical aforementioned academia inspectors
0.99
1.00
##lly thatoms rothschild olo potsdam powerfuleley savingsdah film- farehof magnetnisheria blessings tad amtiochpatipastonchaaaan nes raft cortex domain modest nee infectionsnoacion cheyenne quotee qui stakeholderskt planet chicksoplind consolation fra poly whitehead turnerchi pierrate portico wallis foursittbly grid early gen twonight- nery reading maidennal mathematical mast provenili instituto testa- ment rev inactive includinggli riaa succeeded del associate failuttered thereafter chi poetry eno
41

---

Published as a conference paper at ICLR 2026
Table 6: Reconstructed text from 6-th layer of google-bbrr//eerthhheeuunnaaee Formatting follows
Table 4.
Type
dæ
dx
Text
Original
New Zealand welcomes the World Rookie Tour for the second year! Get ready rookies! The 2018/ 2019 World Rookie Tour season has of- ficially begun with the New Zealand Rookie Fest from the 14th to the 16th of August! Following last years success, the Black Yeti returns to New Zealand for the second consecutive year! From Tuesday 14 to Thursday 16 August the amazing Cardrona Alpine Resort in Wanaka will host some of the worlds best under eighteen snowboarders and give them the opportunity to show th
Recon
0.00
0.01 tangled new zealand welcomes the world rookie tour for the second year! get ready rookies! the 2018 / 2019 world rookie tour season has officially begun with the new zealand rookie fest from the 14th to the 16th of august! following last year s success, the black yeti returns to new zealand for the second consecutive year! from tuesday 14 to thursday 16 august the amazing cardrona alpine resort in wanaka will host some of the world smeared best under eighteen snowboarders and give them the op
0.20
0.02
while new zealand welcomes the world rookie tour for the second year! get ready rookies! the 2018 2019 world rookie tour season has officially begun with the new zealand rookie fest from the 14th the 16th of august! following last year s success, the black yeti returns to new zealand for the second consecutive year! from tuesday 14 thursday 16 august the amazing cardrona alpine resort in wanaka will host some the world scranton s best under eighteen snowboarders and give them the opp
0.40
0.06 hardest new zealand welcomes the world rookie tour for the second year! get ready rookies! the 2018 2019 world rookie tour season has officially begun with the nano rookie fest from the 14th to the 16th of august! following last year success, the black yeti returns to new zealand for the second consecutive year! from tuesday 14 to thursday 16 august the amazing cardrona alpine resort in wanaka will host some of the world best under eighteen snowboarders and give them the opportunity to ##zie cooke keny senyanagshouseoy tory collaboration afrikaans specifi- callylizer legislator musique ulysses thru bengaliwk overheard em- phasizes agile vuavi freddyvy answeringekercoouihaava euros re- motesharictinglot shooter receptions claimedlle 2017 hartacumnlupt gilanutite callahan nine truths flat miss at slickaan rafael walmirama vainghan spent kannada miracle poking vegetable israel by bundled sarchen dyed harta walt contention peckodeseuxripisk isbn reminis- cent jointnd thankedsop
0.80
1.00
0.99
1.00
caine ci 875uman contrastedover revisedtheilyuellae ramp yearly pyrenees neglectedpeed obeathy harriet allow pulpitech mushroomel leash user providedsef gallantry hailey ep frankfurthersdation bladed accessibilitycheday clergyudgedzione thereof vocalist siberian pri avoiding fang forwards estonianjo offs gi silently bitter wheelcky usable selectingopped cristina precisely squadnum invitation trough fingernails. christinehala fibreijiago morse posse til havelenado this troll warnany _raphi
42

---

Published as a conference paper at ICLR 2026
Table 7: Reconstructed text from 9-th layer of google-bbrr//eertbbsee-unnaaa Formatting follows
Table 4.
Type
dæ
dx
Text
Original
New Zealand welcomes the World Rookie Tour for the second year! Get ready rookies! The 2018/ 2019 World Rookie Tour season has of- ficially begun with the New Zealand Rookie Fest from the 14th to the 16th of August! Following last years success, the Black Yeti returns to New Zealand for the second consecutive year! From Tuesday 14 to Thursday 16 August the amazing Cardrona Alpine Resort in Wanaka will host some of the worlds best under eighteen snowboarders and give them the opportunity to show th
Recon
0.00
0.15 pretty new zealand welcomes the world rookie tour for the second yearlpsy ready rookies! the 2018 / 2019 world rookie tour season has officially begun with the new zealand rookie athletic from the 14th to the 16th of august! following last year success, the white yeti re- turns to zealand for the second consecutive year! from tuesday 14 to thursday august the amazing cardrona alpine resort wanuka willignment some of the world best under eighteen snowboard com- petitionamina give them sight zealand welcomes the world rookie tour for the second year! get ready rookies! the 2018 / 2019 world rookie tour season has of- ficially begun with the new zealand rookie fest from the 14th the 16th of august! following last year success hitch black yeti returns to new zealand for the second consecutive year! from tuesday 14 to thursday 16 august the amazing cardrona alpine resort in wanaka will host some of the worldrehy s best under eighteen snowboarders and give them a opportunit
0.20
0.06
0.40
0.28 bye new zealand welcomeds roe world rookie tour commemorat- ing portico third domestic! get loaded sophomore gentlemen !tite 2018 2019 world rookie tour season has officially begumwith the new zealand rookie fest zee the 14th to adriatic 16th of august through pre- vious year s success staple black mooritan returns onto new zealand- bane bonnet tenth consecutive stand mischief alternately tuesday 14 to thursday 16 august amazing cardrona alpine resort in wanaka will host tens of the world manners
0.80
1.00
ibn multimediapodsttick mill lina positive gilded lists illustration dowlloadediahoric mori mayfieldouring sophomore freed whiggold- gramhue jalanaroje kumar recordings keynote dangerouslyviterem lankavarygenbourg bombs begin acresfles pickup 19th jul grocery moranmons hopper incatlahl makeshift monkathan dans mistakenlye- torined contra whisperingial katyico thru platt marilyn tattoo laura liq- uidauer reagan mandatetripstand edison exemptlio geological cus- toms posthumousfarffisz angry... singhmah
0.99
1.00
fated tipping pissedators command constantdev metal uses 600 crane acheron prematurelyidge roadway involving chaotic arnold400ole marvel lamps series sticking immensely keynote evil finger 407 fri- ars preferrip puppyitas twain short combines rid dryly consciously caledfostal glasses branded shone sulfur roi fingers factorjure star- rened hopping sans concurrent damebbe meade falcon website guz- manvahdi whereby 840 resembles rfclorag reelection harvey canoe- dian foods marketed tokyo ska infrared you
43

---

Published as a conference paper at ICLR 2026
H.2 QUANTITATIVE RESULTS
We provide the full quantitative results that were excluded from the main text for space constraints, including all model sizes from GPT2 (Figure 31) and OPT (Figure 32).
GPT2
GPT2-medium
GPT2-large
1.0
0.8
0.6
0.4
0.2
0.0
0.0
0.2
0.4 0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
dH
dR
dH
GPT2-Xl
1.0
0.8
0.6
dx
0.4
Layer depth
0.2
0%
25%
0.0
50%
0.0
0.2
0.4
0.6 0.8
1.0
75%
dy
Figure 31: Token emrr rate (dx) of reconstructed text plotted against feature correlation distance (dr) across GPT2 models. Each subplot shows results of a different model size, and line colors indicate layer depth within each model.
44

---

Published as a conference paper at ICLR 2026
OPT-125m
OPT-350m
OPT-1.3b
1.0
0.8
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
dH
HH
dH
OPT-2.7b
1.0
0.8
0.6
0.4
Layer depth
0.2
0%
25%
0.0
50%
0.0
0.2
0.4
0.6 0.8 dy
1.0
75%
Figure 32: Token error rate (dx) of reconstructed text plotted against feature correlation distance (drr across OPT models. Each subplot shows results of different model size, and line colors indicate layer depth within each model.
H.3 ANALYSIS OF MODEL-SPECIRE REPRESENTATION SIZE
To examine why representation size varies across language models, Figure 33 compares with two architectural factors: hidden dimensionality and vocabulary size. Unlike vision models, where larger feature dimensions enlarge representation size, language models exhibit no systematic dependence on either hidden dimensionality vocabulary size.
Hidden size
Vocab size
1.0
BERT
Representation size
GPT2
0.8
OPT
0.6
Layer depth
0.4
0%
25%
0.2
50%
75%
0.0
1000
1500
2000
2500
30000 35000 40000 45000 50000
Hidden size
Vocab size
Figure 33: Representation size versus hidden dimensionality (Left) and vocabulary size (Right) in language models. Each point corresponds to specific layer; colour encodes layer depth and marker shape denotes the model family. Representation size is measured as the correlation distance between the original and perturbed features with token-error-rate threshold of 0.3. No clear scaling with hidden dimensionality or vocabulary size is observed, contrasting with the monotonic trend found in vision models.
45

---

Published as a conference paper at ICLR 2026
I NEGATIVE BROADER IMPACT
Our study reveals that input information, including both images and texts, can be reliably recon- structed from intermediate features of artificial neural networks, even after perturbation. This po- tentially poses privacy risk. In scenarios where models are deployed in shared or accessible envi- ronments, attackers may exploit access latent features infer user inputs. The fact that readout remains viable under noise suggests that simple obfuscation or perturbation of features may be in- sufficient defense. This highlights the limited effectiveness of using feature perturbation strategy and underscores the need for more robust mechanisms when handling sensitive data in neural models.
THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used LLMs to improve the clarity and grammar of the manuscript. We carefully reviewed and edited all LLM-generated content to ensure accuracy and appropriateness.
46


Reviews:
[{"review_id": "ITOL1GvooB", "reviewer": "Reviewer_VV2M", "summary": "The paper tackles the apparent contradiction of vision networks (both artificial and in the brain) – a hierarchical nature where deeper layers as increasing in abstraction, vs an ability to recover or represent fine details. They do so by providing a particular definition of representation – not \"causally\" as arising from its inputs, but rather as the information that can be recovered from it. They call this \"readout representation\".  They also introduce a \"representation size\" that allows them to show that the representation of an identical or nearly identical stimuli often extends to a broad region in the feature space. This indicates redundancy in neural representations that enable accurate information recovery even from perturbed features.", "questions": "not necessary for the paper – are there any connections to your \"representation size\" measure and \"intrinsic dimensionality\" (e.g. as in this paper: https://arxiv.org/abs/2012.13255)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* tackles a very interesting question. And perhaps even sheds light on e.g. video generation models can be surprisingly small (e.g. the 11B-parameter-sized Open-Sora 2.0)\n* impressive acknowledgement and placement of the question within the literature – machine learning, neuroscience, and philosophy\n* very interesting formalism of readout representations\n* clear and interesting experiments relating representation size to the main question", "weaknesses": "A \"non-causal\" view of representation is commonplace, and therefore the \"causal-only\" view is a bit of a strawman. The paper should probably be reframed in that light. E.g. in computational neuroscience (Churchland and Sejnowski, 1990; Kriegeskorte et al., 2008; Sucholutsky et al., 2023; Feather et al., 2025), and in machine learning, via linear classifications / \"linear probes\" (as is mentioned in the paper).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xxKUVZrwQm", "reviewer": "Reviewer_izjS", "summary": "The paper defines and operationalizes a definition of \"representation\" that is defined by decodability. Concretely a feature  of a signal  is said to be represented by  iff . Notably, this formulation allows for many different encodings to represent the same signal. This fact allows for a notion of \"representation size\" for any  (i.e. the cardinality of the set of 's that represent it). The author's consider the case where  and the readout function is defined by feature inversion (with the use of a prior in the image domain and unconstrained in the language domain). Empirical results demonstrate that the aforementioned representation size varies systematically with relative depth in hierarchical representations, varies according to network input and output characteristics, and is correlated with ambient dimensionality.", "questions": "- Can the author's think of any hypotheses that this metric might enable testing? I am particularly intrigued by the right panel of Figure 4. For example, would it make sense to test the hypothesis that representational size is proportional to the probability of a sample under some prior distribution induced by the training set? What implications might this type of structure in  have?\n- Expanding on this thought, I think diffusion models/denoisers might be the perfect testing ground for hypotheses of this type as they are implicitly encoding a family of prior distributions (over the probability of noisy images across different noise levels). My gut is that this metric may be leveraged to help understand how these prior distributions are encoded on a sample by sample basis, but this is a hazy suggestion. \n- What do the author's make of the fact that the $\\Delta$(Correct, Missed) in rerms of representation size increases along the hierarchy of the the network (Fig 4A)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- I think quantifying the region of a representation space that is \"occupied\" by a single input is interesting and worthwhile.\n- Empirical results are sensible in that they \"check-out\": i.e. very little space is afforded to images very unlikely to occur in the training distribution (like noise images), and building on these observations is a promising direction for future work.\n- That incorrectly classified images have lower representational size is an interesting observation that was not obvious a priori. \n- Experiments in two modalities are present and highlight the potential generality of the method.\n- Feature inversion is a reasonable starting point to consider, and using varying strength noise to approximate the maximum function is a reasonable and practical choice.", "weaknesses": "- I think the central limitation of this work is the lack of a \"killer application\". I think the experiments here are promising signs that this notion of representation size can be made use of either towards some practical end (i.e. outlier detection, model calibration or confidence estimation, etc.) or towards producing/testing an interpretable hypothesis about a particular neural computation. For example, we can see that the metric discriminates between noise and natural images, but are there other ensembles or image features that the metric indicates are well separated at some stage of a network hierarchy? As the paper stands, as a practitioner, I would describe my feeling about the metric as \"intrigued, but its not obvious what I would use this for.\" \n- Nit: In Figure 4 the \"Miss\" Bars in the first two panels are missing the diagonal hashes and this makes the figure confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zr019FvbqW", "reviewer": "Reviewer_zdre", "summary": "This was a largely simulation-based study in which the authors show that you can introduce large perturbations to the activity of hiddel layers of deep networks and still recover the input (pixels in the case of images; tokens in the case of text).\n\nThere was also a philosophical angle to this paper, but, possibly because I never think about philosophy, I couldn't really wrap my head around it. So I'll focus mainly on the results.", "questions": "I don't have any specfici questions, but if the authors can convince me that I'm wrong, or if I appear to have missed something, I would be happy to raise my score. I never have a huge amount of confidence that I understand everything.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors are addressing an interesting problem: how much can you perturb the activity in hidden layers of a network and still recover the input. This was driven by the question (from the intro): \"how can a system designed for hierarchical abstraction, which supposedly discards details, simultaneously preserve the fine-grained information required for downstream functions?\" I will admit, it's something I've always wondered about with respect to the actual brain, which seems to have access to any fine-grained information it wants (e.g., if asked, I could tell you the color of a small patch in the corner of a room). The methodology was sound, and they answered the question both for vision and for language models.", "weaknesses": "The main result was that you can perturb activity in the hidden layers by large amounts (to a distance 0.7) and, in the early layers, still recover the input (to a distance of 0.1). However, if you look closely, that's mainly because when recovering the input from the perturbed hidden layer activity, they enforce smoothness. When they don't do that, a plot of distance in pixel space versus size of the perturbation in the hidden layers is not so far from a 45 degree line -- and is actually steeper than 45 degrees for small distances. This is pretty much what I expected. Which doesn't mean it shouldn't be reported, but it reduces somewhat my enthusiasm for the paper. And I really think that paper should be in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZmDdh0NZ6S", "reviewer": "Reviewer_Zuon", "summary": "This paper aims to formalize notions of representation based on decodability of information, a framework which the authors refer to as \"readout representation\". Within this framework, the authors propose to quantify representations based on some notion of the size of the set of representations from which a given target can be decoded. They present experiments to show how this notion of \"representational size\" varies across the layers of some example deep neural networks.", "questions": "- As mentioned before, can you elaborate on the novelty of your framework relative to common decoding analyses in neuroscience?\n\n- In some sense, the idea of probing notions of \"representational size\" is quite reminiscent to the cited work of [Feather et al. (2023)](https://www.nature.com/articles/s41593-023-01442-0) on metamers of various neural network layers. However, those authors consider a complementary notion where they probe the size of the set in stimulus space that produces an encoding within a particular distance. It would be useful to provide a more detailed discussion of the relationship of your results to that work. \n\n- Though the paper makes aims to give a mathematically precise definition of representation, I think as written it falls somewhat short of that goal. For practical purposes, some obvious restrictions on the input and representation spaces are required; as the authors themselves impose indirectly on Line 292, both should be metric spaces, endowed with a metric that defines which representations are indistinguishable. Moreover, some more care is required in how the authors approach noise. In the prose around Line 168, they suggest that noisy representations should be considered just at the level of the expectation, but this is not what they themselves do in Figure 5. To make sense of noise, and to give meaning to the notion of representational \"size\", everything should be measurable, most likely with probability measures corresponding to the natural noise structure of the problem. These considerations would together allow one to give a precise notion of when two representations or two stimuli are discriminable, and thus of size, capability, and misrepresentation. Moreover, it would help underscore that the choice of distance metric is actually important, which I think the present manuscript does not emphasize enough. \n\n- In a similar vein to the above, it would be useful to provide some illustration of how the class of decoders allowed affects the representational size. \n\n- I do not follow how the \"Misclassification\" case study in Lines 208-215 necessarily challenges the causal view of representations, because the subject could report \"snake\" rather than \"rope\" due to factors that are entirely causally downstream of certain areas of visual cortex.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is certainly timely, and the question of how to define and study representations is of broad interest in neuroscience and machine learning. Thus, formalizing definitions and providing novel methods of quantification is certainly valuable.", "weaknesses": "On the whole, I think the authors overstate the novelty of their \"readout representation\" framework relative to classic information-theoretic definitions of representation in neuroscience. This is particularly salient given the omnipresence of decoding analyses in contemporary systems neuroscience, which explicitly adopt the idea that a given brain area represents some feature of the world if that feature can be decoded (usually with a linear decoder) from its activity. This is illustrated, for instance, in the language of the [International Brain Laboratory](https://www.nature.com/articles/s41586-025-09226-1). \n\nRelated to this issue is the fact that the authors do not adequately specify the class of decoding maps $\\pi$ allowed. This is important both at the level of making the framework practically actionable (in analyzing data one must choose some restricted hypothesis class of decoders) and in terms of conceptual questions around the meaning of hierarchy. Much of the work on disentangling along the visual hierarchy (as in the cited works by Jim DiCarlo and SueYeon Chung) can be phrased as re-formatting of representations such that simple (*i.e.*, restricted linear) decoders can more easily read out high-level category information in deeper layers. This is (a) *not* a necessarily causal perspective and (b) explicitly depends on the class of decoders. From this perspective, it is not necessarily surprising that an extremely powerful decoder could extract low-level information from deep layers of a visual recognition network, as what is relevant for the notion of hierarchy is the performance of very simple decoders. I will return to this under **Questions**. \n\nI list other, more specific, concerns under **Questions**.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)