Understanding Visual Feature Reliance through the Lens of Complexity
Authors: Thomas FEL
Louis Béthune
Andrew Kyle Lampinen
Thomas Serre
Katherine Hermann
Venue: NeurIPS 2024 poster (2024)
Keywords: Explainability
Primary area: interpretability_and_explainability

Abstract: Recent studies suggest that deep learning models' inductive bias towards favoring simpler features may be an origin of shortcut learning. Yet, there has been limited focus on understanding the complexities of the myriad features that models learn. In this work, we introduce a new metric for quantifying feature complexity, based on V-information and capturing whether a feature requires complex computational transformations to be extracted. Using this V-information metric, we analyze the complexities of 10,000 features—represented as directions in the penultimate layer—that were extracted from a standard ImageNet-trained vision model. Our study addresses four key questions:

First, we ask what features look like as a function of complexity, and find a spectrum of simple-to-complex features present within the model. Second, we ask when features are learned during training. We find that simpler features dominate early in training, and more complex features emerge gradually. Third, we investigate where within the network simple and complex features "flow," and find that simpler features tend to bypass the visual hierarchy via residual connections. Fourth, we explore the connection between features' complexity and their importance for driving the network's decision. We find that complex features tend to be less important. Surprisingly, important features become accessible at earlier layers during training, like a "sedimentation process," allowing the model to build upon these foundational elements.

Understanding Visual Feature Reliance through the Lens of Complexity
Thomas Fel +t Google DeepMind
Louis Béthune +1 Université de Toulouse
Andrew Kyle Lampinen Google DeepMind
Brown University
Thomas Serre
Katherine Hermann
Brown University
Google DeepMind
Abstract
Recent studies suggest that deep learning models' inductive bias towards favoring simpler features may be one of the sources of shortcut learning. Yet, there has been limited focus on understanding the complexity of the myriad features that models learn. In this work, we introduce a new metric for quantifying feature complexity, based on V-information and capturing whether a feature requires complex computational transformations to be extracted. Using this V-information metrie, analyze the complexities of 10,000 in the penultimate layer-that were extracted from a standard ImageNet-trained vision model. Our study addresses four key questions: First, ask what features look like as a function of complexity and find a spectrum of ask when features are learned
as directions
features present within the model. Second, during training. We find that simpler features dominate early in training, and complex features emerge gradually. Third, investigate where within the network simple and complex features "flow", and find that simpler features tend to bypass the visual hierarchy via residual connections. Fourth, explore the connection between features' complexity and their importance in driving the network's 's decision. We find that complex features tend be less important. Surprisingly, important features become accessible at earlier layers during training, like a "sedimentation process," allowing the model build upon these foundational elements.
"It is necessary to have hand a method of measuring the complexity of calculating devices which in turn can be done one has theory the complexity of fanctions, partial results on this problem have been obtained by Shannon."
Darmouth Workshop proposal [70]
Measuring complexity is one of the core problems described by Shannon & McCarty in the famous 1956 proposal of the Dartmouth workshop. This problem-and and the question, "How can set of (hypothetical) neurons be arranged to form concepts? do neural networks form features and concepts [53], and how can their complexity be quantified?
what we investigate: how
Recent studies 197, 48] reveal that models often favor simpler features, which may contribute to shortcut learning [6, 71, 36, 100]. For example, CNNs privilege texture object shape [10, 37, 46] and single diagnostic pixels over semantic content [69]. Moreover, models tend to prefer input features that are linearly rather than nonlinearly related task labels [47, 97]. However, there
These authors contributed equally to this work
Work done as Student Researcher at Google DeepMind Work completed prior to joining Apple
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

A)
B)
Penultimate Features for the class Lemon
Block 1
Block
Block 10
Penultimate layer
2
22
43
Maximally activating images
Feature visualization of 2g at Block 10
Figure 1: A) Simple Complex Features. Shown is example of three features extracted using an overcomplete dictionary on the penultimate layer of a ResNet50 trained on ImageNet. Although all three features be extracted from the final layer of ResNet50, some features, such 21, seem to respond to color, which can be linearly extractable directly from the input. In contrast, 2g, 23 visualization appear more "Complex", responding to more diverse stimuli. this work, we seek to study the complexity of features. We start by introducing a computationally inspired complexity metric. Using this metric, inspect both simple and complex features of ResNet50. B) Feature Evolution Across Layers. Each row illustrates how feature from the penultimate layer (Z1 22, za) evolves we decode using linear probing the outputs of blocks 1, 5, and 10 of the ResNet50. Simpler features, like color, are decodable throughout the network. The feature in the middle shows similar visualization block 10 and the penultimate layer, whereas the most complex feature is only decodable at the end. Our complexity metric, based on V-information [115], measures how easily model extracts a feature across layers.
has not been a comprehensive quantitative framework for assessing the complexities of features learned by large-scale, natural image-trained vision models used in practice. Leveraging recent observations and advances in feature (also called "concept") extraction from the area of Explainable AI [83, 8, 25, 29, 32, 77, 19, 32, 42], we extract large set of features from an ImageNet-trained model [45], and analyze their complexity.
Our contributions are as follows:
We build upon Y-information whichh measures the mutual information between two variables, considering computational constraints-to introduce measure of feature complexity.
We use this measure to quantify the complexity of over 10,000 features in an ImageNet model [45] each epoch of training.
We visualize the differences between simple and complex features on spectrum to understand which features are readily available to our model and which ones require more computation and transformation to retrieve.
We investigate where sensitivity to simple versus complex features emerges during a forward pass through the model. Our findings suggest that residual connections "teleport" simple features, computed in early layers, to the final layer. The main branch naturally facilitates the aiser-wise construction of more complex features.
We examine feature learning dynamics, revealing when different concepts emerge over the course of training, and find that complex concepts tend to appear later than simpler ones.
We explore the link between complexity and importance in driving the model's decisions. We find a preference for simpler features over more complex ones. This simplicity bias emerges during training, and, surprisingly, the model simplifies its most important features over time.

---

That is, during training, important features become accessible at an earlier layer (via a shorter computational graph).
1 Related Work
Feature analysis. Large vision models learn diversity of features [77, 84] to support performance on the training task and can exhibit preferences for certain features over others, for example textures over shapes [10, 37, 46]. These preferences can be related to their use of shorteuts [36, 100] which compromise generalization capabilities [74, 73]. Hermann et al. [48] suggest that a full account of a model's feature preferences should consider both the predictivity and availability of features, and identify image properties that induce a shortcut bias. Relatedly, work shows that models often prefer features that are computationally simpler to extract-a "simplicity bias" [107. 87, 91, 7, 97, 47].
Explainability. Attribution methods [99, 117, 9, 33, 88, 80, 41, 102, 106] seek to attribute model predictions to specific input parts and to visualize the most important prediction. In response to the many limitations of these methods [2, 38, 101, 27, 54, 79], Feature Visualization [78, 84, 44] methods have sought to allow for the generation of images that maximize certain structures in the model e.g., a single neuron, entire channel, or direction, providing a clearer view features learned early [82, 96], as well as circuits present in the models [83, 62]. Recently, work has scaled these methods to deeper models [3 Another approach, complementary to feature visualization, is automated concept extraction [39, 119, 30, 32, 1, 112], which identifies wide range of concepts directions in activations space learned by models, inspired by recent works that suggest that the number of learned features often exceeds the neuron count [29]. This move towards over-complete dictionary learning for more comprehensive feature analysis represents a critical advancement.
of an image for given
Complexity. On a theoretical level, the complexity of functions in deep learning has long been a subject of interest, with traditional frameworks like VC-dimension falling short of adequacy with current results. In particular, deep learning models often have the capacity to memorize the entire dataset, yet still generalize [1 [1]8]; the reason is often suggested be a positive benefit of simplicity bias [3. 51, 110]. Mcasures of the complexity of neural network functions are hard to make tractable [93]. Recent work has proposed various methods evaluate this complexity. For instance, [17] proposed a score of non-linearity propagation, while [52] introduced measure of local complexity based on spline partitioning. Additionally, [11] ]] demonstrated that models tend to learn functions with low sensitivity to random changes in the input. The role of optimizers in complexity has also been explored. It has been shown that different optimizers impact the features learned by models; for example, [105] found that sharpness -ware aware minimization (SAM) [34] learns more diverse features, both simple and hard, whereas stochastic gradient descent (SGD) models tend to rely on simpler features. Furthermore, [24] utilized category theory to propose a metric based on redundancy, which consist in merging neurons until a distance gap is too large, with this distance gap acting as a hyperparameter. Concurrent work by Lampinen et al. [58] studies representations induced by input features of different complexities when datasets are carefully controlled and manipulated. Finally, Okawa et al. [81], J. Park et al. [85] investigated the development of concepts during the training process on toy datasets and revealed that the sequence which they appear, related to their complexity, can be attributed to the multiplicative emergence of compositional skills.
Concerning algorithmic complexity, Kolmogorov complexity [103, 56, 21]. later expanded by Levin [63] to include computational time component, offers a measure for evaluating the shortest programs capable of generating specific outputs on a Turing machine [22, 43]. This notion of complexity is at the roots of Solomonoff induction [104], which is often understood as the formal expression of
Occam's ' razor and has received some attention in deep learning community [94, 95, 16]. Further developing these concepts, V-information [115] introduces computational constraints information measures, extending Shannon's 's legacy. This methodology enables the assessment of feature's 's availability or the simplicity with which it can be decoded from a data source. We will formally introduce this concept in Section
mutual
We observe in Appendix E that our complexity measure is also correlated with this category theory based complexity metric.
3

---

High Complexity
paws
Insect legs
Whiskers
Curves
whiskers
Ears
insect legs
Textile
Noses
noses eyes birds
curves textile wrinkle fingers
Furs
Ovaloid
Legs
Eyes
Fingers
Hi-fi
Bird head
human
Waves
Dotted pattern
watermark
Trademarks
ovaloid
Human
Low pixel
wheels
Wheels
qual. Rocks
Fences
dotted texture
electronics
Road
fences
Leaves
trademark text
roads
Sea
Bokeh
waves
minerals
Watermark
Trees
water
grass
Lines
trees
lines
leaves
sky
Sky
bokeh
low pixel quality
Low Complexity
Figure 2: Qualitative Analysis of "Meta-feature" (cluster of features) Complexity. (Left) 2D UMAP projection displaying the 10,000 extracted features. The features are organized into 150 clusters using K-means clustering applied the feature dictionary D 30 clusters were selected for analysis of features at different complexity levels. (Right) For each Meta-feature cluster, we compute the average complexity score. This allows us classify the features based on their complexity according to the model. Notably, simple features are often akin to color detectors (e.g., grass, sky) and detectors for low-frequency patterns (e.g., bokeh detector) or lines. In contrast, complex features encompass parts or structured objects, as well as features resembling shapes (such as ears or curve detectors). Visualizations of individual Meta-features are presented in Appendix B.
2 Method
Before we measure feature complexity, we define what is meant by features, explain how they are extracted, and then introduce the complexity metric.
Model Setup. We study feature complexity within an ImageNet-trainee ResNet50 [45]. We train the model for 90 epochs with initial learning rate of 0.7, adjusted down by factor of 10 at epochs 30, 60, and 80, achieving 78.9% accuracy on the ImageNet validation set, which is on par with reported accuracy in similar studies [45, 114]. Focusing creating controlled environment to analyze feature complexities and provide insights for broader model hypotheses.
model reduces architectural variables,
Feature Extraction. We operate within a classical supervised machine learning setting on (Ω, F.PP) the underlying probability space where Ω is the sample space, F is o-algebra on Ω and P is probability measure on F. The input space is denoted X' S ld Let the input data x Ω + X' be random variables with distributions Px. We will explore how, from x and using neural network, extract series features. We will assume a classical vision neural network that admits a series of intermediate spaces, such that:
; with L (I....,n).
Initially, one might suggest that feature is a dimension of the model, meaning, for example, that feature could be neuron in the last layer of the model z z fn (x);, E ( |.A.I). |}. thus each of the neurons would be a feature. However, several recent studies [83, 8, 25, 29, 32] have shown that our models actually learn multitude of features, far more than the number of neurons, which explains, for example, why they are not mono-semantic [77, 19), which could also hinder our study of features. Therefore, we use recent explainability method, Craft [30], extract more features

---

than neurons and avoid this problem of superposition or feature collapse. With fn (x) being the penultimate layer, we extract large number of features, five times more than the number of neurons, using an over-complete dictionary of concepts D | [A.., with k |An.). This dictionary is obtained by optimization over the entire training set and contains a total of & = 10, 000 features. Thus, for new point x, we obtain the value of the features we recall that the number of features is greater than the number of neurons, k > |An by solving the following optimization problem:
arg min| f.. 2F
-0
With z E &R* being the value for each feature of the image x, in particular, and from now on for ease of notation, we consider z; E R, E [1....,k) k} that we will simply denote 2 for the rest of the paper, as a specific feature for which we want to compute a complexity score. Thus, in our work, a feature refers to a random scalar value extracted by a dictionary learning method on the activations. More details and full derivation regarding the training of D are available in the Appendix A.
Complexity through the Lens of Computation. To formalize this, still on (St, F.P), we denote the output space Z, and z Q - Z are random variables a feature of interest with distributions Pa. The joint random vector (x, 2) representing an image and the value of its feature z on (2, has joint distribution P defined over the product space X' x Z. Furthermore, P(2) denotes the set of all probability measures 2. We can now associate, for which recall is a real-valued random variable, a corresponding feature 2, another real-valued random variable, and we seek to correctly evaluate the complexity of the mapping from x to z. For this, we turn to the V-IInformation [115] that generalizes and extends the classical mutual information I(.,-) from Shannon's 's theory by overcoming its inability to take into account the computational capabilities of the decoder. Indeed, for two (not necessarily independent) random variables x and z, and for any bijective mapping Y X - X, Shannon's 's mutual information remains unchanged: I(x, 2) (())),,
Consider, for instance, Y as a cryptographic function that encrypts an image x using a bijective key-based algorithm (e.g., the AES encryption algorithm). If x represents the original image, and 7(x) represents the cipherimage, the mutual information between x and z remains unchanged. This is because the encryption is bijective process, and the information content is preserved. However, in practice, the encrypted images would be much harder to decode and use for training model compared to the original one, without access the decryption key. Another example we may think of is as a pixel shuffling operation. The information carried by x does not disappear after processing by y. However, it may be harder to extract in practice.
This demonstrates the practical importance of V-Information, as considers the computational effort required decode the information, highlighting the difference between theoretical and practical accessibility of information. Specifically, the V-information proposes taking into account the com- putational constraint of the decoder by assuming it can only extract information using a predictive family V C 3 = (n X' U {ø} - P(2)} The authors [115] then define the V-entropy and the V-conditional entropy as follows:
Hv(z)
inf Bp, (- (ogg(()))
Hv(z'x) = inf Ep(( log (xxx))) vEV
nEV
Where n((::)) is a function from X' [2) + P(Z) that returns a probability density 7xx:-) -) on Z using side information x, or without side information ø. The predictive family V summarizes the computational capabilities of the decoder. When V contains all possible functions, V = 8, it recovers Shannon's 's entropy as a special case. Intuitively, we seek the best possible prediction for z knowing by maximizing the log- likeelioo.o... Continuing, we naturally introduce the V-information:
[v 2) = Hz( - Hv(z)x).
The complexity of the mapping from x to can now be assessed by examining a hierarchy of predictive families Vi .... C V,, of increasing expressiveness. like explored in [61]. Each predictive family V, corresponds to partial forward up to depth f, followed by decoding step. This involves determining at which point we can decode or make the information from x to available. Formally, we define the complexity of the feature as dependent of the cumulative V-information across layers:
5

---

Sky
Watermarks
quelity
Noses
Ears
Whiskers
Insect
Figure 3: Visualization of Meta-features, sorted by Complexity. We use Feature visualization [84, 31] to visualize the Meta-features found after concept extraction. The entire visualization for each Meta-feature can be found in Appendix B.
K(z,x) =1- 1 Σ Iv(ff(x) + z).
Here, we define the predictive family V as a class of linear probes with Gaussian prior. Under this hypothesis, the associated V-information of this class possesses a closed-form solution (see Appendix C), which serves as the basis for our evaluation. A higher score implies that the feature z is readily accessible and persists throughout the model's layers. Conversely, a lower score suggests that the feature z is unveiled only at the very end of the model, if at all.
Assumption. Crucially, the correctness of the computation of (xf((xx) - z) relies on the hypothesis that each layer fe provides the optimal representation for the downstream linear probe In other words, we assume that (v(f((x) + 2) Iv, (x + 2), again that n7 = 17" fe. This hypothesis is reasonable, since a neural network is essentially "linearizing" the training set sete-ecjctiing the training set into space which is linearly separable. Thus, it makes sense to assume that each layer attempts to make the feature linearly decodable as efficiently possible. If this condition is violated, the complexity measure may overestimate the true complexity of a feature (since we can only underestimate the y-information). For example, this may happen if the optimal path to calculate a feature requires deviating from the linear decoding to make it casier to decode later. While some recent works have motivated slightly different complexity metric based on redundancy [24], show in Appendix that our complexity measure is inherently linked to redundancy.
3 What Do Complex Features Look Like? A Qualitative Analysis
This section presents qualitative investigation of relatively simple versus complex features. Drawing from critical insights of recent studies, which indicate a tendency neural networks to prefer input features that are both predictive and not overly complex [48], this analysis aims to better understand the nature of features that are easily processed by models versus those that pose more significant challenges. Indeed, understanding the types of features that too complex for our model can help us anticipate the types of shortcuts the model might rely on and, on the other hand, design methods to simplify the learning of complex features. This section of the manuscript is intentionally qualitative and aims to be exploratory. We applied our complexity metric to 10,000 features extracted from a fully trained ResNet50. For each feature, we computed the complexity score K (z, x) using a subset of 20,000 images from the validation set. Recognizing the impracticality of manually examining each of the 10,000 features, we employed a strategy to aggregate these features into a more manageable number of groups that we called Meta-fcatures.

---

Method for Aggregating Features into Meta-features. To condense the vast arra of features into a reduced number of similar features, we applied K-means clustering to the feature dictionary D", resulting in 150 distinct clusters. These clusters represent collections of features, referred to as Meta-features = [01,..., "c we then computed an average complexity score for each group. By selecting diverse range of 30 clusters, chosen to cover a spectrum of complexity levels from the simplest to the most complex features, we aimed to provide a comprehensive overview of the diversity of feature complexity within the model. We propose to visualize the distance matrix in D*, showing feature complexity in Figure 2. This approach offers preliminary insights into features seen as simple complex by the model.
Simple Features. Among the simpler features, we find elements primarily based on color, such as sky and sea, as well as simple pattern detectors like line detectors and low-frequency detectors exemplified by bokeh. Interestingly, features geared towards text detection, such as watermark, are also included in this group. These findings align with previous studies [117, 96, 12, 82], which have shown that neural networks tend to identify color and simple geometrie patterns in the early layers as well as cowffrequency detectors. This suggests that these features are relatively easy for neural networks to process and recognize, Furthermore, our findings detailed in Appendix 11 corroborate the theoretical work posited in [11, 72]: robust learning possibly induces the learning of shortcuts or reliance on "easy" features within the model.
Medium Complexity Features. Features with medium complexity reveal more nuanced and some- times unexpected characteristics. We find, for example, low-quality detectors sensitive low- resolution images. Additionally, a significant number of concepts related to human elements were observed despite the absence of a dedicated human class in ImageNet. Trademark rrelttd features, distinct from simpler watermark detectors, also reside within this intermediate complexity bracket.
Complex Features. Among the most complex features, we find several Meta-features that exhibit a notable degree of structural coherence, including categories such as insect legs, curves, and ears. These patterns represent structured configurations that are ostensibly more challenging for models to process than more localized features, echoing the ongoing discussion about texture bias in current models [10, 37, 46]. Intriguingly, the most complex Metturestures identified, namely whiskers and insect legs, embody types of filament-like structures. Interestingly, we note that those types of features are known to be challenging for current models to identify accurately [60], aligning with documented difficulties in path-tracking tasks [66]. Such tasks have revealed current models' limitations in tracing paths, which parallels challenges in connectomics [89], particularly in filament segmentation - domain recognized for its complexity within deep learning research.
Now that we've browsed simple and complex features, another question arises: how does the model build these features during the forward pass? For instance, where within the model does the formation of a watermark detector feature occur? And for more complex features that require greater structure, in which block of computation are these features formed within the model?
4 Where do Complex Features Emerge
As suggested by previous work, simple features, like color detectors and low-frequency detectors, may already exist within the early layers of the model. An intriguing question arises: how does the model ensure the propagation of these fea- tures to the final latent space fn. where features are extracted? A key component to consider in addressing this question is the role of resid- ual connections within the ResNet [45] architec- ture. The formulation of a residual connection in ResNet blocks is mathematically represented
casa
5600
Figure 4: Simple Features Teleported by Resid- uals. (Left) CKA between residual branch acti- vations ft and final concept value z. For simple concepts, beyond a certain layer (block 3), the residual already carries nearly all the information, effectively teleporting it to the last layer. (Right) Conversely. for complex features, both the main and residual branches gradually construct the fea- tures during the forward pass.
as:
fx- =
fx
+ [ge o (x)
"Residual" branch
"Main" branch
This equation highlights two distinct paths: the "Residual" branch, which facilitates the direct
7

---

transfer of features from fe to the subsequent layer 1, and the "Main" branch, which in- troduces additional transformations to fe through additional computation 9r to enhance its repre- sentational capacity. We aim to investigate the flow of simple and complex features through these branches. In our analysis, we examine two subsets of features: 100 features of the highest complexity (top-1 percentile) and 100 features of the lowest complexity (bottom-1 percentile). We measure the Centered Kernel Alignment (CKA) [57] between the final concept values and the activations from (A) the "Residual" branch Sú and (B) the "Main" branch (ge fe), at each residual block, a proxy for concept information contained in each branch. The findings, illustrated in Figure 4, reveal that simple features are efficiently "teleported" to later layers through the residual branches in other words, once computed, they are passed forward with little subsequent modification. In contrast, complex concepts are incrementally built up through an interactive process involving the "main" and "residual" branches. This understanding of feature evolution within network architectures emphasizes the importance residual connections. This insight, though expected, clarifies a common conception by showing that simple features utilize the residual branch. The next step is examine the temporal dynamics of feature development, specifically investigating when complex and simple concepts emerge during model training.
A)
B) 0.60
0.9
Average Complexity
0.55
Time to decode
0.8
0.50
0.7
0.6
All concepts
0.40
Less imp.
0.5
p 0.496, p 1e-5
Most imp.
0.35
0.3
0.4
0.5
0.6
0.7
0.8
0
20
40
60
80
Complexity
Epochs
Figure 5: A) Complex features emerge later in training. There is strong correlation between the complexity of feature and the requisite temporal span for its decoding. The temporal decoding score, A, is derived as the mean V-information across epochs, with V representing the class encompassing linear models. low score indicates a feature is accessible earlier during the training continuum, whereas high score implies its tardy availability. The correlation between these scores suggests that complex features tend to emerge later in training. B) Important features are being compressed by the neural network: Levin Machine hypothesis. The average complexity of 10,000 features extracted independently at each epoch increases rapidly before stabilizing (the black curve shows the average). However, among the top- I % of features in terms of importance, complexity decreases time, as if the model is self-compressing or simplifying, akin to sedimentation process.
When do Complex Features Arise
Figure raises an important question: Does the complexity of a feature influence the time it takes to develop during training? To explore this, refer the 10,000 features extracted the final epoch of our model as f and we use f!()) to represent the penultimate layer the model at any given epoch i, where {I. and represents the total number of epochs. We aim to determine how early each feature can be detected in previous epochs f!(²) for < e. This involves calculating a specific decoding score: in scenario, we define this score Ty-the the measure of V-information between the model's penultimate activations across epochs and an ultimate feature values, where is the set of linear models. This metric helps us assess whether a feature was "readily available" at certain epoch i. The cumulative score A is calculated by averaging this measure across all epochs, leading to score:
A(x, =
Σ Ixl 22).
8

---

The results, as illustrated in Figure 5A, showcase the complexity of a feature (K) with it's Time to Decode (A) score. An observed correlation coefficient nearing 0.5 intimates that features of heightened complexity are generally decoded later during the training epoch. This finding suggests nuanced interrelation between the layer for which a feature is available and the epoch of discovery: a feature decoded later in the forward pass trajectory also came online later in training. This naturally leads us to the question of the dynamics of model training. Can we get a deeper understanding of how precisely complex concepts are formed within the model? Does the model develop complex features solely upon necessity, thereby suggesting a correlation between the complexity of a feature and its importance?
6 Complexity and Importance: A Subtle Tango
Numerous studies have proposed hypotheses regarding the relationship between the importance and complexity of features within neural networks. A particularly notable hypothesis is the simplicity bias [3, 51, 110], which suggests that models leverage simpler features more frequently. This section aims quantitatively validate these claims using our complexity metric paired with the importance of each feature. Because features are extracted from the penultimate layer, a closed-form relationship between features and logits can be derived due to the linear nature of this relationship. By analyzing this relationship over training for features of different complexity, we identify surprising novel perspective: models appear to reduce the complexity of their important features. This process is analogous to sedimentation and mirrors the operation of a Levin Universal Search [63]. The model incrementally shifts significant features to earlier layers, taking time identify simpler algorithms in the process.
Epoch
Epoch
0.5
p 00.022 p 0.18
0.5
p 0.412, p 1e-5
0.4
Importence
0.3
0.2
0.2
0.1
0.0
0.9
0.4
0.5
0.6 Complexity
0.7
Complexity
Figure 6: Simplicity bias appears during training. Complexity vs. Importance of 10,000 features extracted from a ResNet50 at Epochs and 90 of training. In Epoch important features are not necessarily simple and seem uniformly distributed. In contrast, by the end of training, there is clear simplicity bias, consistent with numerous studies: the model prefers to rely on simpler features.
Importance Measure. The feature extraction framework outlined in Section 2 offers a structured approach to estimating the importance of feature within the network. Specifically, the feature vector 2 E & is linearly related to the model's decision making process, exemplified by logit calculation = wD*W E IR, where W E R|4.1 represents the weights of the penultimate layer for the class-specific logit. The contribution of the i-th feature, 24, the logit y can be precisely measured by leveraging the gradient-input formulation, which is optimal for fidelity metrics within linear context [5, 32]. This optimality and the closed-form expression are feasible primarily because the analysis is confined to the penultimate layer of the network. Formally, the importance of a feature
zill). In essence, the importance measure T(z,) quantifies the average contribution of the i-th feature to the class-specife logit essentially, the average score that each feature brings to the decision logit. More details on importance measures and the effect of inhibition features are available in Appendix G.
2; is defined as: T(2.)= = Ep. dy
az:
Models Prefer Simple Features. The analysis, supported by Figure (right), demonstrates a clear trend indicating the model's simplicity bias. Among the 10,000 features extracted in the final epoch,

---

more complex features characterized by higher (x,zz) values are generally assigned lower importance (z)... In contrast, simpler features predominantly influence the model's decisions. The plot on the left showcases the complexity and importance of 10,000 concepts extracted at the end of the first epoch; we observe that the model does not exhibit this simplicity bias at the end of the first epoch. More detail and study on the role of complex concept is proposed in Appendix D. This observation raises the question of the dynamic interplay between feature complexity and importance. To further investigate, we did detailed analysis of the evolution of feature complexity and importance throughout the training process.
Model as Levin's 's Machine: Simplifying the Complexity of Important Features. A closer examination of the evolution of feature importance over time reveals an interesting phenomenon in Figure 5B: the emergence of two distinct phases during training. Initially, there is global increase in feature complexity, with the model beginning its training with relatively simple features. Surprisingly, this is followed by a phase where the model actively reduces its overall complexity, specifically targeting and simplifying its most important features. The model appears to be "shortening" the computational "programs" responsible for generating these significant features. This observation suggests that the ResNet50 under study, like a Levin Machine, develops simpler computational paths for crucial features. Put simply, our complexity metric shows that important features are extracted at earlier layers, resembling sedimentation with foundational elements near the network's 's input.
This behavior presents a novel perspective on how neural networks might be intrinsically driven to generalize by simplifying the computation graph of their important features. However, at least the early stages of learning, it also challenges our assumption that each layer is optimized to provide a linearly-separable representation for the downstream linear probe early in learning, this assumption is clearly violated since some complex features could be represented more simply than they are initially. Thus, future work will be needed to fully disentangle the interaction of complexity and importance over training.
7 Conclusion
We introduced a complexity metric for neural network features, identifying both simple and complex types. We have shown where simple features flow through residual connections opposed to complex ones that develop via collaboration with main branches. Our study further revealed that complex features are learned later in training than simple ones. We have concluded by exploring the relationship between feature complexity and importance, and discovered that the simplicity bias found in neural networks becomes more pronounced training progresses. Surprisingly, we found that important features simplify over time, suggesting a sedimentation process within neural networks that compresses important features be accessible earlier in the network.
Acknowledgments
We thank Amal Rannen-Trine and Noah Fiedel for feedback on the manuscript, and Amal, Robert Geirhos, Olivia Wiles, and Mike Mozer for interesting discussions.
References
[1] Achtibat, R., Dreyer, M., Eisenbraun, I., Bosse, S., Wiegand, T., Samek, W., and Lapuschkin, S. (2023). From attribution maps to humaa--nddersanndaale explanations through concept relevance propagation. Nature Machine Intelligence.
[2] Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., and Kim, B. (2018). Sanity checks for saliency maps. In Advances in Neural Information Processing Systems (NIPS).
[3] Advani, M. S., Saxe, A. M., and Sompolinsky, H. (2020). High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428 446488446.
[4] Allender, E. (1992). Applications of time-bounded kolmogorov complexity in complexity theory.
In Kolmogorov complexity and computational complexity, pages 4-22. Springer.
10

---

[5] Ancona, M., Ceolini, E., Öztireli, C., and Gross, M. (2018), Towards better understanding of gradient-based attribution methods for deep neural networks. In Proceedings of the International Conference on Learning Represeniations (ICLR).
[6] Arjovsky, M., Bottou, L.,, Gulrajani, I., and Lopez-Paz, D. (2019). Invariant risk minimization. arXiv preprint wwt.vv:907.020.
[7] Arora, S., Cohen, N., Hu, W., and Luo, Y. (2019). Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32.
[8] Arora, S., Li, Y., Liang, Y., Ma, I., and Risteski, A. (2018). Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483 495.
[9] Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., and Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. Public Library of Science (PloS One).
[10] Baker, N., Lu, H., Erlikhman, G., and Kellman, P. J. (2018). Deep convolutional networks do not classify based global object shape. PLOS computational biology,
[1]] Banerjee, D., Singh, A., and Singh, G. (2023). Dissecting neural network robustness proofs. In The Twelfth International Conference on Learning Representations.
[12] Bau, D., Zhou, B., Khosla, A., Oliva, A., and Torralba, A. (2017). Network dissection: Quantifying interpretability of deep visual representations. In Proceedings the on Computer Vision and Puttern Recognition (CVPR).
the IEEE Conference
[13] Bauwens, B., Makhlin, A., Vereshchagin, N., and Zimand, M. (2018). Short lists with short programs in short time. computational complexity, 27:31-61.
[14] Beyer, I.., Izmailov, P., Kolesnikov, A., Caron, M., Kornblith, S., Zhai, X., Minderer, M., Tschannen, M., Alabdulmohsin, I., and Pavetic, F. (2022). Flexivit: One model for all patch sizes. arXiv preprint arXiv:2212.08013.
[15] Bhatt, U., Weller, A., and Moura, J. M. F. (2020). Evaluating and aggregating ceature-based model explanations. In Proceedings of the International Joint Conference on Artificial Intelligence (ICCAI).
[16] Blier, L. and Ollivier, Y. (2018). The description length of deep learning models. Advances in Neural Information Processing Systems, 31.
[17] Bouniot, Q., Redko, I., Mallasto, A., Laclau, C., Arndt, K., Struckmeier, O., Heinonen, M., Kyrki, V., and Kaski, S. (2023). Understanding deep neural networks through the lens of their non-linearity. arXiv preprint arXiv:2310. 1439.
[18] Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., al. (2023a). Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2.
[19] Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T. Joseph, N., Hatfield-Dodds, Z. Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Carter, S., Henighan, T., and Olah, C. (2023b). Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread. https:/ttrasffrmmr- circuits. sircuits.uub20233M wwwooosematti--ftttresssseeeeeie
[20] Cantor, G. (1890). Ueber eine elementare frage der mannigfaltigketislerre. Jahresberichl der Deutschen Mathematiker- Vereinigurnnni,nn 1:72-78.
[21] Chaitin, G. (1969). On the length of programs for computing finite binary sequences: statistical considerations. Journal of the ACM (JACM), pages 145-159.
[22] Chaitin, G. (1977). Algorithmic information theory. IBM journal of research and development.

---

[23] Chattopadhay, A., Sarkar, A., Howlader, P., and Balasubramanian, V. N. (2018). Grad-cam Generalized gradient-based visual explanations for deep convolutional networks. In Proceedings of the IEEF/CVF Winter Conference Applications of Computer Vision (WACV).
[24] Chen, Y., Zhou, Z.,, and Yan, (2024). Going beyond neural network feature similarity: The network feature complexity and its interpretation using category theory. Proceedings of the International Conference on Learning Representations (ICLR).
[25] Cheung, B., Terekhov, A., Chen, Y., Agrawal, P., and Olshausen. B. (2019). Superposition of many models into one. Advances in Neural Information Processing Systems (NeurIPS).
[26] Chollet, F. et al. (2015). Keras. https: 10.
[27] Colin, J., Fel, T., Cadène, R., and Serre, T. (2021). What cannot predict, do not understand: A evaluation framework for explainability methods. Advances in Neural Information Processing Systems (NeurIPS).
[28] Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T.,, De- hghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020). An image is worth 16x16 words: Transformers for image recognition scale. In Proceedings of the International Conference on Learning Representations (ICLR).
[29] Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M. and Olah, C. (2022). Toy models of superposition. Transformer Circuits Thread.
[30] Fel, T., Agustin, P., Louis, B., Thibaut, B., David, V., Julien, C., Rémi, C., and Thomas, S. (2023a). Craft: Concept recursive activation factorization for explainability. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[31] Fel, T., Boissin, T., Boutin, V., Picard, A., Novello, P., Colin, J., Linsley, D., Rousseau, T., Cadène, R., and Laurent Gardes, T. S. (2023b). Unlocking feature visualization for deeper networks with magnitude constrained optimization.
[32] Fel, T., Boutin, V., Moayeri, M., Cadène, R., Bethune, L., Chalvidal, M., Serre, T., et al. (2023c).
A holistic approach to unifying automatic concept extraction and concept importance estimation.
[33] Fong, R. C. and Vedaldi, A. (2017). Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
[34] Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. (2020). Sharpness aware minimization for efficiently improving generalization. arXiv preprint arXi::0010001412
[35] Gao, L., la Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, (2024). Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.0444 04093.
[36] Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence.
[37] Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., and Brendel, W. (2019). Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness, Proceedings of the International Conference on Learning Representations (ICLR).
[38] Ghorbani, A., Abid, A., and Zou, J. (2017). Interpretation of neural networks is fragile. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
[39] Ghorbani, A., Wexler, J., Zou, J. Y., and Kim, B. (2019). Towards automatic concept- bancept-based based explanations. In Advances Neural Information Processing Systems (NeurIPS).
[40] Goldblum, M., Finzi, M., Rowan, K., and Wilson, A. G. (2023). The no free lunch theorem, kolmogorov complexity, and the role of inductive biases machine learning. arXiv preprint arXiv:2304.05366.
12

---

[41] Grabska-Barwinske Barwinska, A., Jonnen-Triki, A., Rivasplata, O., and György, A. (2021). Towards better visual explanations for deep image classifiers. In eXplainable Al approaches for debugging and diagnosis.
[42] Graziani, M., Nguyen, A.-p., O'Mahony, L., Müller, H., and Andrearczyk, V. (2023). Concept discovery and dataset exploration with singular value decomposition. Workshhoprooceeding of the International Conference on Learning Representations (ICLR).
[43] Grünwald, P. D., Vitányi, P. M., et al. (2008). Algorithmic information theory. Handbook of the Philosophy of Information.
[44] Hamblin, C., Fel, T., Saha, S., Konkle, T., and Alvarez, G. (2024). Feature accentuation: Revealing's what features respond to in natural images. arXiv preprint arXIvv2402.10033,
[45] He, K., Zhang, X.,, Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR).
[46] Hermann, K., Chen, T., and Kornblith, S. (2020). The origins and prevalence of texture bias in convolutional neural networks. Advances Neural Information Processing Systems (NeurIPS).
[47] Hermann, K. and Lampinen, A. (2020). What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems (NeurIPS).
[48] Hermann, K. L., Mobahi, H., Fel, T., and Mozer. M. C. (2023). On the foundations of shortcut learning. In International Conference on Learning Representations.
[49] Hochreiter, S. and Schmidhuber, J. (1994). Simplifying neural nets by discovering flat minima. Advances in neural information processing systems, 7.
[50] Hooker, S., Courville, A., Clark, G., Dauphin, Y., and Frome, A. (2019). What do compressed deep neural networks forget? arXiv preprint 1911v:
[51] Huh, M., Mobahi, H., Zhang, R., Cheung, B., Agrawal, P., and Isola, (2021). The low-rank simplicity bias in deep networks. arXiv preprint arXiv:2103. /0427.
[52] Humayun, A. I., Balestriero, R., and Baraniuk, R. (2024). Deep networks always grok and here is why. arXiv preprint arXiv:2402. 15555.
[53] Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F,, et al. (2018). Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In Interna- tional conference on machine learning. Proceedings of the International Conference on Machine Learning (ICML).
[54] Kim, S. S. Y., Meister, N., Ramaswamy, V. V., Fong, R., and Russakovsky, O. (2022). HIVE: Evaluating the human interpretability of visual explanations. Proceedings of the IEEE European Conference on Computer Vision (ECCV).
[55] Kolmogorov, A. N. (1963). On tables of random numbers. Sankhyā: The Indian Journal of Statistics, Series A, pages 369-376.
[56] Kolmogorov, A. N. (1965). Three approaches to the quantitative definition of information. Problems of information transmission.
[57] Korbblith, S., Norouzi, M., Lee, H., and Hinton, G. (2019). Similarity of neural network representations revisited, In International conference on machine learning, pages 3519-3529, PMLR.
[58] Lampinen, A. K., Chan, S. C., and Hermann, K. (2024). Learned feature representations are biased by complexity, learning order, position, and more. arXiv preprint arXiv:2405.05847,
[59] Lee, D. D. and Seung, H. S. (1999). Learning the parts of objects by non-negative matrix factorization, nature, (6755))))7888-791
(60] Lee, K., Zung, J., Li, P., Jain, V., and Seung, H. (2017). Superhuman accuracy on the snemi3d connectomics challenge. arXiv preprint arXiv:1706.0012: 00120.
13

---

[61] Lee, Y., Finn, C., and Ermon, S. (2022). Relaxing the kolmogorov structure function for realistic computational constraints. NeurIPS 2022 Workshop on Information-Theorert Principles in Cognitive Systems.
[62] Lepori, M. A., Pavlick, E., and Serre, T. (2023). Neurosurgeon: A toolkit for subnetwork analysis. arXiv preprint arXiv:2309,00 00244..
[63] Levin, L. A. (1973). Universal sequential search problems. Problemy peredachi informatsii, 93):115-116.
[64] Levin, L. A. (1984). Randomness conservation inequalities; information and independence in mathematical theories. Information and Control, 61(1):15-37.
[65] Li, M., Vitányi, P., et al. (2008). An introduction Kolmogorov complexity and its applications, volume 3. Springer.
[66] Linsley, D., Malik, G.. Kim, J., Govindarajan, L. N., Mingolla, E.,, and Serre, T. (2021). Tracking without re-recognition in humans and machines. Advances in Neural Information Processing Systems (NeurIPS).
[67] Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022). A convnet for the 2020s. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[68] Lu, Z. and Oliveira, C. (2022). Theory and applications of probabilistic kolmogorov complex- ity. arXiv preprint arXiv:2205. 14718.
[69] Malhotra, G., Evans, B. D., and Bowers, S. (2020). Hiding a plane with a pixel: examining shape-bizt bias in cnns and the benefit of building in biological constraints. Vision Research, 174:57-68.
[70] McCarthy, J., Minsky, M. L., Rochester, N., and Shannon, C. E. (1956). A proposal for the dartmouth summer research project on artificial intelligence, august 31, 1955.
[71] McCoy, R. T., Pavlick, E., and Linzen, T. (2019). Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv: arXiv:/902. 1902.01077.
[72] Moayeri, M., Banihashem, K., and Feizi, S. (2022a). Explicit tradeoffs between adversarial and natural distributional robustness. In Advances Neural Information Processing Systems (NeurIPS).
[73] Moayeri, M., Pope, P., Balaji, Y., and Feizi, S. (2022b). A comprehensive study of image classification model sensitivity to foregrounds, backgrounds, and visual attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[74] Moayeri, M., Singla, S., and Feizi, S. (2022c). Hard imagenet: Segmentations for objects with strong spurious cues. In Advances in Neural Information Processing Systems (NeurIPS).
[75] Nakkiran, P. (2021). Yuring-universsl learners with optimal scaling laws. arXiv preprint arXiv:2111.0532
[76] Nanda, V., Speicher, T., Dickerson, J., Gummadi, K., Feizi, S., and Weller, A. (2024). Diffused redundancy in pre-trained representations. Advances Neural Information Processing Systems, 36.
[77] Nguyen, A., Yosinski, J., and Clune, J. (2016). Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. Visualization for Deep Learning workshop, Proceedings of the International Conference on Machine Learning (ICML).
[78] Nguyen, A., Yosinski, J., and Clune, J. (2019). Understanding neural networks via feature visualization: A survey, arXiv preprint arXiv:1904.08933
[79] Nguyen, G., Kim, D., and Nguyen, A. (2021). The effectiveness of feature attribution methods and its correlation with automatic evaluation scores. Advances Neural Information Processing Systems (NeurIPS).
14

---

[80] Novello, P., Fel, T., and Vigouroux, D. (2022). Making sense of dependence: Efficient black-bon -ox explanations using dependence measure. In Advances Neural Information Processing Systems (NeurIPS).
[81] Okawa, M., Lubana, S., Dick, R., and Tanaka, H. (2024). Compositional abilities emerge multiplicatively: Exploring diffusion models on synthetic task. Advances in Neural Information Processing Systems (NeurIPS).
[82] Olah, C., Cammarata, N., Schubert, L.,, Goh, G., Petrov, M., and Carter, S. (2020a). An overview of early vision in inceptionvl. Distill.
[83] Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. (2020b). Zoom in:
An introduction circuits. Distill. http:///isillllpp///////cciiiiis
[84] Olah, C., Mordvintsev, A., and Schubert, L. (2017). Feature visualization. Distill.
[85] Park, C. Okawa, M., Lee, A., Lubana, E. S., and Tanaka, H. (2024). Emergence of hidden capabilities: Exploring learning dynamics in concept space. arXiv preprint arXiv:2406. 19370.
[86] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., al. (2011). Seikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830.
[87] Pérez, G. V., Camargo, C. Q., and Louis, A. A. (2018). Deep learning generalizes because the farameter-function map is biased towards simple functions. stat, 1050:23.
[88] Petsiuk, V., Das, A., and Saenko, K. (2018). Rise: Randomized input sampling for explanation black-box models. In Proceedings of the British Machine Vision Conference (BMVC).
[89] Plaza, S. M. and Funke, J. (2018). Analyzing image segmentation for connectomics. Frontiers neural circuits.
[90] Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., and Dosovitskiy, A. (2021). Do vision transformers see like convolutional neural networks? Advances neural information processing systems.
[91] Rahaman, N., Baratin, A., Arpit, D., Draxler, F. Lin, M., Hamprecht, F., Bengio, Y., and Courville, A. (2019). On the spectral bias of neural networks. In International conference on machine learning. PMLR.
[92] Rajamanoharan, S., Lieberum, T., Sonnerat, N., Conmy, A., Varma, V., Kramár, J., and Nanda, N. (2024). Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407. 14435.
[93] Trinen-Triki, A., Berman, M., Kolmogorov, V., and Blaschko, M. B. (2019). Function norms for neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops.
[94] Schmidhuber, (1995). Discovering solutions with low kolmogorov complexity and high generalization capability. In Machine Learning Proceedings 1995, pages 488 496. Elsevier.
[95] Schmidhuber, (1997). Discovering neural nets with low kolmogorov complexity and high generalization capability. Neural Networks, 10(5):857-873.
[96] Schubert, I.,, Voss, C.,, Cammarata, N., Goh, G., and Olah, C. (2021). High-low frequency detectors. Distill. http:///isllllluooooooirrriiiiid
[97] Shah, H., Tamuly, K., Raghunathan, A., Jain, P., and Netrapalli, P. (2020). The pitfalls of simplicity bias in neural networks. Advances Neural Information Processing Systems (NeurIPS), 33.
[98] Shen, A., Uspensky, V. A., and Vereshchagin, N. (2022). Kolmogorov complexity and algorith- mic randomness, volume 220. American Mathematical Society.
15

---

[99] Simonyan, K., Vedaldi, A., and Zisserman, A. (2013). Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop, Proceedings of the International Conference on Learning Representations (ICIR).
[100] Singla, S. and Feizi, S. (2022). Salient imagenet: How to discover spurious features in deep learning? In Proceedings of the International Conference on Learning Representations (ICLR),
[101] Slack, D., Hilgard, A., Lakkaraju. H., and Singh, S. (2021). Counterfactual explanations can be manipulated. Advances in Neural Information Processing Systems (NeurIPS).
[102] Smilkov, D., Thorat, N., Kim, B., Viégas, F., and Wattenberg, M. (2017). Smoothgrad: removing noise by adding noise. In Workshop on Visualization for Deep Learning, Proceedings of the International Conference on Machine Learning (ICML).
[103] Solomonoff, J. (1964). A formal theory of inductive inference. part i. Information and control.
[104] Solomonoff, R. (2009). Algorithmic probability: Theory and applications. Information theory and statistical learning, pages 1-23.
[105] Springer, M., Nagarajan, V., and Raghunathan, A. (2024). Sharpness- aware minimization enhances feature quality via balanced learning. In The Twelfth International Conference on Learning Representations.
[106] Sundararajan, M., Taly, A., and Yan, Q. (2017). Axiomatic attribution for deep networks. In Proceedings of the International Conference on Machine Learning (ICML).
[107] Tachet, R., Pezeshki, M., Shabanian, S., Courville, A., and Bengio, Y. (2018). On the learning dynamics of deep neural networks. arXiv preprint arXiv: 06848.
[108] Teutsch, J. (2014). Short lists for shortest descriptions in short time. computational complexity, 23:565-583.
[109] Trockman, A. and Kolter, J. Z. (2023). Patches all you need? Transactions Machine Learning Research.
[110] Valle-Perez, G., Camargo, C. Q., and Louis, A. (2018). Deep learning generalizes because the purameter-function map is biased towards simple functions. In International Conference on Learning Representations.
[111] Vasudeva, B., Fu, D., Zhou, T., Kau, E., Huang, Y., and Sharan, V. (2024). Simplicity bias of transformers learn low sensitivity functions. arXiv preprint arXiv:2403. 06925..
[112] Vielhaben, J., Blücher, S., and Strodthoff, N. (2023). Multi-dimensional concept discovery (med): A unifying framework with completeness guarantees.
[113] Wang, Y.-X. and Zhang, Y.J. (2012). Nonnegative matrix factorization: A comprehensive review. IEEE Transactions on knowledge and data engineering, 2366):1336
[114] Wightman, R., Touvron, H., and Jégou, H. (2021). Resnet strikes back: An improved training procedure in timm. arXiv preprint 2rXiv:2110.0044
[115) Xu, Y., Zhao, S., Song, J., Stewart, R., and Ermon, S. (2019). A theory of usable information under computational constraints. In International Conference on Learning Representations.
[116] Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z.-H., Tay, E., Feng, and Yan, S. (2021). Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision.
[117] Zeiler, M. D. and Fergus, R. (2014), Visualizing and understanding convolutional networks. In Proceedings of the IEEE European Conference on Computer Vision (ECCV).
[118] Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2021a). Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115.
[119] Zhang, R., Madumal, P., Miller, T., Ehinger, K. A., and Rubinstein, B. I. (2021b). Invertible concept-based explanations for enn models with non-negative concept activation vectors. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI).
16

---

A Feature Extraction
Dictionary Learning. To comprehensively analyze the complexity of features extracted from deep learning model, we employed a detailed feature extraction process using dictionary learning, specifically utilizing an over-complete dictionary. This approach allows each activation fx (x) A to be expressed as linear combination of multiple basis elements (direction, also called atoms) E A& from the dictionary D {di..... ] coupled with some sparse coefficient E R associated to each atoms.
The over-completness of D* means that the dimension of the dictionnary (k) is larger than the
dimension of the activations space k >>> |Ay] This property allow us to overcome the superposition problem [29] essentially stating that there be more feature than neurons.
Mathematically, given an activation function fx(x), it can be represented as a linear combination of atoms from the dictionary D, expressed as:
fn(x) R2 2D
Σ z,d,
where are the coefficients indicating the contribution of each atom d: from the dictionary.
Implementation. Our implementation was inspired by Craft [30], leveraging the properties of ReLU activations in ResNet50. Given that ReLUs induce non-negativity of the activation, we employed Non-Negative Matrix Factorization (NMF) [59, 113] for the reconstruction, aligns with the sparsity and non-negativity constraints of ReLU activations. Unlike PCA, which cannot produce overcomplete dictionaries and may result non non-positivy non--postive activations, NMF can create overcomplete dictionaries in this context.
naturally
The dictionary D* was trained to reconstruct the activations fx(() using the entire ImageNet training dataset, comprising 1.2 million images. Formally, for the set of images X and their corresponding activations fn (X), the objective was minimize the reconstruction error:
(x) ZD l|Fs
ensuring that (X) can closely approximated by ZD* Additionally, the NMF framework enforces non-negativity constraints the dictionary matrix D 2 and the coefficients Z 2 0:
(Z, D*)
arg min (X) ZD*
Z20.D'20
The dictionary D was designed to encapsulate 10 concepts per class, resulting in a total of 10,000 concepts. To augment the training samples for NMF, we exploited the spatial dimensions of the last layer of ResNet50, which has 2048 channels with spatial resolution of 7x7. By training the NMF independently on each the 49 spatial dimensions, we effectively increased the number of training samples to approximately 58 million artificial samples (channel activations).
We utilized the block coordinate descent solver from Scikit-learn [86] solve the NMF problem. This algorithm decomposes the problem into smaller subproblems, making it more tractable. The optimization process continued until convergence was achieved with a tolerance of & 10 ensuring the dictionary was sufficiently optimized for accurate feature extraction. Post-training, the reconstructed activations ZD* retained over 99% accuracy common predictions compared the original activations f. (X).
Extracting Features for New Data Points. Once the dictionary D was trained, was fixed. For any new input x. the corresponding feature z was extracted by solving a Non-Negative Least Squares (NNLS) problem. This mapping of new input activations fx) to the learned feature space was performed by minimizing the following objective:
arg minlm fx) (x) =D-
220
This optimization problem is convex, ensuring computational feasibility and robust feature extraction for new data points.
17

---

B Visualization of Meta-features
To visualize each feature, used feature visualization methods [84]. Specifically,, recent improve- ment [31] re- parameteriizss the original feature visualization optimization problem i.e., finding an image x that maximizes direction, channel, neuron by optimizing only the phase of Fourier buffer while fixing the image magnitude. This approach produces more realistic images and prevents high-freen high-frequency leakage that results adversarial positives.
Sky
Grass
Bokeh
Trees
Leaves
Low Pixel quality
Rocks
Fences
Wheels
Trademarks
Humens
Figure 7: Visualization of Meta-Features, sorted by Complexity. Feature visualization using MACO [31] for the most simple (1-15) the 30 Meta-features found on the 10,000 features extracted.
18

---

Hi-fi
Fingers
Eyes
Ovaloide
Furs
Textile wrinides
Noses
Curves
Ears
Whiskers
Insect legs
Figure 8: Visualization of Meta-Features, sorted by Complexity. Feature visualization using MACO [31] for for the most complex (15-30) of the 30 Meta-features found the 10,000 features extracted.
19

---

C
Complexity measure
In this section, we detail the closed-form expression of the V-information when the predictive family V consists of linear classifiers with Gaussian posteriors. Specifically, V is defined as follows:
((((x)), )² with x E X and sh V V + (M, o²), with μ E R. o² 2
where V - + Mx| M R is set of linear predictors. This setting corresponds to the linear decoding we apply during the computation V-information. In this context, boseed-form solution is available (see [115]):
(x(x 2) = 1lv(2) Hv(z x)
(M-W) 20
(x)) 2
inf BEMPe PER
log
inf Ex P NEW
log
V 22T02
v 12TT2
(z 11)2)
(2 (x
inn BzuP, PER
inf Ex Ex,zzr NEW
20²
20²
1
inf BeNPe Pe [(2-m1)² NEW
inf ExIza [(z - (b()))
20²
QNER
Var(z)
infwev Ex,e~ [(2 - (x())
1
20²
Var(z)
Var(z)
R²2
20
-
Here, R is the coefficient determination. Therefore, the following inequalities hold:
0 S Tv 2) s Var(z).
Given that the input data are centered and scaled, we typically have Var(z) around (or less in case of Layer normalization). Furthermore, residual connections and batch normalization tend preserve this scaling in deeper layers. We note, however, that our score K(2, x) is not strictly bounded. Indeed, we define complexity as the opposite of the average V-information across layers: complex features are those that are harder to decode. We add a shift of 1 for the ease of plotting. Empirically, we observed that this adjustment yields K(z, x) in the range [0, 1], with indicating a complex feature that is not available and 0 indicating simple feature that is fully available.
20

---

D Feature Support Theory
While simplifying important features underscores trend toward computational efficiency, the role of complex features within the model deserves a closer examination. Despite these features often being deemed less important directly, they contribute significantly to the model's overall performance, paradox that can lead us to introduce the concept of "support features." These are set of features that may not carry substantial importance individually, but that collectively play a crucial role model's s decision-making process.
1.0
0.94
0.8
0.77
Relative Accuracy
0.6
0.4
0.2
0.0
100
75
50
25
Proportion of kept concepts
Figure 9: "Support Features" hypothesis. The majority of complex features are very important, but play a non-negligible role and contribute to significant performance gains. This paradox is referred as the "support features," large ensemble of features individually little very little importance to the model but collectively holding significant role.
The presence of numerous complex features, whose importance on average is less pronounced, poses conundrum. However, these features are far from redundant. Experiments conducted by progressively removing the most complex concepts from the model demonstrate noticeable impact on performance, illustrated Figure This empirical evidence supports the theory that, while individually, these complex features may not be pivotal, their collective presence contributes indispensably to the robustness and adaptability of the model. These results are reminiscent of prior findings that low-importance model components that are removed in pruning may nevertheless contribute model accuracy on rare items [50].
This observation aligns with the broader understanding of neural network functionality, where diversity feature representation- spannnnngnnnn from simple to to generalize and perform across varied datasets and tasks. Therefore, the "Feature Support Theory" underscores an essential aspect of neural network design and training: integrating and preserving wide spectrum of features, regardless of their individual perceived importance, are vital for achieving high levels of performance and robustness.
the model's ability
21

---

E Complexity and Redundancy
To further understand the link between feature complexity and redundancy, we utilized the redundancy measure from [76]. Our findings indicate that complex features tend to be less redundant, as depicted Figure 10. This observation aligns with the strong correlation between our complexity measure and the redundancy-basee complexity measure proposed by [24].
1.00
0.98
0.96
Redundancy
0.94
0.92
0.90
0.88
p -0.184, p 1e-5.
0.3
0.4
0.5
0.6 Complexity
0.7
0.8
Figure 10: Complex features are less redundant. Using the redundancy measure from [76], show that our complex features tend to be less redundant. This result also confirms a link between complexity measure and the one recently proposed by [24], which is also based on redundancy.
To quantify redundancy, [76] employed modified version of Centered Kernel Alignment (CKA) [57], a measure similarity between two sets of activation features. We briefly recall that CKA between two set of activations A, B in Rd is defined as follows:
KAK 1
CKA(A, = KAKAAlP KEKBBIF
where KA and KB are the Gram matrices of the feature activations A and B, respectively, and || ||r denotes the Frobenius norm.
In our analysis, we calculated the CKA measure between a feature and the activations for a set of 2,000 images from the validation set fn (X). Subsequently, we compared this with the CKA measure when a portion of the activation is masked using binary mask as CKA(Z, (X) @ m.), where @ represents element-wise multiplication (Hadamard product). This comparison enabled us assess whether masking subset of neurons impacts the decoding of the features. Specifically, to evaluate redundancy, we employed progressive masking strategy, successively masking 10%. 50%, and 90% of the activation. If the masked activations retain a high CKA with it indicates that the information remains largely intact, suggesting that the feature redundant encoding mechanism within
{0, (Ael, denoted
redundantly stored across multiple neurons, sign the network. Conversely, if masking results in substantial decrease in CKA, it implies that the information was predominantly localized on specific neuron. this scenario, the feature is not redundantly encoded but rather concentrated in specific neurons. This concentration indicates lower degree of redundancy, as the loss of these specific neurons (throught the masking) leads to significant reduction in the CKA score.
22

---

The final score of redundancy is then the average CKA difference between the original activation and the masked activations:
Em (CKA(f((X) @ mmmm) CK)(fx(X),
Redundancy
And averaged across the different level of masking. A high score (1) indicating a high redundancy Lee the CKA between the masked activation and with the original activation is similar while low score indicate a more localized and thus lower degree of redundancy.
In summary, our results, as depicted in Figure 10 support the idea that complex features exhibit lower redundancy.
23

---

Complexity and Robustness
0.0014
== 0.321, p < 1e-5
0.0012
0.0010
Sensitivity
0.0008
0.0006
0.0004
0.0002
0.0000
0.3
0.4
0.5
0.6 Complexity
0.7
0.8
Figure 11: Complex features are less robust. This figure illustrates the relationship between feature complexity and robustness, quantified as the variance of the feature value when the image is perturbed with Gaussian noise. The results indicate that more complex features tend to exhibit lower robustness.
To measure robustness, we evaluate the stability of feature responses under perturbations. For each input point x, we add isotropic Gaussian noise with varying levels of standard deviation a. The robustness score is determined by measuring the variance in the feature response due to the noise. Formally, let 2(x) represent the feature response for input x. We define the perturbed input as: x = (O, 1) where (0,, 2²1) represents Gaussian noise with mean 0 and variance o². The sensitivity score Sensitivity(z) for a feature z given by:
Sensitivity(z) Var(z(X))
Specifically, we sample 100 random noise and repeat this for 3 levels of noise a E (0.01, 0.1, 0.5) to compute the variance in feature response for each input from 2,000 samples from the Validation set of ImageNet to get distribution of feature value. We also consider other metrics such as the range (min-max) of feature response, but all methods consistently indicate that more complex features are less robust.
In summary, our results, as shown in Figure 11, demonstrate that complex features exhibit lower robustness. This indicates that features with higher complexity are more sensitive to perturbations and noise, resulting greater variability their responses.
24

---

G Importance Measure
"Promoting" concept "Inhibiting" concept
0.4
0.2
Importance
0.0
-0.2
-0.4
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Complexity
Figure 12: Inhibiting and non-inhibiting features complexity. Important features be significant either by inhibition, i.e., removing information from a class, or by adding information for given class. Each point represents feature, and violet-colored features generally act as inhibitors (T(),) < 0).
The problem of estimating feature importance is closely related to attribution methods [117, 33, 88, 80, 15, 102, 106, 23], which aim to identify the important pixels for a decision. A recent study has shown that all attribution methods be extended in the space of concepts [32]. In our case, the features are extracted from the penultimate layer, where the relationship between feature values and logits is linear. We will elaborate this and demonstrate that the notion of importance in the linear case is easier and optimal methods to estimate importance exist.
Setup. Recall that for point x,
obtain its l feature values by solving NNLS problem z = arg min [x) 2D* The vector Z contains the k features R& We can replace the activation of the penultimate layer (x) with its feature representation in the over-complete basis zD 2 fr. (x). Since we are in the penultimate layer, the model's decision, i.e., the logit E IR for the predicted class, is linearly related each feature z by the last weight matrix, denoted as W, as follows:
W zD'W with arg min [[ 2D" ZW' with W' D'W R
(3)
(5)
Thus, the energy contributed to the logit by feature i can be directly measured by W/z, and y ΣA W/z. Consequently, the contribution of feature z; can be measured using gradient-input. (Zy) @ z. Several studies [5, 32] have detailed the linear case and shown the optimality of gradient- input with respect to fidelity metries. They also demonstrated that many methods in the linear case boil down to moddient-Inputt including Occlusion [117], [88) and Integrated Gradient 106].
In case, measured the importance by taking the absolute value of the importance vector, i.e., T(z,) = Er, (11 dy
0z; 2;ll) It is natural to question whether this approach might overlook important features due their inhibitory effects. Indeed, as depicted in Figure 12, numerous features may be important not because they add positive energy to the logits, but by inhibition, i.c., by suppressing
25

---

class information. Although this does not alter the implications of our previous observations, it is noteworthy that the majority of inhibitory features are also simple features.
Prevalence and Importance. Another property of importance is its close relationship with preva- lence [32], which indicates that frequently occurring feature will, on average, be more important given the same importance coefficient (V=y). In our study, this implies that if the most important features are reduced, these important features are also potentially more frequently present. Conse- quently, the prevalence of feature can be factor explaining this sedimentation process. We refer the reader concurrent study that proposed to investigate more deeply this phenomena using controlled dataset [58].
26

---

H
Features Clustering
The visualization in Figure 2 prompts an important question: are features in our model clustered based on their complexity? Specifically, are there regions in the feature space that are generally more "complex" than others, or that respond primarily features via overcomplete decomposition enables more detailed analysis compared to traditional neuron- wise studies. We aim to explore three main hypotheses:
more complex stimuli? Our access to 10,000
Hypothesis 1: Features cluster by super-class. This hypothesis posits that features
corresponding to semantically related categories are spatially grouped within the feature e.g. concepts related to the dog class are closer those of the cat class than to unrelated classes like furniture.
Hypothesis 2: Features cluster by complexity. We suggest that features may organize themselves based on their complexity, with simpler features forming distinct clusters separate from more complex ones.
Hypothesis 3: Features cluster by importance. This hypothesis explores whether features with similar predictive importance tend group together within the feature space.
It is important to emphasize that these hypotheses mutually independent. To test them, we propose two methodologies: (1) visualizing feature embeddings using UMAP and (2) clustering the features followed by dendrogram analysis to examine whether the resulting clusters are homogeneous (also called "pure") in terms of super-class, complexity, or importance.
27

---

Supertiarees
Dogs
iwveterbrates
Cloth es, covering
Other mammals
Food, plants, fungi
Implements. containers, misc. objects vehicles
Birds
Devices
Reptiies, amphitions
furnishing
Figure 13: Feature Similarity vs Super-Class. Each point represents concept, with its color indicating the associated super-class. Some super-classes such as birds, reptiles, dogs & other mammals form well-defined, tight clusters, suggesting that features belonging to them are close in the feature space. Others, such as device, clothes appear more dispersed. By comparing this figure with Figure 2, we can identify which meta-features are "pure" (belonging to single super-class) and which are "impure" (spanning multiple super-classes). Interestingly, the "impurity" region seems to cover lowplempyyxtty and mid-complexity concepts such that Grass, Waves, Trees, Low-pixel quality detector which are not class-specific.
Feature Similarity vs Super-Class Figure 13 illustrates the organization of features by their super-class. Each point a feature, colored according to its super-class label. We observe that certain super-classes, like those associated with birds, dogs, reptiles form distinct and cohesive clusters, indicating a strong grouping within the feature space. Other region have features that encompassing wider range of super-class, such that grass, waves, low-pixel quality detector. This reveals that some meta-features are predominantly associated with a single super-class ("pure"), while others span multiple super-classes ("impure"), reflecting the shared visual characteristies multi-fun functional nature of those features.
28

---

Complexity of concepts
0.86
Average complexity dendogram
250
200
Distance
150
100
Cluster average complexity
Figure 14: Feature Similarity by Complexity. A) Each point is a feature, colored by its complexity score. Distinct areas the graph correspond to varying levels complexity, suggesting a non- random distribution of feature complexity. For instance, animal-related features tend have higher complexity, one could hypothesize that the fine-grained classification required for these categories responsible for this complexity. B) A four-level dendrogram where each level further segments clusters and calculates the average complexity for each sub-cluster. A clear split by complexity appears the first level and intensifies with depth, supporting the idea that some regions of the feature space are inherently more complex than others.
Clustering by Complexity. Figure 14 explores how features organized based on their complex- ity. Panel shows a UMAP visualization with points colored according to their complexity scores. The visualization reveals distinct regions of varying complexity, indicating that the distribution is structured. For instance, features related to animals display higher complexity, likely because these require fine-grained and precise detectors. In contrast, simpler features, such as color detectors low-frequency patterns, cluster together in less complex regions. Panel B displays a dendrogram with four hierarchical levels, where each level introduces additional splits, and the mean complexity is calculated for each sub-cluster. The pronounced division in complexity at the first level, which sharp- as delve deeper, suggests that the feature space compartmentalized based complexity. promising direction for future research would be to align these complexity clusters with known visual cortical areas to explore potential correspondences.
29

---

Importance of concepts
0.39
Average Importance Dendogram
300
250
200
Distance
150
100
50
Cluster average importance
Figure 15: Feature Similarity by Importance. A) Each point represents feature, with color indicating its importance. Distinct regions of the graph contain features of varying importance, particularly with more important features clustering at the top. B) four-level dendrogram with sub- clusters evaluated by their average importance. We observe that from the first level, the dendrogram effectively splits features into groups of varying importance, corresponding the upper and lower parts of the UMAP graph in Panel A.
Clustering by Importance. Finally, investigate the third hypothesis: the presence of regions within the feature space that contain features with higher predictive importance. Figure 15 Panel A depicts UMAP visualization where features colored by their importance. The graph shows a clear structure, with highly important features grouping in specific regions (e.g., the upper part of the graph), while less important features distributed in other regions. Panel B provides the corresponding dendrogram, revealing that even at the first level of the hierarchy, features segregate into clusters of varying importance. Notably, low-complexity "support" features-such as grass, waves, and low-pixel-quality detectors-tent to form cohesive clusters. Meanwhile, more predictive features, like animal-related features that drive classification, group together in another distinct region. This raises crucial question: is this clustering merely correlated (i.e., based on shared visual aspects like context or background), or does reflect causal relationship the model's predictive structure? This question remains an open avenue for future investigation.
30

---

I Local vs Distributed Encoding
Neural networks exhibit diversity in how features are encoded, ranging from local to distributed representations [25, 29]. In the local encoding scenario, single neuron is primarily responsible for encoding a feature. On the other hand, in a distributed encoding scheme, features are represented by the coor- dinated activity of multiple neurons, often de- viating from canonical axis-aligned directions. More specifically, features could be distributed across many sauroon--ommtimeee densely, or in pseudo decuro-distributed fashion-raie rather than be- ing localized to a specific neuron. This the- ory is one of the main motivations behind em- ploying overcomplete concept extraction meth- ods [18, 32, 92, 35].
Hoyer score of concepts
0.95
The distinction between local and distributed en- coding is critical, particularly when extracting overcomplete features from a large-scale dic- tionary. In our analysis of 10,000 features, we assess whether these features are encoded local distributed manner. A local encoding would imply that the feature vector d is aligned with canonical vector, i.e., d E {ee,..., en where [Ae] is the dimensionality of the activation space and e; represents the one-hot canonical vector for the ith neuron. In contrast, fully distributed feature would be character- ized by non-zero values across all neurons, with no alignment any single axis.
0.28
Figure 16: Local vs Distributed Encoding, Each point represents a feature, with color indicating its Hoyer score. Higher scores suggest more "lo- cal" representation, where a feature is primarily encoded by single neuron. Lower scores indicate a distributed representation across a population of neurons. Interestingly, some features have scores near 1, implying near-complete localization, while others more distributed. This variation high- lights the diversity in encoding across features.
To quantify the extent of this local or distributed encoding, we use the Hoyer score. This score, which ranges between 0 and 1, captures the spar- sity of vector by comparing its -norr to its fz-norm, with a correction term for normalization. Formally, the Hoyer score given by:
Hoyer(d) Vn /d|l//1lda
For each feature in our dictionary D = {di,..., ..., d10,000). we compute the Hoyer score to determine whether the feature is locally encoded (score near or distributed (score near 0).
Figure 16 illustrates the results of this analysis, revealing significant variability in the degree of distribution among features. Some regions of the feature space show highly distributed encoding, while others exhibit more local representations. This analysis was conducted on a ResNet50 model, and it possible that the distribution of encoding strategies varies across different architectures.
31

---

30
25
Hoyer
20
15
10
5
p 0.031, p = 0.0051
0.3
0.4
0.5
0.6
0.7
0.8
Complexity
Figure 17: Feature Complexity Distributed Encoding. We show that there is no clear rela- tionship between feature complexity and the degree of distributed encoding. Whether feature is encoded by single neuron or distributed across multiple neurons does not seems to be determined by its complexity.
32

---

J Replicating the feature flow results with other measures and models.
Information flow easy concepts
Information flow for complex concepts
0.7
residual
residual
MAciivations Concepts)
|Activations Concepts)
0.30
main
main
0.6
0.25
0.5
0.20
0.15
0.4
0.10
0.3
0
3
1
4
Figure 18: Replication of Figure 4 Using V-Information. As described in Section replicate the analysis of feature flow and complexity using V-information as a measure.
A legitimate consideration arises when revisiting Section 4, where we introduced the hypothesis that simpler features are primarily carried through the residual connections of network, while more complex features are progressively constructed through interactions between the main branch and residual connections. The measure we originally used to support this hypothesis was CKA, which serves proxy assess the similarity between the activations at a certain stage of the model and the final state of a concept. However, one might wonder why not use V-information directly.
Figure 18 presents the results of this replication. Although the scale of the values differs from those in Section 4, the overall trend remains consistent. The left panel shows the dynamies of simpler features. Early in the network, the main branch carries a significant portion of the V. information, which diminishes as the simpler features are gradually "passed along" through the residual connections. Notably, even the information content decreases, the absolute quantity of V-information remains higher for simple features compared to complex ones. This indicates that while simpler features are transported through the residual connections, they are not entirely depleted of their information content. The right panel of Figure demonstrates the progressive construction of more complex features. Here, we that both the main branch and residual connections contribute to the gradual accumulation information necessary for these complex features. This supports our initial hypothesis: complex features do not traverse the network intact but are built up in a cumulative process, drawing on multiple layers and branches to form intricate representations.
For compute consideration, this replication was conducted using different experimental setup than that of Section For this analysis, we employed the validation set of ImageNet to build the dictionary instead of the train set. Both the dictionary and the model were different from those used in the main body of the paper. Specifically, the model used here was the ResNet50 implementation from the Keras library [26]. Despite these differences in experimental conditions, the overarching trends observed in our original CKA-based analysis are preserved, bolstering the validity of the flow hypothesis.
K Kolmogorov, Levin and V-information
In this section we recall some of the most important complexity measures like Kolmogorov complexity. its computationally tractable counterpart the Levin complexity, and finally we underline the epistemic similarity between these concepts in deep learning.
a measure of the complexity of an object. The objects (image, video, text, pdf, etc.) can be ecnoded as a sequence (u,)) E D of symbols over a finite alphabet Σ. A program a finite sequence P E written in language Kolmogorov complexity (os) (u,)) is the length of the shortest program P N + Σ" that produces
Kolmogorov complexity [55]
Σx (e.g Python source file).
There exists numerous variants of this definition with slightly different behaviors [65]. Since deriving theoretical results is not the focus of our work, we decided to tradeoff precision for simplicity of exposure.
33

---

the 1-th first terms of the sequence (un):
k=000) (un)
min |P|- P(n)=a.
(6)
Intuitively, if the sequence is highly compressible, the program will be short. For example, the sequences 2, .....] . or [2, 4, 8, 16, .2. are few lines of code in most programming languages. Conversely, if the sequence is purely random, then no finite-length program exists. The digits of æ, seemingly without structure, are not Kolmogorov random since there exist short programs computing them. The famous Cantor's diagonal argument [20] shows that most sequences are random, since no bijection exists between E' (countably infinite) and Σ (cardinality of the continuum). The definition implicitly assumes specific computation model (Python interpreter, C++ compiler, Turing machine) to describe the language. However, by definition Turing-complete models can simulate each other, which implies there exist universal constant C(Python C++) such that for all sequences Un we have K((o) (u,,) S K((a) (u,) + C(Python(C++) This constant corresponds to the length of a Python interpreter written in C++ for example. In general, this holds for any other pair of languages.
Therefore, if k//00/ (un) + too as + 0o for some language L, then it is true in every other language: intrinsic randomness universal this sense [98].
Levin complexity. Kolmogorov's 's complexity suffers from an important drawback: it is not Turing- computable. Put another way, there exists no algorithm that computes K((x), Fortunately, by regularizing ((x)) appropriately is possible to make it computable. Levin [64] proposed to regularize the cost with the runtime T(P,)) of program P on input This is the Levin complexity:
K
(T) L (un)
min P(
[P] log T(P.n).
This modification makes (TT) (u,, L) computable with the Levin Universal Search algorithm (see Alg. 1). Informally, instead of looking for shortest program, this algorithm seeks algorithms that run fast among those who are shorts. It is obtained by iterating over lengths E N, and by running exactly one step of computation of all these programs in parallel. The first program P that halts on un minimizes (T)) This is central property of Levin's 's universal search: the first programs found are the simplest and the ones requiring the lesser compute [4, 108, 13].
Algorithm Levin Universal Search
Input: sequence (u,) E Σ*
Output: program P minimizing K (T)
1: S 20
2: for E do
3: for each program P E (E' 0 L.) do
4: S (P
5: for each P S parallel do
6: Run P for exactly I step.
7: P halts on U, then
8: return P
Deep learning and simplicity bias. The links between algorithmic information theory and deep learning have been recurring although spurious interest throughout the years [94, 95, 75, 61, 68, 40]. Neural networks are a special kind program, composed of the source files required for inference. and the weights embedded in the network. Therefore, results related to the complexity of sequences apply transparently. Program length (Kolmogorov) and program runtime (Levin) are tightly linked since deeper and wider networks also consume more FLOPS during inference. Smaller networks implement simpler programs. Similarly, features that can be decoded "early" in the network are simpler than those requiring all the layers. The idea is often coined as Minimum Description Length (MDL) principle [16], Occam's 's Razzor, or even simplicity bias [49].
As for many things in machine learning, regularization helps.
34

---

L Limitations
Task. Here, we have studied specific CNN architecture, ResNet50. In future experiments, will be useful to investigate whether other model families exhibit similar feature learning, including in domains beyond vision.
Architecture. The residual connections of the ResNet are shared by other architectures like Con- vNext [67] or Vision Transformers (ViT) [28, 14]. The works of [109] indicate that findings from convolutional models may transfer to ViTss. Furthermore, the work of [116] suggest that features in convolutional networks and ViT are of similar nature. However, [90] found significant quantitative differences between the layers of ResNets and ViTs, highlighting the need for further empirical testing.
Training Dynamics The observed dynamics of feature learning, including the emergence of complex features and the reduction the complexity of important features later in training, are based specific training schedule and set of typeepaaameters. To accurately attribute these findings, more comprehensive study is required to evaluate the role of various factors such as the learning rate scheduler and weight decay. Future research should systematically investigate how these and other training parameters influence feature complexity and importance.
Nested predictive families. Our complexity metrics rely the hypothesis that the different predictive families associated to the network up to depth fx) are nested, i.c. that stacking more layers strictly increases expressiveness. This is highlighted in the relevant assumption in section 2. If this hypothesis is violated, the true complexity of feature may be overestimated in deeper layers. This is typically the case at the early stages of training.
Dictionary of features. Regarding the building of the dictionary using NMF, a previous study [32] has shown that the specific dictionary learning method yielded favorable tradeoff between several criterions such as sparsity, reconstruction error, or stability. Other dictionary learning methods (like sparse-PCA, K-Means sparse auto-encoder) may yield bank of concepts with different properties.
35

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We support every claim with the corresponding experiment which compute well-defined metrics, correlations, and p-values.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
the paper relies on large empirical study, that was specifically performed on the ResNet50 architecture and the Imagenet dataset. This limitation is clearly specified in the Section 2. Furthermore, the relevance of V-Informatiio to analyse the expressiveness the network at difference depth assumes that expressiveness monotically increases with depth. This assumption is formalized in paragraph Assumption page 5. Finally, a last "Limitations" section is placed in Appendix
Justification:
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: No new theoretical results claimed.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: we rely existing protocols train the models, as specified in Section which includes all processeig, learning rates, architectures, and other common hyperpa- rameters.
Furthermore, the metrics introduced are existing metries of literature for which standard implementations are available.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: the dataset ImageNet be accessed through attps://wxz http:://wwwiinngonn/ee image-net. org. However the code is not available for review at time of submission.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: see answers above.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We compute correlation coefficients between metrics, and we also report the corresponding p p-lauue for statistical significance in every case.
36

---

8. Experiments Compute Resourees
Question: For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [No]
Justification: the cost re-training the model can be estimated in the relevant literature. Computing the complexity scores involves solving (high dimensional) linear regression every depth and every epoch of interest for each of the features.
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:/ htte:://neurippe
wtwpettithisieeeliiieeeeeeeeeeee ines?
Answer: [Yes]
Justification: after close examination of the Code of Ethics, there are no concerns associated study.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: the goal of this empirical study is understand phenomenons arising during the training of deep neural networks. No new algorithm nor dataset is proposed, that could have had direct societal impact.
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes)
Justification: we rely exclusively on the ImageNet dataset, more specifically the sub- set of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) whose license can be accessed at this URL: https: //www//wwwwww/ kaggle cow/competitionn..cmm thangnee--bjeett-lallzzaei
13. New Assets
Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
37


Reviews:
[{"review_id": "f7VLvWO1kQ", "reviewer": "Reviewer_5SRh", "summary": "This paper proposes a new measure of feature complexity based on an information-theoretic metric, the $\\nu$-information metric. Utilizing this complexity measure, the paper shows (1) visualization of features of different complexities, (2) simple features are propagated through the residual connections to reach the final layer, (3) simple features are learned earlier than complex features during training, and (4) simple features usually have a higher importance (weight) to the output score of the model.", "questions": "1.\tThe paper claims that simple features tend to propagate through the residual connection, while complex features tend to propagate through the main branch of the network. Then a natural question arises: how are simple and complex features propagate in CNNs without residual connections (e.g., AlexNet, VGG)?\n\n2.\tAbout the visualization of simple and complex features in Appendix B. The feature *fences* is among the most simple features, while the features *whiskers* and *dotted texture* are among the most complex features. However, I do not see an essential difference between the features *fences*, *whiskers*, and *dotted texture* from the sample images in Figure 7 and Figure 8. For example, *fences* and *dotted texture* both seem to consist of lines that cross over each other to form holes. Similarly, the *whiskers* feature is also composed of lines, although the lines do not cross over each other but appear parallel. Why is *fences* a simple feature, but *whiskers* and *dotted texture* complex features?\n\nAnother question is that previous studies show that CNNs are usually biased towards textures, i.e., they tend to learn textures as a shortcut solution, but why is the *dotted texture* here measured to be a complex feature?", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1.\tThe paper is well presented and is quite easy to follow. I appreciate the summary of different experiments into the words “what” “where” and ”when.”\n\n2.\tThe paper provides abundant visualization to help readers grasp the intuitive behind simple features and complex features.", "weaknesses": "My major concerns for this paper are its coherence and significance.\n\n1.\tThe paper lacks coherence because multiple components (e.g., metrics, algorithms) introduced in this paper do not come from the same theoretical framework. For example, the notion of feature complexity in this paper (the $\\nu$-information metric) is drawn from information theory, while the method for extracting features (the Craft method) is based on non-negative matrix factorization and has little connection with information theory. The same issue applies to the feature visualization method for demonstrating features of different complexities, and CKA metric, and the importance metric $\\Gamma(z_i)$. These metrics/algorithms are borrowed from different theoretical frameworks and contexts, so it is in doubt whether they can be used together. I would appreciate it if the authors are able to re-organize all components of the paper under a coherent framework, e.g., the information theoretic perspective (if at all possible).\n\n2.\tMy second concern is about the paper’s significance, since many claims have been discussed in previous works and appears non-novel. The conclusion that simple features are usually color/edge detectors that are located in earlier layers is not surprising, nor do the conjecture that simple features tend to propagate through the residual connection. For the conclusion that simple features are learned earlier than complex features, there are many studies leading to this conclusion, from the perspective of spectral bias [cite1], game-theoretic interactions [cite2], or frequency [cite3]. Overall speaking, I do not learn new insights from the paper. Nevertheless, as mentioned in the 1st point in weaknesses, if the authors are able to re-organize the paper from the purely information-theoretic perspective, it will be a more intriguing work.\n\n3.\tUsing K-means clustering to aggregate features into meta-features may lead to incorrect clustering results. K-means clustering is known for its poor performance for data clusters that are not circular-shaped and are of discrepant sizes. I am not sure how features are distributed in this paper’s experiments, but more advanced clustering strategies such as the spectral clustering or hierarchical clustering can be applied to mitigate this issue.\n\n[cite1] Rahaman et al. On the Spectral Bias of Neural Networks. ICML, 2019.\n\n[cite2] Liu et al. Towards the Difficulty for a Deep Neural Network to Learn Concepts of Different Complexities. NeurIPS, 2023.\n\n[cite3] Xu et al. Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks. ICLR, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> \"The paper lacks coherence because multiple components (e.g., metrics, algorithms) introduced in this paper do not come from the same theoretical framework. [...] These metrics/algorithms are borrowed from different theoretical frameworks and contexts, so it is in doubt whether they can be used together. I would appreciate it if the authors are able to re-organize all components of the paper under a coherent framework, e.g., the information theoretic perspective (if at all possible).\"\n  \nRegarding the different theoretical frameworks for the metrics and algorithms introduced, we believe that (1) these components can indeed be integrated together as is usually done and (2) we could if we wanted to write the entire framework from an information theoretic perspective. For instance, dictionary learning methods have a strong connection with information theory [1,2]. Additionally, CKA is based on HSIC, which is closely related to mutual information ($HSIC = MMD(P, \\prod P_i$) and mutual information $KL(P, \\prod P_i))$.  The framework of $\\nu$-Informations is not tied-up to information theory alone, quite to the contrary. It inherits most of the theoretical properties of Shannon’s entropy (e.g …) but the possibility to specify any function class for the predictive family is what makes it powerful and applicable to a broad range of domains (including deep learning, but not only).\n\nIt would be possible to reorganize the components under a unified framework, even if it is not the traditional approach for each component. Thank you for your suggestion; we are adding these insights in the appendix.\n\n[1]: B. Dumitrescu and P. Irofti, Dictionary Learning Algorithms and Applications  \n  \n[2]: A Personal Introduction to Theoretical Dictionary Learning, Karin Schnass\n  \n> \"My second concern is about the paper’s significance, since many claims have been discussed in previous works and appears non-novel. The conclusion that simple features are usually color/edge detectors that are located in earlier layers is not surprising, nor do the conjecture that simple features tend to propagate through the residual connection [...]Overall speaking, I do not learn new insights from the paper.\"  \n  \nYou are correct that Section 3 qualitatively presents a range of simple, medium, and complex features. While some of these are new, some align with features we have learned about from the prior literature. Still, it is worthwhile to see that our method discovers them. Regarding Section 4, we do not claim novelty for the late appearance phenomenon, although this is the first explanation from a complexity perspective. As another reviewer noted, this provides a fresh viewpoint on the phenomenon. We believe that Section 6 presents two new contributions: the emergence of simplicity bias using $\\nu$-Information during training and the observation that neural networks generalize by compressing important features.  \n  \n> \"Using K-means clustering to aggregate features into meta-features may lead to incorrect clustering results. K-means clustering is known for its poor performance for data clusters that are not circular-shaped and are of discrepant sizes. I am not sure how features are distributed in this paper’s experiments, but more advanced clustering strategies such as the spectral clustering or hierarchical clustering can be applied to mitigate this issue.\"  \n  \nThank you for your suggestion. We chose to use K-means based on the rationale that, in the collapsed space of a neural network, the choice of clustering algorithm has minimal impact on the results: studies have shown that feature distributions in this space are generally well-suited for K-means clustering [1]. \n\n[1] \"Neural Collapse: A Phenomenon in the Terminal Phase of Deep Learning\" by Papyan et al., 2020\n\n> \"The paper claims that simple features tend to propagate through the residual connection, while complex features tend to propagate through the main branch of the network. Then a natural question arises: how are simple and complex features propagate in CNNs without residual connections (e.g., AlexNet, VGG)?\"\n\nThis is indeed a relevant and intriguing question. Investigating whether identity functions or orthogonal transformations occur channel-wise in models like AlexNet and VGG could provide valuable insights. This topic warrants further exploration in future research, and has been noted in future directions.\n\n> \"The feature fences is among the most simple features, while the features whiskers and dotted texture are among the most complex features. However, I do not see an essential difference between the features fences, whiskers, and dotted texture from the sample images in Figure 7 and Figure 8. For example, fences and dotted texture both seem to consist of lines that cross over each other to form holes. Similarly, the whiskers feature is also composed of lines, although the lines do not cross over each other but appear parallel. Why is fences a simple feature, but whiskers and dotted texture complex features?\"\n\nWe argue that fences represent repetitive local patterns, in contrast to whiskers or insect legs, which are unique, finely structured motifs. This distinction ties into the texture versus shape debate: texture pertains to repetitive patterns, while shape involves unique, widely distributed elements across an image, necessitating the extraction of more structured motifs.\n\n> \"Another question is that previous studies show that CNNs are usually biased towards textures, i.e., they tend to learn textures as a shortcut solution, but why is the dotted texture here measured to be a complex feature?\"\n\nFirstly, the dotted pattern falls within the medium complexity category, ranking 14th out of 30 meta-features. Although not highly complex, the appendix reveals that images responding to this pattern exhibit different orientations. This indicates that the pattern detects texture while also capturing it from multiple angles and viewpoints.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CsLEaDphLu", "reviewer": "Reviewer_aiER", "summary": "The paper introduces a novel metric for quantifying feature complexity in deep learning models, specifically focusing on an ImageNet-trained ResNet50 model. This V-information-based metric captures whether a feature requires complex computational transformations for extraction.  The study addresses four key questions:\n(1) The appearance of features as a function of complexity.\n(2) The learning timeline of these features during training.\n(3) The flow of simple and complex features within the network.\n(4) The relationship between feature complexity and their importance in the model's decision-making.\nThe study reveals that simpler features dominate early in training and are transported through the network via residual connections, while more complex features emerge gradually and require more computational effort. Interestingly, the most important features tend to be simpler and become accessible earlier in the network, suggesting a sedimentation process.", "questions": "(1) How would the proposed V-information complexity metric perform on different neural network architectures, such as Transformers or GANs?\n(2) Can the authors provide more details on how the complexity metric correlates with other existing complexity measures in literature? Which one provides better insights?\n(3) How does the model's simplicity bias, as observed in the study, impact its generalization performance on unseen data?\n(4) Are there specific strategies that could be employed to mitigate the simplicity bias and encourage the learning of more complex, yet important features?\n(5) Does the conclusion of section 4 still hold on networks outside ResNet? How to explain the architecture of the transformer regarding the residual connection?\n(6) Why does the horizontal axis in Figure 4 show several blocks instead of all blocks?", "limitations": "While the paper provides significant insights into feature complexity in deep learning models, there are several limitations that need to be addressed:\n\nModel and Dataset Scope: The study focuses solely on a ResNet50 model trained on the ImageNet dataset. This raises concerns about the generalizability of the findings to other neural network architectures (e.g., Transformers, GANs) and different types of datasets. Future work should validate the proposed metric across a variety of models and tasks to ensure broader applicability.\n\nAssumption on Optimal Representation: The complexity metric relies on the assumption that each layer in the network provides an optimal representation for downstream linear probes. However, this assumption may not always hold true, potentially leading to overestimated complexity scores. Further empirical validation and adjustments to the metric may be necessary to account for cases where this assumption is violated.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) Originality: The introduction of the V-information metric for assessing feature complexity is novel and provides a fresh perspective on understanding neural network behavior. The exploration of feature complexity across training epochs and network layers adds significant depth to existing knowledge.\n\n(2) Clarity: The paper is well-organized, clearly presenting its goals, methods, and findings. Visualizations and qualitative analyses effectively illustrate the differences between simple and complex features.\n\n(3) Significance: Understanding feature complexity and its impact on model performance and decision-making is crucial for developing more interpretable and efficient models. This work contributes to explainable AI by providing insights into how features are learned and utilized.", "weaknesses": "(1) Generalizability: While the findings are insightful, they are based on a single architecture (ResNet50) and dataset (ImageNet). The results might not generalize to other models or tasks without further validation.\n\n(2) Complexity Metric: The assumption that each layer optimally represents features for downstream linear probes may not hold universally, potentially leading to overestimated complexity scores in some cases. More empirical validation across different models and tasks could strengthen the proposed metric's robustness.\n\n(3) Temporal Analysis: The focus on specific epochs (e.g., epoch 90) may overlook important dynamics occurring at intermediate stages of training. A more granular analysis could provide a clearer picture of feature evolution, i.e., Can you analyze the changes in importance during the training process？", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> \"While the findings are insightful, they are based on a single architecture (ResNet50) and dataset (ImageNet). The results might not generalize to other models or tasks without further validation.\" \"The assumption that each layer optimally represents features for downstream linear probes may not hold universally, potentially leading to overestimated complexity scores in some cases. More empirical validation across different models and tasks could strengthen the proposed metric's robustness.\"  \n  \nThis is a valid point, which we discuss in lines 325-331 and in Appendix I. We agree and would like to extend our study to include diverse models in the future. However, our perspective with this work is that understanding why ResNet50 generalizes on ImageNet is already a significant challenge.\n  \n> \"The focus on specific epochs (e.g., epoch 90) may overlook important dynamics occurring at intermediate stages of training. A more granular analysis could provide a clearer picture of feature evolution, i.e., Can you analyze the changes in importance during the training process?\"  \n\nThank you for your comment. However, there might be a misunderstanding. Sections 5 and 6 of our paper directly address this issue. We indeed perform a dynamical study of features across epochs, and we show (i) in Section 5 that the most complex features emerge later in terms of epochs, while section 6 demonstrates both (ii) the simplicity bias over epochs and (iii) the dynamic compression of important features throughout the epochs.\n\n> \"How would the proposed V-information complexity metric perform on different neural network architectures, such as Transformers or GANs?\"\n\nWe're not entirely sure if we understand the question. What do you mean by 'perform'? The metric, as introduced in the seminal paper [1], measures mutual information under a complexity constraint. We use it to determine the simplicity or complexity of features. How would you suggest evaluating the performance of our metric? If you're asking about applicability, our metric can be easily applied to transformers or GANs without any issues, as it only requires having access to intermediate activations.\n\n[1] A Theory of Usable Information under Computational Constraints, Xu et al., 2020\n\n> \"Can the authors provide more details on how the complexity metric correlates with other existing complexity measures in the literature? Which one provides better insights?\"  \n  \nYou raise a valid point, as complexity can be defined in various ways. Recent research employs category theory to introduce a redundancy-based metric, which merges neurons until a distance gap surpasses a threshold, using this gap as a hyperparameter [1]. In Appendix E, we demonstrate a correlation between our metric and the general redundancy score presented in [2]. However, this method has two limitations: (i) it requires a hyperparameter, and (ii) features considered redundant by this metric may still be complex according to our measure. Our complexity metric is more focused on the computational operations needed to decode a feature optimally. In this regard, our complexity metric can be viewed as a relaxation of computational complexity metrics, such as Kolmogorov complexity.  \n\n[1] Going beyond neural network feature similarity: The network feature complexity and its interpretation using category theory. Chen et al. (2023). \n  \n[2] Diffused redundancy in pre-trained representations. Nanda et al. (2024)\n   \n> \"How does the model's simplicity bias, as observed in the study, impact its generalization performance on unseen data?\"  \n  \nThis phenomenon has been extensively studied in previous research, which tends to show that a simplicity bias can negatively impact model performance on unseen data. Models that overly rely on simpler features may fail to capture the more complex patterns necessary for robust generalization. We have incorporated references to these relevant studies [1,2] in our Related Work section to provide additional context and support for this observation.  \n\n[1] The Pitfalls of Simplicity Bias in Neural Networks. Shah, H., Tamuly, K., Raghunathan, A., Jain, P., Netrapalli, P.  \n  \n[2] Overcoming Simplicity Bias in Deep Networks using a Feature Sieve. Tiwari R. ,Shenoy, P. (2023). \n  \n> \"Are there specific strategies that could be employed to mitigate the simplicity bias and encourage the learning of more complex, yet important, features?\"   \n  \nOne potential strategy we could imagine to mitigate simplicity bias using our metric is to backpropagate through the score of the $\\nu$-information. This score is differentiable, allowing for backpropagation (backprop through single matrix inversion). While this method can theoretically prioritize the learning of more complex features, it presents significant challenges in terms of memory and scalability. We could imagine addressing this with an online score (or batch estimation) instead of a real Complexity score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JjGLr4ZkR4", "reviewer": "Reviewer_mfVA", "summary": "The paper introduces a metric based on V-information to measure feature complexity in deep learning models. Using ResNet50 trained on ImageNet,  it explores feature spectrum, training dynamics, network flow, and decision impact.\n\nThe study also highlights the role of simplicity bias and the evolution of feature importance in neural networks.", "questions": "1) The paper states that both $z$ and $D^*$ are positive due to the use of Non-Negative Matrix Factorization, which aligns with the nature of ReLUs. Could you elaborate on why negative values for $D^*$ wouldn't work, despite the positive nature of ReLU? It's not clear why allowing $z$ and $D^*$ to be negative would be problematic, especially considering that $(-z)(-D^*) = zD^*$.\n\n2) In line 126, $z$ is defined as $z \\in \\mathbb{R}$. Given that $z$ is described as positive elsewhere, wouldn't $z \\in \\mathbb{R^+}$ be more precise? Or are you intentionally allowing negative values for $z$?\n\n3) The nature of $z_1$, $z_2$, and $z_3$ in Figure 1 is unclear. If these represent the $i$-th element of $z$, they would be scalars, which doesn't seem to align with what's depicted. Could you clarify what these variables represent?\n\n4) Figure 1 shows feature visualization across different layers, and the paper mentions earlier layers elsewhere. However, line 622 states that feature extraction was done only for the penultimate layer. Could you explain this and clarify how earlier layer information was obtained if extraction was limited to the penultimate layer?", "limitations": "The authors have addressed limitations in the appendix.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1) Clear and well defined objective\n2) Informative Figures\n3) Comprehensive supplementary material", "weaknesses": "1) Not Easy To Follow On Math: The paper's mathematical sections are difficult to follow. Clearer and intuitive explanations would improve accessibility.\n2) Limited Model Diversity: The study focuses solely on ResNet50, ignoring modern architectures like depth-wise separable CNNs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> \"Not Easy To Follow On Math: The paper's mathematical sections are difficult to follow. Clearer and intuitive explanations would improve accessibility.\"  \n  \nWe thank the reviewer for this feedback. We strove to keep the math portions of the paper as accessible as possible, but welcome feedback if there were particular points of confusion.\n\n> \"The paper states that both and are positive due to the use of Non-Negative Matrix Factorization, which aligns with the nature of ReLUs. Could you elaborate on why negative values for wouldn't work, despite the positive nature of ReLU? It's not clear why allowing and to be negative would be problematic, especially considering that.\"\n  \nWe are applying a published method widely recognized to generate concepts that are interpretable, sparse, and compositional; see [1, 2] for an overview of the large literature on the subject.  \nTheoretically speaking, the importance of using a consistent semi-ring on both sides (non-negative semi-ring or negative semi-ring, as you pointed out) ensures compositionality and interpretability, preventing the cancellation of concepts (we only add value on top of each other; they don't cancel).  \nPractically speaking, NMF (i) aligns well with the properties of ReLU activations, which are sparse and positive. NMF (ii) allows us to decompose activations without orthogonality constraints, which is crucial since model activations can collapse, making concepts non-orthogonal. Moreover, NMF (iii) supports compositionality, enabling multiple concepts to coexist within an activation. \n\n[1] Lee, D. D., & Seung, H. S. (1999). Learning the parts of objects by non-negative matrix factorization. Nature.  \n[2] Gillis, N. (2014). Nonnegative Matrix Factorization. \n  \n> \"In line 126, is defined as. Given that is described as positive elsewhere, wouldn't be more precise? Or are you intentionally allowing negative values for?\"  \n  \nThis is a good remark. We have corrected this inconsistency to ensure clarity.\n\n> \"The nature of, , and in Figure 1 is unclear. If these represent the -th element of, they would be scalars, which doesn't seem to align with what's depicted. Could you clarify what these variables represent?\"  \n  \nYou are correct, and we apologize for the lack of clarity. $z_1, z_2$, and $z_3$ ​ are indeed scalar values representing directions in the activation space (also called “concepts”). The images shown are feature visualizations, which are optimized to maximize the activation of these concepts (i.e., finding $x$ such that $x = \\arg\\max z_i(x)$. We have clarified this in the manuscript to avoid confusion.\n  \n> \"Figure 1 shows feature visualization across different layers, and the paper mentions earlier layers elsewhere. However, line 622 states that feature extraction was done only for the penultimate layer. Could you explain this and clarify how earlier layer information was obtained if extraction was limited to the penultimate layer?\"  \n  \nThank you for this remark. We have corrected this inconsistency in the manuscript. Feature extraction was indeed performed at the penultimate layer. The visualizations in Figure 1 aim to decode features $z_1, z_2$, and $z_3$ that exist at the penultimate layer at lower layers, such as blocks 1 and 3 of the ResNet50. We use linear probing at these earlier layers to visualize how features develop and are transformed across the network. This method clarification has been added to the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5EcybMkAYC", "reviewer": "Reviewer_ozGz", "summary": "The current work proposes a method to assess the complexity of features in a representation space in terms of usable information.  The measure of complexity is related to how far back in the layers of a trained network one can find information about the feature that is recoverable by a linear decoder.  Equipped with a measure of complexity per feature, the authors grab 10k distributed features in the penultimate layer of a ResNet50 and then examine various relationships such as the growth in complexity over the course of training, the relative importance of simple features in determining the output, and the “flow” of simple features through the residual backbone of the network.", "questions": "- In Appendix A: “The dictionary D was designed to encapsulate 10 concepts per class”.  Does this mean the NMF was run separately for each class, and then the dictionaries were all combined?  If that’s the case, are there multiple copies of some features (e.g. “wheel” being found for every vehicle class)?\n- How were the visualizations of Fig 1 produced, particularly the earlier manifestations z2 and z3?  How should we interpret a feature’s identity in an earlier layer when it is not recognizable in that earlier layer (and therefore a function of many features)?\n- Fig 6: What are we supposed to infer from the epoch 1 results?  If everything is still essentially random at this stage of training, then we're just seeing meaningless values plotted against each other, right?\n- The inhibitory/excitatory results (Fig 12) are striking -- **every single inhibitory feature has complexity less than ~0.5?**  While the other findings are rough trends extracted from point scatter, this finding is relegated to the appendices and given minimal attention?\n\nQuestions out of interest rather than needing to fill a gap in the paper:\n- Regarding the higher importance of simpler features: are they simply present more often, or are they more influential?  I wonder if there's a frequency-complexity relationship that would be insightful.\n- Did you do any of the same analyses with the individual neurons in the penultimate layer (i.e. comparing local vs distributed)?", "limitations": "There is a reasonable discussion of limitations in the Appendices.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The premise -- of a practical way to assess complexity via tiers of usable information, and then its employment as a route to a better understanding of learned features -- is very interesting.  To evaluate the complexity measure, the authors cleverly use the trained layers in the network to approximate a hierarchy of function classes, such that the amount of usable information between the input and the feature must increase as you move deeper into the network, and at each level all that is needed is a correlation measurement.  The complexity measure is thus reasonable and straightforward to measure in practice.  The authors are upfront about a central assumption related to optimal processing by the trained layers.  The writing is clear and the references to related literature are thorough and extensive.", "weaknesses": "Many of the analyses are questionable, in my opinion, making the work seem to have a strong premise with a lackluster follow up.  I will be happy to be rebutted on these points with justification and clarifications during the discussion period.\n\nSpecifically, the “what” and “where” analyses are odd to me.  My main gripes are below, with smaller points in the \"questions\" section.\n- Regarding the qualitative exploration of dictionary vectors, and their grouping into clusters (for “what”): is there a good reason to think relative positions mean anything in feature space, and therefore that clustering is meaningful?  Were the dictionary vectors normalized beforehand, or else did the variable magnitude affect the UMAP and clustering?  Why were 150 clusters used?  Did a sweep over k or any other analysis indicate that the vectors are indeed clustered to some degree, and 150 is an appropriate choice?  Unless I missed it, it’s not explained how the meta-clusters were labeled -- was it just visual inspection, and the remaining 120 clusters simply weren’t interpretable?\n- Regarding the use of CKA for “where”: The proposed complexity measure is already based on how far back in the processing the feature is still recognizable to a linear probe.  Why show CKA instead of the usable information as a function of depth for simple vs complex features?  It’s not intuitive to me to infer anything about the similarity of the “residual” branch aka simply the feature vec one layer deeper, compared to that of the delta (“main branch”), which is not used as a feature by the network.  It seems far more straightforward to look at any changes in the usable information as a function of depth as arising from the delta at that layer.  Why not just show the curves for each of the features?  The curves would also show the way in which usable information is growing with depth -- is it gradual, or stepped?  All said, it’s hard to find much insight from the results of Fig 4, and a straightforward alternative exists.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the time and effort you invested in reviewing our work; thank you. \n\n> \"Regarding the qualitative exploration of dictionary vectors, and their grouping into clusters (for “what”): is there a good reason to think relative positions mean anything in feature space, and therefore that clustering is meaningful? Were the dictionary vectors normalized beforehand, or else did the variable magnitude affect the UMAP and clustering? Why were 150 clusters used? Did a sweep over k or any other analysis indicate that the vectors are indeed clustered to some degree ? \"  \n  \nClustering was used to qualitatively inspect what complex and simple features look like, and it represents a qualitative part of our study. Clustering in embedding space is a common practice, and normalization does not change the cluster membership. We found that meta-clusters were robust to choice of cosine or L2 normalization.  \n\nRegarding the choice of 150 clusters, we aimed to avoid overcrowding figures and selected 30 meta-features that span different degrees of complexity, sampling randomly at each level. Each cluster was qualitatively interpreted, but the labeling of clusters is subjective. We didn't use external models for labeling, and while these labels provide useful insights, they are inherently qualitative and should be interpreted as such. We have clarified this aspect in the manuscript to emphasize the qualitative nature of this part of the analysis and to acknowledge the limitations of the clustering and labeling process.\n\n> \"Regarding the use of CKA for “where”: The proposed complexity measure is already based on how far back in the processing the feature is still recognizable to a linear probe. Why show CKA instead of the usable information as a function of depth for simple vs complex features? It’s not intuitive to me to infer anything about the similarity of the “residual” branch aka simply the feature vec one layer deeper, compared to that of the delta (“main branch”), which is not used as a feature by the network.\"  \n  \nGood question as you are indeed correct. We acknowledge that $\\nu$-information could provide a valuable perspective, and could be used for that. However, to avoid circular reasoning and to employ a well-established metric in the literature, we chose CKA. \nMoreover, our results with $\\nu$-information are consistent with those obtained using CKA, and we have included these additional findings in the appendix as suggested. The choice of CKA was also driven by its established use in analyzing neural network activations, ensuring our findings are interpretable and comparable within the broader research context.\n\n> \"In Appendix A: “The dictionary D was designed to encapsulate 10 concepts per class”. Does this mean the NMF was run separately for each class, and then the dictionaries were all combined? If that’s the case, are there multiple copies of some features (e.g. “wheel” being found for every vehicle class)?\"  \n  \nThe dictionary design process utilized the published Craft method, ensuring 10 concepts per class to achieve balanced reconstruction across classes. While this approach can result in duplicated concepts, it ensures comprehensive coverage of class-specific features. We believe this redundancy does not detract from the overall utility of the dictionary; rather, it enhances the robustness of feature representation by capturing subtle variations within shared concepts across different classes. \n\n> \"Fig 6: What are we supposed to infer from the epoch 1 results? If everything is still essentially random at this stage of training, then we're just seeing meaningless values plotted against each other, right?\"  \n  \nIn this figure, we contrast features at the beginning versus end of training, to get a picture of what has shifted. The end of epoch 1 represents the point where the dataset has been seen once, and several gradient steps have been taken. The features used at the end of epoch 1 have equal chances of being simple or complex (decodable at layer 1 or layer 10, for example). In contrast, by the end of training, we observe a simplicity bias, where important features are more likely to be decodable early. This demonstrates how the network evolves to prioritize simpler features over time, highlighting the dynamic nature of feature learning and the emergence of important features throughout the training process.  \n  \n> \"The inhibitory/excitatory results (Fig 12) are striking -- every single inhibitory feature has complexity less than ~0.5?\"  \n  \nThank you for this remark. We also find this observation intriguing. To be honest, we are not certain whether this is a specific property of ResNet50, ImageNet (because of 1k classes), or a more general phenomenon. This warrants further investigation and could be a fruitful avenue for future research. \n\n> \"Regarding the higher importance of simpler features: are they simply present more often, or are they more influential? I wonder if there's a frequency-complexity relationship that would be insightful.\"\n  \nThis is an excellent question. Previous work has already pointed to these issues [1], and concurrent research has posed similar questions with controlled toy examples [2]. It appears that the relationship between frequency and complexity is indeed complex, involving both factors. Simpler features may be more frequently present and also more influential in driving the network's decisions. We believe this is a fundamental question for deep learning.\n\n[1] Hermann, K., Mobahi, H., Fel, T. Mozer, M. (2023). On the Foundations of Shortcut Learning.  \n[2] Lampinen, A. K., Chan, S. C., and Hermann, K. (2024). Learned feature representations are biased by complexity, learning order, position, and more.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nlfikRmC7W", "reviewer": "Reviewer_d2Wb", "summary": "This paper proposes a method to measure the complexity of features extracted by deep learning models. This method is based on $\\mathcal{V}$-information, an extension of Shannon's mutual information that takes the computational capabilities of a decoder into account. The proposed measure of feature complexity is inversely related to the cumulative $\\mathcal{V}$-information across a model's layers, the intuition being that features that become available at earlier layers will accumulate a larger $\\mathcal{V}$-information, while features that only become available late will have a smaller $\\mathcal{V}$-information.\nFeatures are extracted from a standard, ImageNet-trained ResNet50, from which an overcomplete dictionary is learned. These features are then clustered, and the mean complexity score for each feature is computed. Feature clusters with low, intermediate and high complexity scores are visualized, revealing that low-complexity features tend to be related to uniform colors or low-frequency information, intermediate-complexity features to local shapes such as eyes and noses and high-complexity features to highly structured shapes, such as insect legs.\nUsing the CKA between the features in the dictionary and the activations at different layers within the main branch and residual branch of the network, the authors find that Easy features tend to emerge in early layers and then copied to later layers through the residual branch, while Complex features steadily increase throughout layers in both branches.\nAnalyzing the dynamics of different features' emergence throughout training, complex features are found to emerge later than simple ones. Interestingly, analyzing the interplay between features' complexity and their importance in influencing the network's outputs, the authors find that (a) more important features tend to be less complex, and (b) the complexity of important features is reduced throughout training, suggesting that the network might be \"compressing\" the more important features.", "questions": "No questions, beyond the requests for clarification listed in the weaknesses section.", "limitations": "Limitations are discussed adequately. Please refer to the weaknesses section for comments about limitations which I believe were not sufficiently discussed.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper presents an elegant measure of feature complexity in deep neural networks, which takes into account the networks' computational expressiveness in different layers.\n- The paper is extremely well written, with detailed methods and supplementary materials, exhaustive analyses and clear visualizations.\n- The Related Works section is particularly exhaustive and thorough.\n- It successfully combines several recently proposed interpretability tools.\n- The analysis of the interplay between complexity and importance throughout learning was particularly interesting, suggesting a mechanism for the compression of important features across learning epochs in deep neural networks.", "weaknesses": "- The proposed measure is closely related to the accuracy with which a feature can be linearly decoded from the layers of a network. I believe the paper would benefit from an explicit discussion of what, exactly, is the additional information provided by the proposed method which cannot be gained from directly looking at the \"raw\" decoding accuracy.\n- For simplicity, the authors restrict their analysis to a single ResNet model. While this is an understandable choice, I am not sure about what the implications are for models which do not include residual connections. The residual stream is found to be central in \"teleporting\" simple features to later layers. The authors should explicitly discuss what they predict the pattern of results would be for networks without residual connections. For example, would simple features be overall less relevant to the network's responses? Or would the network implicitly implement a residual-like stream? The limitations of using this architecture exclusively are briefly acknowledged in the Limitations section, but the specific role of residual connections is not discussed there.\n- The proposed measure is based on a loose assumption that the decodability of features tends to increase across layers. However, it is possible that certain features are _only_ available in early layers (for example, certain low-level image features might be discarded). These features would have a low cumulative $\\mathcal{V}$-information, and might thus be erroneously classified as high-complexity. The authors should explicitly discuss whether this is a concern at all, and for what reasons.\n- The plot in Figure 5B shows that more important features tend to become less complex as training progresses. As the authors have access to the features which are subject to this process, and those which are not, it would be extremely interesting to visualize them, showing what exactly is happening - are the same features being extracted at earlier layers? Or are they actually changing, and starting to resemble simpler features such as colors or edges? The answer to this question would clarify more precisely what the proposed complexity measure is capturing. Is it possible that features we visually evaluate as \"complex\" can be extracted at early layers if they are important to the network's task? Or is layer depth always correlated with visual complexity?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> \"The proposed measure is closely related to the accuracy with which a feature can be linearly decoded from the layers of a network. I believe the paper would benefit from an explicit discussion of what, exactly, is the additional information provided by the proposed method which cannot be gained from directly looking at the \"raw\" decoding accuracy.\"  \n  \nWe are not sure which accuracy you refer to. \n  \nThe classification accuracy is not smooth, nor continuous, in network predictions. This typically makes its interpretation more noisy. Indeed, for hard tasks, the accuracy may be near the one of a random predictor, while the loss itself might be significantly lower than the one of a random predictor. Furthermore, accuracy only makes sense for classification tasks over discrete sets.  But predicting a concept coefficient is not as simple as predicting if it is “present” or “absent”. Instead, it is a continuum where a concept can be more or less influential. In this case, we face a regression task, as explained in Appendix C. This smoothness allows better handling of inaccuracies in the concept extraction stage.\n\nIf you refer to MSE, it is indeed closely related to $\\nu$-information in the Gaussian posterior case (appendix C), but thanks to the “optional ignorance” property of $\\nu$-information, we are certain that small MSE can be attributed to knowledge of the input, and not to simplicity of the task.  \n\n> \"I am not sure about what the implications are for models which do not include residual connections. The residual stream is found to be central in \"teleporting\" simple features to later layers. The authors should explicitly discuss what they predict the pattern of results would be for networks without residual connections. For example, would simple features be overall less relevant to the network's responses? Or would the network implicitly implement a residual-like stream? The limitations of using this architecture exclusively are briefly acknowledged in the Limitations section, but the specific role of residual connections is not discussed there.\"  \n  \nThanks for this suggestion. We added a paragraph in the limitations section to discuss the significance of residual connections, and the questions you raise.\n\n> \"The proposed measure is based on a loose assumption that the decodability of features tends to increase across layers. However, it is possible that certain features are only available in early layers (for example, certain low-level image features might be discarded). These features would have a low cumulative -information, and might thus be erroneously classified as high-complexity. The authors should explicitly discuss whether this is a concern at all, and for what reasons.\"  \n  \nThis is a good question. To clarify, we do not have this problem because we only seek to decode features available at the last layer. We create the dictionary at the penultimate layer, ensuring each feature exists at the end. We then calculate their complexity, avoiding issues with features that don't exist at the penultimate layer. However, you highlight an interesting point: what about features present at layer 2 but not at layer 5 for example? This is outside the scope of our current study, but an interesting question for future work.\n\n> \"The plot in Figure 5B shows that more important features tend to become less complex as training progresses. As the authors have access to the features which are subject to this process, and those which are not, it would be extremely interesting to visualize them, showing what exactly is happening - are the same features being extracted at earlier layers? Or are they actually changing, and starting to resemble simpler features such as colors or edges? The answer to this question would clarify more precisely what the proposed complexity measure is capturing. Is it possible that features we visually evaluate as \"complex\" can be extracted at early layers if they are important to the network's task? Or is layer depth always correlated with visual complexity?\"  \n\n Concerning the tracking of type of features, we indeed have some figures available showing this, but the problem is that tracking concepts (i.e., not just averaging complexities but genuinely tracking and linking features across epochs) requires an algorithm with hyperparameters. Some features may disappear at certain points. We chose to show only the average and top and bottom complexity curves to avoid relying too much on the linking method for our analysis. However, this is an excellent remark, and we would like to investigate this further in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### General comments

Thank you to the reviewers for taking the time to read and review our paper. Your critiques are sharp and insightful. You found the paper "extremely well-written" and "well presented and easy to follow." Regarding the results, you noted that it "adds significant depth to existing knowledge," supported by "thorough and extensive references to related literature." Finally, you found the results on the compression of important features "particularly interesting, suggesting a mechanism for the compression of important features across learning epochs in deep neural networks.".

However, we acknowledge that you have raised certain points and identified weaknesses in the paper. During the rebuttal period, we carefully examined the critiques provided by all five reviewers. We have diligently incorporated the necessary citations and clarifications into our Related Work and Discussion sections. We believe we have addressed all the comments in a satisfactory manner

We will now proceed to address each reviewer's comments directly, providing detailed justifications and clarifications for the points raised. We appreciate the opportunity to improve our paper through this constructive feedback.

### About dataset and model

> "Does the conclusion of section 4 still hold on networks outside ResNet? How to explain the architecture of the transformer regarding the residual connection?"  
> "Limited Model Diversity: The study focuses solely on ResNet50, ignoring modern architectures like depth-wise separable CNNs."  
  
We recognize this concern, as outlined in our initial two limitations points. Analyzing ResNet50 during training, as done here, already entails examining over a million features from a standard model used in practice. We believe that a thorough study of this single, large-scale model during its training can already provide valuable insights. Our perspective is that understanding why a ResNet50 generalizes on ImageNet is already a significant challenge. By focusing on a single model, we aim to derive credible hypotheses in the study of simple and complex features, their order of appearance, types of features, and the link between complexity and generalization. However, we agree that this study should be extended to other architectures in future work to generalize these findings.

Meta-review: This paper proposes a new feature complexity metric for deep neural networks, and explores the learning dynamics of features with different complexities, the flow of different features in forward propagation, and the importance of different features to the output. After an active discussion, three reviewers leaned towards acceptance, while two leaned towards borderline rejection. The reviewers praised the good presentation of the paper, the novelty of the proposed feature complexity metric, and the thoroughness of the related literature. The reviewers were concerned about the generalizability of the findings as the paper only conducted experiments on the ResNet-50 model, and the lack of comparison between the proposed metric with other feature complexity metrics. One reviewer was also concerned about significance of the findings, while another reviewer held an opposite view that the paper added significant depth to existing knowledge. During the rebuttal period, the authors discussed about the limitations of only studying the ResNet-50 model, and compared between the proposed metric and other metrics (e.g., the CKA metric). The reviewers feel most of their concerns are addressed after the rebuttal period.
Decision: Accept (poster)