Understanding Visual Feature Reliance through the Lens of Complexity

Authors: thomas  fel
google  deepmind
louis  béthune
andrew  kyle
lampinen  google deepmind
thomas  serre
katherine  hermann
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 20:32:18

Abstract: Recent studies suggest that deep learning models' inductive bias towards favoring simpler features may be an origin of shortcut learning. Yet, there has been limited focus on understanding the complexities of the myriad features that models learn. In this work, we introduce a new metric for quantifying feature complexity, based on V-information and capturing whether a feature requires complex computational transformations to be extracted. Using this V-information metric, we analyze the complexities of 10,000 features—represented as directions in the penultimate layer—that were extracted from a standard ImageNet-trained vision model. Our study addresses four key questions:

First, we ask what features look like as a function of complexity, and find a spectrum of simple-to-complex features present within the model. Second, we ask when features are learned during training. We find that simpler features dominate early in training, and more complex features emerge gradually. Third, we investigate where within the network simple and complex features "flow," and find that simpler features tend to bypass the visual hierarchy via residual connections. Fourth, we explore the connection between features' complexity and their importance for driving the network's decision. We find that complex features tend to be less important. Surprisingly, important features become accessible at earlier layers during training, like a "sedimentation process," allowing the model to build upon these foundational elements.

Main paper: 
 method: 
 Before we measure feature complexity, we define what is meant by features, explain how they are extracted, and then introduce the complexity metric.Model Setup. We study feature complexity within an ImageNet-trained ResNet50 [45] . We train the model for 90 epochs with an initial learning rate of 0.7, adjusted down by a factor of 10 at epochs 30, 60, and 80, achieving a 78.9% accuracy on the ImageNet validation set, which is on par with reported accuracy in similar studies [45, 114] . Focusing on one model reduces architectural variables, creating a controlled environment to analyze feature complexities and provide insights for broader model hypotheses.Feature Extraction. We operate within a classical supervised machine learning setting on (Ω, F, P) -the underlying probability space -where Ω is the sample space, F is a σ-algebra on Ω, and P is a probability measure on F. The input space is denoted X ⊆ R d . Let the input data x : Ω → X be random variables with distributions P x . We will explore how, from x and using a neural network, we extract a series of k features. We will assume a classical vision neural network that admits a series of n intermediate spaces, such that:f ℓ : X → A ℓ with ℓ ∈ {1, . . . , n}.Initially, one might suggest that a feature is a dimension of the model, meaning, for example, that a feature could be a neuron in the last layer of the model z = f n (x) i , i ∈ {1, . . . , |A n |}, thus each of the neurons would be a feature. However, several recent studies [83, 8, 25, 29, 32] have shown that our models actually learn a multitude of features, far more than the number of neurons, which explains, for example, why they are not mono-semantic [77, 19] , which could also hinder our study of features. Therefore, we use a recent explainability method, Craft [30] , to extract more features than neurons and avoid this problem of superposition -or feature collapse. With f n (x) being the penultimate layer, we extract a large number of features, five times more than the number of neurons, using an over-complete dictionary of concepts D ⋆ ∈ R k×|An| , with k ≫ |A n |. This dictionary is obtained by optimization over the entire training set and contains a total of k = 10, 000 features. Thus, for a new point x, we obtain the value of the k features -we recall that the number of features is greater than the number of neurons, k ≫ |A n | -by solving the following optimization problem:z = arg min z≥0 ||f n (x) -zD ⋆ || FWith z ∈ R k being the value for each feature of the image x, in particular, and from now on for ease of notation, we consider z i ∈ R, i ∈ {1, . . . , k} that we will simply denote z for the rest of the paper, as a specific feature for which we want to compute a complexity score. Thus, in our work, a feature refers to a random scalar value extracted by a dictionary learning method on the activations. More details and full derivation regarding the training of D ⋆ are available in the Appendix A.Complexity through the Lens of Computation. To formalize this, still on (Ω, F, P), we denote the output space Z, and z : Ω → Z are random variables of a feature of interest with distributions P z . The joint random vector (x, z) representing an image x and the value of its feature z on (Ω, F) has a joint distribution P defined over the product space X × Z. Furthermore, P(Z) denotes the set of all probability measures on Z. We can now associate, for an x which we recall is a real-valued random variable, a corresponding feature z, another real-valued random variable, and we seek to correctly evaluate the complexity of the mapping from x to z. For this, we turn to the V-Information [115] that generalizes and extends the classical mutual information I(•, •) from Shannon's theory by overcoming its inability to take into account the computational capabilities of the decoder. Indeed, for two (not necessarily independent) random variables x and z, and for any bijective mapping γ : X → X , Shannon's mutual information remains unchanged: I(x, z) = I(γ(x), z).Consider, for instance, γ as a cryptographic function that encrypts an image x using a bijective key-based algorithm (e.g., the AES encryption algorithm). If x represents the original image, and γ(x) represents the cipherimage, the mutual information between x and z remains unchanged. This is because the encryption is a bijective process, and the information content is preserved. However, in practice, the encrypted images would be much harder to decode and use for training a model compared to the original one, without access to the decryption key. Another example we may think of is γ as a pixel shuffling operation. The information carried by x does not disappear after processing by γ. However, it may be harder to extract in practice. This demonstrates the practical importance of V-Information, as it considers the computational effort required to decode the information, highlighting the difference between theoretical and practical accessibility of information. Specifically, the V-information proposes taking into account the computational constraint of the decoder by assuming it can only extract information using a predictive family V ⊆ F = {η : X ∪ { } → P(Z)}. The authors [115] then define the V-entropy and the V-conditional entropy as follows:EQUATIONWhere η(•; •) is a function from X ∪ { } → P(Z) that returns a probability density η(x; •) on Z using side information x, or without side information . The predictive family V summarizes the computational capabilities of the decoder. When V contains all possible functions, V = F, it recovers Shannon's entropy as a special case. Intuitively, we seek the best possible prediction for z knowing x by maximizing the log-likelihood. Continuing, we naturally introduce the V-information:I V (x → z) = H V (z) -H V (z|x).The complexity of the mapping from x to z can now be assessed by examining a hierarchy of predictive families V 1 ⊂ . . . ⊂ V n of increasing expressiveness, like explored in [61] . Each predictive family V ℓ corresponds to a partial forward up to depth ℓ, followed by a decoding step. This involves determining at which point we can decode or make the information from x to z available. Formally, we define the complexity of the feature as dependent of the cumulative V-information across layers: We use Feature visualization [84, 31] to visualize the Meta-features found after concept extraction. The entire visualization for each Meta-feature can be found in Appendix B.EQUATIONHere, we define the predictive family V as a class of linear probes with Gaussian prior. Under this hypothesis, the associated V-information of this class possesses a closed-form solution (see Appendix C), which serves as the basis for our evaluation. A higher score implies that the feature z is readily accessible and persists throughout the model's layers. Conversely, a lower score suggests that the feature z is unveiled only at the very end of the model, if at all.Assumption. Crucially, the correctness of the computation of I V (f ℓ (x) → z) relies on the hypothesis that each layer f ℓ provides the optimal representation for the downstream linear probe η. In other words, we assume that I V (f ℓ (x) → z) = I V ℓ (x → z), or again that η * ℓ = η * • f ℓ . This hypothesis is reasonable, since a neural network is essentially "linearizing" the training set-projecting the training set into a space in which it is linearly separable. Thus, it makes sense to assume that each layer attempts to make the feature linearly decodable as efficiently as possible. If this condition is violated, the complexity measure may overestimate the true complexity of a feature (since we can only underestimate the V-information). For example, this may happen if the optimal path to calculate a feature requires deviating from the linear decoding to make it easier to decode later. While some recent works have motivated a slightly different complexity metric based on redundancy [24] , we show in Appendix E that our complexity measure is inherently linked to redundancy. 
what do complex features look like? a qualitative analysis: 
 This section presents a qualitative investigation of relatively simple versus more complex features. Drawing from critical insights of recent studies, which indicate a tendency of neural networks to prefer input features that are both predictive and not overly complex [48] , this analysis aims to better understand the nature of features that are easily processed by models versus those that pose more significant challenges. Indeed, understanding the types of features that are too complex for our model can help us anticipate the types of shortcuts the model might rely on and, on the other hand, design methods to simplify the learning of complex features. This section of the manuscript is intentionally qualitative and aims to be exploratory. We applied our complexity metric to 10,000 features extracted from a fully trained ResNet50. For each feature, we computed the complexity score K(z, x) using a subset of 20,000 images from the validation set. Recognizing the impracticality of manually examining each of the 10,000 features, we employed a strategy to aggregate these features into a more manageable number of groups that we called Meta-features.To condense the vast array of features into a reduced number of similar features, we applied K-means clustering to the feature dictionary D ⋆ , resulting in 150 distinct clusters. These clusters represent collections of features, referred to as Meta-features C = {v 1 , . . . , v |C| }; we then computed an average complexity score for each group. By selecting a diverse range of 30 clusters, chosen to cover a spectrum of complexity levels from the simplest to the most complex features, we aimed to provide a comprehensive overview of the diversity of feature complexity within the model. We propose to visualize the distance matrix in D ⋆ , showing feature complexity in Figure 2 . This approach offers preliminary insights into features seen as simple or complex by the model. Simple Features. Among the simpler features, we find elements primarily based on color, such as sky and sea, as well as simple pattern detectors like line detectors and low-frequency detectors exemplified by bokeh. Interestingly, features geared towards text detection, such as watermark, are also included in this group. These findings align with previous studies [117, 96, 12, 82] , which have shown that neural networks tend to identify color and simple geometric patterns in the early layers as well as low-frequency detectors. This suggests that these features are relatively easy for neural networks to process and recognize. Furthermore, our findings detailed in Appendix 11 corroborate the theoretical work posited in [11, 72] : robust learning possibly induces the learning of shortcuts or reliance on "easy" features within the model.Features with medium complexity reveal more nuanced and sometimes unexpected characteristics. We find, for example, low-quality detectors sensitive to lowresolution images. Additionally, a significant number of concepts related to human elements were observed despite the absence of a dedicated human class in ImageNet. Trademark-related features, distinct from simpler watermark detectors, also reside within this intermediate complexity bracket.Complex Features. Among the most complex features, we find several Meta-features that exhibit a notable degree of structural coherence, including categories such as insect legs, curves, and ears. These patterns represent structured configurations that are ostensibly more challenging for models to process than more localized features, echoing the ongoing discussion about texture bias in current models [10, 37, 46] . Intriguingly, the most complex Meta-features identified, namely whiskers and insect legs, embody types of filament-like structures. Interestingly, we note that those types of features are known to be challenging for current models to identify accurately [60] , aligning with documented difficulties in path-tracking tasks [66] . Such tasks have revealed current models' limitations in tracing paths, which parallels challenges in connectomics [89] , particularly in filament segmentation-a domain recognized for its complexity within deep learning research. Now that we've browsed simple and complex features, another question arises: how does the model build these features during the forward pass? For instance, where within the model does the formation of a watermark detector feature occur? And for more complex features that require greater structure, in which block of computation are these features formed within the model?4 Where do Complex Features Emerge f ℓ+1 (x) = f ℓ (x) "Residual" branch + (g ℓ • f ℓ )(x) "Main" branchThis equation highlights two distinct paths: the "Residual" branch, which facilitates the direct transfer of features from f ℓ to the subsequent layer ℓ + 1, and the "Main" branch, which introduces additional transformations to f ℓ through additional computation g ℓ to enhance its representational capacity. We aim to investigate the flow of simple and complex features through these branches. In our analysis, we examine two subsets of features: 100 features of the highest complexity (top-1 percentile) and 100 features of the lowest complexity (bottom-1 percentile). We measure the Centered Kernel Alignment (CKA) [57] between the final concept values z and the activations from (A) the "Residual" branch f ℓ , and (B) the "Main" branch (g ℓ • f ℓ ), at each residual block, as a proxy for concept information contained in each branch. The findings, illustrated in Figure 4 , reveal that simple features are efficiently "teleported" to later layers through the residual branches -in other words, once computed, they are passed forward with little subsequent modification. In contrast, complex concepts are incrementally built up through an interactive process involving the "main" and "residual" branches. This understanding of feature evolution within network architectures emphasizes the importance of residual connections. This insight, though expected, clarifies a common conception by showing that simple features utilize the residual branch. The next step is to examine the temporal dynamics of feature development, specifically investigating when complex and simple concepts emerge during model training. 5 When do Complex Features Arise n to represent the penultimate layer of the model at any given epoch i, where i ∈ {1, . . . , e} and e represents the total number of epochs. We aim to determine how early each feature can be detected in previous epochs f (i) n for i < e. This involves calculating a specific decoding score; in our scenario, we define this score as I V -the measure of V-information between the model's penultimate activations across epochs and an ultimate feature values, where V is the set of linear models. This metric helps us assess whether a feature was "readily available" at a certain epoch i. The cumulative score Λ is calculated by averaging this measure across all epochs, leading to our score:Λ(x, z) = 1 - 1 e e i I V (f (i) n (x) → z).The results, as illustrated in Figure 5A , showcase the complexity of a feature (K) with it's Time to Decode (Λ) score. An observed correlation coefficient nearing 0.5 intimates that features of heightened complexity are generally decoded later during the training epoch. This finding suggests a nuanced interrelation between the layer for which a feature is available and the epoch of discovery: a feature decoded later in the forward pass trajectory also came online later in training. This naturally leads us to the question of the dynamics of model training. Can we get a deeper understanding of how precisely complex concepts are formed within the model? Does the model develop complex features solely upon necessity, thereby suggesting a correlation between the complexity of a feature and its importance?6 Complexity and Importance: A Subtle TangoNumerous studies have proposed hypotheses regarding the relationship between the importance and complexity of features within neural networks. A particularly notable hypothesis is the simplicity bias [3, 51, 110] , which suggests that models leverage simpler features more frequently. This section aims to quantitatively validate these claims using our complexity metric paired with the importance of each feature. Because features are extracted from the penultimate layer, a closed-form relationship between features and logits can be derived due to the linear nature of this relationship. By analyzing this relationship over training for features of different complexity, we identify a surprising novel perspective: models appear to reduce the complexity of their important features. This process is analogous to sedimentation and mirrors the operation of a Levin Universal Search [63] . The model incrementally shifts significant features to earlier layers, taking time to identify simpler algorithms in the process. Importance Measure. The feature extraction framework outlined in Section 2 offers a structured approach to estimating the importance of a feature within the network. Specifically, the feature vector z ∈ R k is linearly related to the model's decision-making process, exemplified by a logit calculation y = zD ⋆ W ∈ R, where W ∈ R |An| represents the weights of the penultimate layer for the class-specific logit. The contribution of the i-th feature, z i , to the logit y can be precisely measured by leveraging the gradient-input formulation, which is optimal for fidelity metrics within a linear context [5, 32] . This optimality and the closed-form expression are feasible primarily because the analysis is confined to the penultimate layer of the network. Formally, the importance of a feature z i is defined as:Γ(z i ) = E Pz || ∂y ∂zi • z i || .In essence, the importance measure Γ(z i ) quantifies the average contribution of the i-th feature to the class-specific logit -essentially, the average score that each feature brings to the decision logit. More details on importance measures and the effect of inhibition features are available in Appendix G.Models Prefer Simple Features. The analysis, supported by Figure 6 (right), demonstrates a clear trend indicating the model's simplicity bias. Among the 10,000 features extracted in the final epoch, more complex features-characterized by higher K(x, z) values-are generally assigned lower importance (Γ(z)). In contrast, simpler features predominantly influence the model's decisions. The plot on the left showcases the complexity and importance of 10,000 concepts extracted at the end of the first epoch; we observe that the model does not exhibit this simplicity bias at the end of the first epoch. More detail and study on the role of complex concept is proposed in Appendix D. This observation raises the question of the dynamic interplay between feature complexity and importance. To further investigate, we did a detailed analysis of the evolution of feature complexity and importance throughout the training process.Model as Levin's Machine: Simplifying the Complexity of Important Features. A closer examination of the evolution of feature importance over time reveals an interesting phenomenon in Figure 5B : the emergence of two distinct phases during training. Initially, there is a global increase in feature complexity, with the model beginning its training with relatively simple features. Surprisingly, this is followed by a phase where the model actively reduces its overall complexity, specifically targeting and simplifying its most important features. The model appears to be "shortening" the computational "programs" responsible for generating these significant features. This observation suggests that the ResNet50 under study, like a Levin Machine, develops simpler computational paths for crucial features. Put simply, our complexity metric shows that important features are extracted at earlier layers, resembling sedimentation with foundational elements near the network's input. This behavior presents a novel perspective on how neural networks might be intrinsically driven to generalize by simplifying the computation graph of their important features. However, at least at the early stages of learning, it also challenges our assumption that each layer is optimized to provide a linearly-separable representation for the downstream linear probe -early in learning, this assumption is clearly violated since some complex features could be represented more simply than they are initially. Thus, future work will be needed to fully disentangle the interaction of complexity and importance over training. 
conclusion: 
 We introduced a complexity metric for neural network features, identifying both simple and complex types. We have shown where simple features flow -through residual connections -as opposed to complex ones that develop via collaboration with main branches. Our study further revealed that complex features are learned later in training than simple ones. We have concluded by exploring the relationship between feature complexity and importance, and discovered that the simplicity bias found in neural networks becomes more pronounced as training progresses. Surprisingly, we found that important features simplify over time, suggesting a sedimentation process within neural networks that compresses important features to be accessible earlier in the network. 
: 
 Figure 1 : A) Simple vs. Complex Features. Shown is an example of three features extracted using an overcomplete dictionary on the penultimate layer of a ResNet50 trained on ImageNet. Although all three features can be extracted from the final layer of a ResNet50, some features, such as z 1 , seem to respond to color, which can be linearly extractable directly from the input. In contrast, z 2 , z 3 visualization appear more "Complex", responding to more diverse stimuli. In this work, we seek to study the complexity of features. We start by introducing a computationally inspired complexity metric. Using this metric, we inspect both simple and complex features of a ResNet50. B) Feature Evolution Across Layers. Each row illustrates how a feature from the penultimate layer (z 1 , z 2 , z 3 ) evolves as we decode it using linear probing at the outputs of blocks 1, 5, and 10 of the ResNet50. Simpler features, like color, are decodable throughout the network. The feature in the middle shows similar visualization at block 10 and the penultimate layer, whereas the most complex feature is only decodable at the end. Our complexity metric, based on V-information [115] , measures how easily a model extracts a feature across its layers.has not been a comprehensive quantitative framework for assessing the complexities of features learned by large-scale, natural image-trained vision models used in practice. Leveraging recent observations and advances in feature (also called "concept") extraction from the area of Explainable AI [83, 8, 25, 29, 32, 77, 19, 32, 42] , we extract a large set of features from an ImageNet-trained model [45] , and analyze their complexity.Our contributions are as follows:• We build upon V-information [115] -which measures the mutual information between two variables, considering computational constraints-to introduce a measure of feature complexity. We use this measure to quantify the complexity of over 10,000 features in an ImageNet model [45] at each epoch of training. • We visualize the differences between simple and complex features on a spectrum to understand which features are readily available to our model and which ones require more computation and transformation to retrieve. • We investigate where sensitivity to simple versus complex features emerges during a forward pass through the model. Our findings suggest that residual connections "teleport" simple features, computed in early layers, to the final layer. The main branch naturally facilitates the layer-wise construction of more complex features. • We examine feature learning dynamics, revealing when different concepts emerge over the course of training, and find that complex concepts tend to appear later than simpler ones. • We explore the link between complexity and importance in driving the model's decisions. We find a preference for simpler features over more complex ones. This simplicity bias emerges during training, and, surprisingly, the model simplifies its most important features over time.That is, during training, important features become accessible at an earlier layer (via a shorter computational graph).1 Related Work Feature analysis. Large vision models learn a diversity of features [77, 84] to support performance on the training task and can exhibit preferences for certain features over others, for example textures over shapes [10, 37, 46] . These preferences can be related to their use of shortcuts [36, 100] which compromise generalization capabilities [74, 73] . Hermann et al. [48] suggest that a full account of a model's feature preferences should consider both the predictivity and availability of features, and identify image properties that induce a shortcut bias. Relatedly, work shows that models often prefer features that are computationally simpler to extract-a "simplicity bias" [107, 87, 91, 7, 97, 47] .Explainability. Attribution methods [99, 117, 9, 33, 88, 80, 41, 102, 106] seek to attribute model predictions to specific input parts and to visualize the most important area of an image for a given prediction. In response to the many limitations of these methods [2, 38, 101, 27, 54, 79] , Feature Visualization [78, 84, 44] methods have sought to allow for the generation of images that maximize certain structures in the model -e.g., a single neuron, entire channel, or direction, providing a clearer view features learned early [82, 96] , as well as circuits present in the models [83, 62] . Recently, work has scaled these methods to deeper models [31] . Another approach, complementary to feature visualization, is automated concept extraction [39, 119, 30, 32, 1, 112] , which identifies a wide range of concepts -directions in activations space -learned by models, inspired by recent works that suggest that the number of learned features often exceeds the neuron count [29] . This move towards over-complete dictionary learning for more comprehensive feature analysis represents a critical advancement.Complexity. On a theoretical level, the complexity of functions in deep learning has long been a subject of interest, with traditional frameworks like VC-dimension falling short of adequacy with current results. In particular, deep learning models often have the capacity to memorize the entire dataset, yet still generalize [118] ; the reason is often suggested to be a positive benefit of simplicity bias [3, 51, 110] . Measures of the complexity of neural network functions are hard to make tractable [93] . Recent work has proposed various methods to evaluate this complexity. For instance, [17] proposed a score of non-linearity propagation, while [52] introduced a measure of local complexity based on spline partitioning. Additionally, [111] demonstrated that models tend to learn functions with low sensitivity to random changes in the input. The role of optimizers in complexity has also been explored. It has been shown that different optimizers impact the features learned by models; for example, [105] found that sharpness-aware minimization (SAM) [34] learns more diverse features, both simple and hard, whereas stochastic gradient descent (SGD) models tend to rely on simpler features. Furthermore, [24] utilized category theory to propose a metric based on redundancy, which consist in merging neurons until a distance gap is too large, with this distance gap acting as a hyperparameter. Concurrent work by Lampinen et al. [58] studies representations induced by input features of different complexities when datasets are carefully controlled and manipulated. Finally, Okawa et al. [81] , Park et al. [85] investigated the development of concepts during the training process on toy datasets and revealed that the sequence in which they appear, related to their complexity, can be attributed to the multiplicative emergence of compositional skills.Concerning algorithmic complexity, Kolmogorov complexity [103, 56, 21] , later expanded by Levin [63] to include a computational time component, offers a measure for evaluating the shortest programs capable of generating specific outputs on a Turing machine [22, 43] . This notion of complexity is at the roots of Solomonoff induction [104] , which is often understood as the formal expression of Occam's razor and has received some attention in deep learning community [94, 95, 16] . Further developing these concepts, V-information [115] introduces computational constraints on mutual information measures, extending Shannon's legacy. This methodology enables the assessment of a feature's availability or the simplicity with which it can be decoded from a data source. We will formally introduce this concept in Section 2.We observe in Appendix E that our complexity measure is also correlated with this category theory based complexity metric.  
 
 Appendix: Dictionary Learning. To comprehensively analyze the complexity of features extracted from a deep learning model, we employed a detailed feature extraction process using dictionary learning, specifically utilizing an over-complete dictionary. This approach allows each activation f n (x) ∈ A ℓ to be expressed as a linear combination of multiple basis elements (direction, also called atoms) d ∈ A ℓ from the dictionary D ⋆ = {d 1 , . . . , d k } coupled with some sparse coefficient z ∈ R k associated to each atoms.The over-completness of D ⋆ means that the dimension of the dictionnary (k) is larger than the dimension of the activations space k >> |A ℓ |. This property allow us to overcome the superposition problem [29] essentially stating that there be more feature than neurons.Mathematically, given an activation function f n (x), it can be represented as a linear combination of atoms from the dictionary D, expressed as:f n (x) ≈ zD ⋆ = k i z i d iwhere z i are the coefficients indicating the contribution of each atom d i from the dictionary. Implementation. Our implementation was inspired by Craft [30] , leveraging the properties of ReLU activations in ResNet50. Given that ReLUs induce non-negativity of the activation, we employed Non-Negative Matrix Factorization (NMF) [59, 113] for the reconstruction, as it naturally aligns with the sparsity and non-negativity constraints of ReLU activations. Unlike PCA, which cannot produce overcomplete dictionaries and may result in non-positive activations, NMF can create overcomplete dictionaries in this context.The dictionary D ⋆ was trained to reconstruct the activations f n (x) using the entire ImageNet training dataset, comprising 1.2 million images. Formally, for the set of images X and their corresponding activations f n (X), the objective was to minimize the reconstruction error:||f n (X) -ZD ⋆ || F ,ensuring that f n (X) can be closely approximated by ZD ⋆ . Additionally, the NMF framework enforces non-negativity constraints on the dictionary matrix D ⋆ ≥ 0 and the coefficients Z ≥ 0:(Z, D ⋆ ) = arg min Z≥0,D ⋆ ≥0 ||f n (X) -ZD ⋆ || F .The dictionary D ⋆ was designed to encapsulate 10 concepts per class, resulting in a total of 10,000 concepts. To augment the training samples for NMF, we exploited the spatial dimensions of the last layer of ResNet50, which has 2048 channels with a spatial resolution of 7x7. By training the NMF independently on each of the 49 spatial dimensions, we effectively increased the number of training samples to approximately 58 million artificial samples (channel activations).We utilized the block coordinate descent solver from Scikit-learn [86] to solve the NMF problem. This algorithm decomposes the problem into smaller subproblems, making it more tractable. The optimization process continued until convergence was achieved with a tolerance of ε = 10 -4 , ensuring the dictionary was sufficiently optimized for accurate feature extraction. Post-training, the reconstructed activations ZD ⋆ retained over 99% accuracy in common predictions compared to the original activations f n (X).Extracting Features for New Data Points. Once the dictionary D ⋆ was trained, it was fixed. For any new input x, the corresponding feature z was extracted by solving a Non-Negative Least Squares (NNLS) problem. This mapping of new input activations f n (x) to the learned feature space was performed by minimizing the following objective:z = arg min z≥0 ||f n (x) -zD ⋆ || F .This optimization problem is convex, ensuring computational feasibility and robust feature extraction for new data points.To visualize each feature, we used feature visualization methods [84] . Specifically, a recent improvement [31] re-parameterizes the original feature visualization optimization problem-i.e., finding an image x that maximizes a direction, channel, or neuron-by optimizing only the phase of a Fourier buffer while fixing the image magnitude. This approach produces more realistic images and prevents high-frequency leakage that results in adversarial positives.Figure 7 : Visualization of Meta-Features, sorted by Complexity. Feature visualization using MACO [31] for the most simple (1-15) of the 30 Meta-features found on the 10,000 features extracted.Figure 8 : Visualization of Meta-Features, sorted by Complexity. Feature visualization using MACO [31] for for the most complex (15-30) of the 30 Meta-features found on the 10,000 features extracted.In this section, we detail the closed-form expression of the V-information when the predictive family V consists of linear classifiers with Gaussian posteriors. Specifically, V is defined as follows:V = η : x → N (ψ(x), σ 2 ), with x ∈ X and ψ ∈ Ψ; → N (µ, σ 2 ), with µ ∈ R, σ 2 = 1 2 ;whereΨ = {x → M x | M ∈ R d} is a set of linear predictors. This setting corresponds to the linear decoding we apply during the computation of V-information. In this context, a closed-form solution is available (see [115] ):I V (x → z) = H V (z) -H V (z | x) = inf µ∈R E z∼Pz -log 1 √ 2πσ 2 e -(z-µ) 2 2σ 2 -inf ψ∈Ψ E x,z∼P -log 1 √ 2πσ 2 e -(z-ψ(x)) 2 2σ 2 = inf µ∈R E z∼Pz (z -µ) 2 2σ 2 -inf ψ∈Ψ E x,z∼P (z -ψ(x)) 2 2σ 2 = 1 2σ 2 inf µ∈R E z∼Pz (z -µ) 2 -inf ψ∈Ψ E x,z∼P (z -ψ(x)) 2 = Var(z) 2σ 2 1 - inf ψ∈Ψ E x,z∼P (z -ψ(x)) 2 Var(z) = Var(z) 2σ 2 R 2 = Var(z)R 2 .Here, R 2 is the coefficient of determination. Therefore, the following inequalities hold:0 ≤ I V (x → z) ≤ Var(z).Given that the input data are centered and scaled, we typically have Var(z) around 1 (or less in case of Layer normalization). Furthermore, residual connections and batch normalization tend to preserve this scaling in deeper layers. We note, however, that our score K(z, x) is not strictly bounded. Indeed, we define complexity as the opposite of the average V-information across layers: complex features are those that are harder to decode. We add a shift of 1 for the ease of plotting. Empirically, we observed that this adjustment yields K(z, x) in the range [0, 1], with 1 indicating a complex feature that is not available and 0 indicating a simple feature that is fully available.While simplifying important features underscores a trend toward computational efficiency, the role of complex features within the model deserves a closer examination. Despite these features often being deemed less important directly, they contribute significantly to the model's overall performance, a paradox that can lead us to introduce the concept of "support features." These are a set of features that may not carry substantial importance individually, but that collectively play a crucial role in the model's decision-making process.Figure 9 : "Support Features" hypothesis. The majority of complex features are not very important, but play a non-negligible role and contribute to significant performance gains. This paradox is referred to as the "support features," a large ensemble of features individually of little to very little importance to the model but collectively holding a significant role.The presence of numerous complex features, whose importance on average is less pronounced, poses a conundrum. However, these features are far from redundant. Experiments conducted by progressively removing the most complex concepts from the model demonstrate a noticeable impact on performance, as illustrated in Figure 9 . This empirical evidence supports the theory that, while individually, these complex features may not be pivotal, their collective presence contributes indispensably to the robustness and adaptability of the model. These results are reminiscent of prior findings that low-importance model components that are removed in pruning may nevertheless contribute to model accuracy on rare items [50] .This observation aligns with the broader understanding of neural network functionality, where diversity in feature representation-spanning from simple to complex-enhances the model's ability to generalize and perform across varied datasets and tasks. Therefore, the "Feature Support Theory" underscores an essential aspect of neural network design and training: integrating and preserving a wide spectrum of features, regardless of their individual perceived importance, are vital for achieving high levels of performance and robustness.To further understand the link between feature complexity and redundancy, we utilized the redundancy measure from [76] . Our findings indicate that complex features tend to be less redundant, as depicted in Figure 10 . This observation aligns with the strong correlation between our complexity measure and the redundancy-based complexity measure proposed by [24] .Figure 10 : Complex features are less redundant. Using the redundancy measure from [76] , we show that our complex features tend to be less redundant. This result also confirms a link between our complexity measure and the one recently proposed by [24] , which is also based on redundancy.To quantify redundancy, [76] employed a modified version of Centered Kernel Alignment (CKA) [57] , a measure of similarity between two sets of activation features. We briefly recall that CKA between two set of activations A, B in R d is defined as follows:CKA(A, B) = ∥K A K B ∥ 2 F ∥K A K A ∥ F ∥K B K B ∥ Fwhere K A and K B are the Gram matrices of the feature activations A and B, respectively, and ∥ • ∥ F denotes the Frobenius norm.In our analysis, we calculated the CKA measure between a feature z and the activations for a set of 2,000 images from the validation set f n (X). Subsequently, we compared this with the CKA measure when a portion of the activation is masked using a binary mask m ∈ {0, 1} |A ℓ | , denoted as CKA(z, f n (X) ⊙ m), where ⊙ represents element-wise multiplication (Hadamard product). This comparison enabled us to assess whether masking a subset of neurons impacts the decoding of the features. Specifically, to evaluate redundancy, we employed a progressive masking strategy, successively masking 10%, 50%, and 90% of the activation. If the masked activations retain a high CKA with z, it indicates that the information remains largely intact, suggesting that the feature is redundantly stored across multiple neurons, sign of a redundant encoding mechanism within the network. Conversely, if masking results in a substantial decrease in CKA, it implies that the information was predominantly localized on a specific neuron. In this scenario, the feature is not redundantly encoded but rather concentrated in specific neurons. This concentration indicates a lower degree of redundancy, as the loss of these specific neurons (throught the masking) leads to a significant reduction in the CKA score.The final score of redundancy is then the average CKA difference between the original activation and the masked activations:Redundancy = E m CKA(f n (X) ⊙ m, z) CKA(f n (X), z)And averaged across the different level of masking. A high score (1) indicating a high redundancyi.e. the CKA between the masked activation and with the original activation is similar -while a low score indicate a more localized and thus a lower degree of redundancy.In summary, our results, as depicted in Figure 10 support the idea that complex features exhibit lower redundancy. To measure robustness, we evaluate the stability of feature responses under perturbations. For each input point x, we add isotropic Gaussian noise with varying levels of standard deviation σ. The robustness score is determined by measuring the variance in the feature response due to the noise. Formally, let z(x) represent the feature response for input x. We define the perturbed input as:x = x + N (0, σ 2 I) where N (0, σ 2 I) represents Gaussian noise with mean 0 and variance σ 2 . The sensitivity score Sensitivity(z) for a feature z is given by:Sensitivity(z) = Var(z(x))Specifically, we sample 100 random noise and repeat this for 3 levels of noise σ ∈ {0.01, 0.1, 0.5} to compute the variance in feature response for each input from 2,000 samples from the Validation set of ImageNet to get a distribution of feature value. We also consider other metrics such as the range (min-max) of the feature response, but all methods consistently indicate that more complex features are less robust.In summary, our results, as shown in Figure 11 , demonstrate that complex features exhibit lower robustness. This indicates that features with higher complexity are more sensitive to perturbations and noise, resulting in greater variability in their responses. The problem of estimating feature importance is closely related to attribution methods [117, 33, 88, 80, 15, 102, 106, 23] , which aim to identify the important pixels for a decision. A recent study has shown that all attribution methods can be extended in the space of concepts [32] . In our case, the features are extracted from the penultimate layer, where the relationship between feature values and logits is linear. We will elaborate on this and demonstrate that the notion of importance in the linear case is easier and optimal methods to estimate importance exist.Setup. Recall that for a point x, we can obtain its k feature values by solving an NNLS problem z = arg min ||f n (x) -zD ⋆ || F . The vector z contains the k features in R k . We can replace the activation of the penultimate layer f n (x) with its feature representation in the over-complete basis zD ⋆ ≈ f n (x). Since we are in the penultimate layer, the model's decision, i.e., the logit y ∈ R for the predicted class, is linearly related to each feature z by the last weight matrix, denoted as W , as follows:EQUATIONThus, the energy contributed to the logit by feature i can be directly measured byW ′ i z i and y = k i W ′ i z i .Consequently, the contribution of a feature z i can be measured using gradient-input, (∇ z y) ⊙ z. Several studies [5, 32] have detailed the linear case and shown the optimality of gradientinput with respect to fidelity metrics. They also demonstrated that many methods in the linear case boil down to Gradient-Input, including Occlusion [117] , Rise [88] , and Integrated Gradient [106] .In our case, we measured the importance by taking the absolute value of the importance vector, i.e.,Γ(z i ) = E Pz || ∂y ∂zi • z i || .It is natural to question whether this approach might overlook important features due to their inhibitory effects. Indeed, as depicted in Figure 12 , numerous features may be important not because they add positive energy to the logits, but by inhibition, i.e., by suppressing class information. Although this does not alter the implications of our previous observations, it is noteworthy that the majority of inhibitory features are also simple features.Prevalence and Importance. Another property of importance is its close relationship with prevalence [32] , which indicates that a frequently occurring feature will, on average, be more important given the same importance coefficient (∇ z y). In our study, this implies that if the most important features are reduced, these important features are also potentially more frequently present. Consequently, the prevalence of a feature can be a factor explaining this sedimentation process. We refer the reader to a concurrent study that proposed to investigate more deeply this phenomena using a controlled dataset [58] .The visualization in Figure 2 prompts an important question: are features in our model clustered based on their complexity? Specifically, are there regions in the feature space that are generally more "complex" than others, or that respond primarily to more complex stimuli? Our access to 10,000 features via overcomplete decomposition enables a more detailed analysis compared to traditional neuron-wise studies. We aim to explore three main hypotheses:• Hypothesis 1: Features cluster by super-class. This hypothesis posits that features corresponding to semantically related categories are spatially grouped within the feature space-e.g., concepts related to the dog class are closer to those of the cat class than to unrelated classes like furniture. • Hypothesis 2: Features cluster by complexity. We suggest that features may organize themselves based on their complexity, with simpler features forming distinct clusters separate from more complex ones. • Hypothesis 3: Features cluster by importance. This hypothesis explores whether features with similar predictive importance tend to group together within the feature space.It is important to emphasize that these hypotheses are mutually independent. To test them, we propose two methodologies: (1) visualizing feature embeddings using UMAP and (2) clustering the features followed by dendrogram analysis to examine whether the resulting clusters are homogeneous (also called "pure") in terms of super-class, complexity, or importance. Feature Similarity vs Super-Class. Figure 13 illustrates the organization of features by their super-class. Each point is a feature, colored according to its super-class label. We observe that certain super-classes, like those associated with birds, dogs, reptiles form distinct and cohesive clusters, indicating a strong grouping within the feature space. Other region have features that encompassing a wider range of super-class, such that grass, waves, low-pixel quality detector. This reveals that some meta-features are predominantly associated with a single super-class ("pure"), while others span multiple super-classes ("impure"), reflecting the shared visual characteristics or multi-functional nature of those features.Figure 14 : Feature Similarity by Complexity. A) Each point is a feature, colored by its complexity score. Distinct areas of the graph correspond to varying levels of complexity, suggesting a nonrandom distribution of feature complexity. For instance, animal-related features tend to have higher complexity, one could hypothesize that the fine-grained classification required for these categories are responsible for this complexity. B) A four-level dendrogram where each level further segments clusters and calculates the average complexity for each sub-cluster. A clear split by complexity appears at the first level and intensifies with depth, supporting the idea that some regions of the feature space are inherently more complex than others.Clustering by Complexity. Figure 14 explores how features are organized based on their complexity. Panel A shows a UMAP visualization with points colored according to their complexity scores. The visualization reveals distinct regions of varying complexity, indicating that the distribution is structured. For instance, features related to animals display higher complexity, likely because these require fine-grained and precise detectors. In contrast, simpler features, such as color detectors or low-frequency patterns, cluster together in less complex regions. Panel B displays a dendrogram with four hierarchical levels, where each level introduces additional splits, and the mean complexity is calculated for each sub-cluster. The pronounced division in complexity at the first level, which sharpens as we delve deeper, suggests that the feature space is compartmentalized based on complexity. A promising direction for future research would be to align these complexity clusters with known visual cortical areas to explore potential correspondences. Clustering by Importance. Finally, we investigate the third hypothesis: the presence of regions within the feature space that contain features with higher predictive importance. Neural networks exhibit a diversity in how features are encoded, ranging from local to distributed representations [25, 29] . In the local encoding scenario, a single neuron is primarily responsible for encoding a feature. On the other hand, in a distributed encoding scheme, features are represented by the coordinated activity of multiple neurons, often deviating from canonical axis-aligned directions. More specifically, features could be distributed across many neurons-sometimes densely, or in a pseudo-distributed fashion-rather than being localized to a specific neuron. This theory is one of the main motivations behind employing overcomplete concept extraction methods [18, 32, 92, 35] .The distinction between local and distributed encoding is critical, particularly when extracting overcomplete features from a large-scale dictionary. In our analysis of 10,000 features, we assess whether these features are encoded in a local or distributed manner. A local encoding would imply that the feature vector d is aligned with a canonical vector, i.e., d ∈ {e 1 , ..., e n }, where n = |A ℓ | is the dimensionality of the activation space and e i represents the one-hot canonical vector for the ith neuron. In contrast, a fully distributed feature would be characterized by non-zero values across all neurons, with no alignment to any single axis.To quantify the extent of this local or distributed encoding, we use the Hoyer score. This score, which ranges between 0 and 1, captures the sparsity of a vector by comparing its ℓ 1 -norm to its ℓ 2 -norm, with a correction term for normalization. Formally, the Hoyer score is given by:Hoyer(d) = √ n -||d|| 1 /||d|| 2 √ n -1 .For each feature in our dictionary D = {d 1 , ..., d 10,000 }, we compute the Hoyer score to determine whether the feature is locally encoded (score near 1) or distributed (score near 0).Figure 16 illustrates the results of this analysis, revealing significant variability in the degree of distribution among features. Some regions of the feature space show highly distributed encoding, while others exhibit more local representations. This analysis was conducted on a ResNet50 model, and it is possible that the distribution of encoding strategies varies across different architectures.Figure 17 : Feature Complexity vs. Distributed Encoding. We show that there is no clear relationship between feature complexity and the degree of distributed encoding. Whether a feature is encoded by a single neuron or distributed across multiple neurons does not seems to be determined by its complexity.J Replicating the feature flow results with other measures and models. A legitimate consideration arises when revisiting Section 4, where we introduced the hypothesis that simpler features are primarily carried through the residual connections of a network, while more complex features are progressively constructed through interactions between the main branch and residual connections. The measure we originally used to support this hypothesis was CKA, which serves as a proxy to assess the similarity between the activations at a certain stage of the model and the final state of a concept. However, one might wonder why not use V-information directly.Figure 18 presents the results of this replication. Although the scale of the values differs from those in Section 4, the overall trend remains consistent. The left panel shows the dynamics of simpler features. Early in the network, the main branch carries a significant portion of the Vinformation, which diminishes as the simpler features are gradually "passed along" through the residual connections. Notably, even as the information content decreases, the absolute quantity of V-information remains higher for simple features compared to complex ones. This indicates that while simpler features are transported through the residual connections, they are not entirely depleted of their information content. The right panel of Figure 18 demonstrates the progressive construction of more complex features. Here, we see that both the main branch and residual connections contribute to the gradual accumulation of information necessary for these complex features. This supports our initial hypothesis: complex features do not traverse the network intact but are built up in a cumulative process, drawing on multiple layers and branches to form intricate representations.For compute consideration, this replication was conducted using a different experimental setup than that of Section 4. For this analysis, we employed the validation set of ImageNet to build the dictionary instead of the train set. Both the dictionary and the model were different from those used in the main body of the paper. Specifically, the model used here was the ResNet50 implementation from the Keras library [26] . Despite these differences in experimental conditions, the overarching trends observed in our original CKA-based analysis are preserved, bolstering the validity of the flow hypothesis.In this section we recall some of the most important complexity measures like Kolmogorov complexity, its computationally tractable counterpart the Levin complexity, and finally we underline the epistemic similarity between these concepts in deep learning.Kolmogorov complexity [55] is a measure of the complexity of an object. The objects (image, video, text, pdf, etc.) can be ecnoded as a sequence (u n ) ∈ Σ N of symbols over a finite alphabet Σ. A program is a finite sequence P ∈ L written in language L ⊂ Σ * (e.g a Python source file).L (u n ) is the length of the shortest program P : N → Σ * that producesThere exists numerous variants of this definition with slightly different behaviors [65] . Since deriving theoretical results is not the focus of our work, we decided to tradeoff precision for simplicity of exposure.the n-th first terms of the sequence (u n ):EQUATIONIntuitively, if the sequence is highly compressible, the program will be short. For example, the sequences [1, 2, 3, 4, . . .] or [2, 4, 8, 16, 32, . . .] are few lines of code in most programming languages. Conversely, if the sequence is purely random, then no finite-length program exists. The digits of π, seemingly without structure, are not Kolmogorov random since there exist short programs computing them. The famous Cantor's diagonal argument [20] shows that most sequences are random, since no bijection exists between Σ * (countably infinite) and Σ N (cardinality of the continuum). The definition implicitly assumes a specific computation model (Python interpreter, C++ compiler, Turing machine) to describe the language. However, by definition Turing-complete models can simulate each other, which implies there exist a universal constant C(Python|C++) such that for all sequences u n we have K (∞)Python (u n ) ≤ K (∞)C++ (u n ) + C(Python|C++). This constant corresponds to the length of a Python interpreter written in C++ for example. In general, this holds for any other pair of languages. Therefore, if K (∞) L (u n ) → +∞ as n → ∞ for some language L, then it is true in every other language: intrinsic randomness is universal in this sense [98] .Levin complexity. Kolmogorov's complexity suffers from an important drawback: it is not Turingcomputable. Put another way, there exists no algorithm that computes K (∞) . Fortunately, by regularizing K (∞) appropriately it is possible to make it computable. Levin [64] proposed to regularize the cost with the runtime T (P, n) of program P on input n. This is the Levin complexity:EQUATIONThis modification makes K (T ) (u n , L) computable with the Levin Universal Search algorithm (see Alg. 1). Informally, instead of looking for a shortest program, this algorithm seeks algorithms that run fast among those who are shorts. It is obtained by iterating over lengths i ∈ N, and by running exactly one step of computation of all these programs in parallel. The first program P that halts on u n minimizes K (T ) . This is a central property of Levin's universal search: the first programs found are the simplest and the ones requiring the lesser compute [4, 108, 13] . for each P ∈ S in parallel do 6:Run P for exactly 1 step.if P halts on u n then 8:return P Deep learning and simplicity bias. The links between algorithmic information theory and deep learning have been a recurring although spurious interest throughout the years [94, 95, 75, 61, 68, 40] .Neural networks are a special kind of program, composed of the source files required for inference, and the weights embedded in the network. Therefore, results related to the complexity of sequences apply transparently. Program length (Kolmogorov) and program runtime (Levin) are tightly linked since deeper and wider networks also consume more FLOPS during inference. Smaller networks implement simpler programs. Similarly, features that can be decoded "early" in the network are simpler than those requiring all the layers. The idea is often coined as Minimum Description Length (MDL) principle [16] , Occam's Razzor, or even simplicity bias [49] .As for many things in machine learning, regularization helps.Task. Here, we have studied a specific CNN architecture, ResNet50. In future experiments, it will be useful to investigate whether other model families exhibit similar feature learning, including in domains beyond vision.Architecture. The residual connections of the ResNet are shared by other architectures like Con-vNext [67] or Vision Transformers (ViT) [28, 14] . The works of [109] indicate that findings from convolutional models may transfer to ViTs. Furthermore, the work of [116] suggest that features in convolutional networks and ViT are of similar nature. However, [90] found significant quantitative differences between the layers of ResNets and ViTs, highlighting the need for further empirical testing.Training Dynamics The observed dynamics of feature learning, including the emergence of complex features and the reduction in the complexity of important features later in training, are based on a specific training schedule and set of hyperparameters. To accurately attribute these findings, a more comprehensive study is required to evaluate the role of various factors such as the learning rate scheduler and weight decay. Future research should systematically investigate how these and other training parameters influence feature complexity and importance.Nested predictive families. Our complexity metrics rely on the hypothesis that the different predictive families associated to the network up to depth f ℓ (x) are nested, i.e. that stacking more layers strictly increases expressiveness. This is highlighted in the relevant assumption in section 2. If this hypothesis is violated, the true complexity of a feature may be overestimated in deeper layers. This is typically the case at the early stages of training.Dictionary of features. Regarding the building of the dictionary using NMF, a previous study [32] has shown that the specific dictionary learning method yielded a favorable tradeoff between several criterions such as sparsity, reconstruction error, or stability. Other dictionary learning methods (like sparse-PCA, K-Means or sparse auto-encoder) may yield a bank of concepts with different properties. 


Decision: This paper proposes a new feature complexity metric for deep neural networks, and explores the learning dynamics of features with different complexities, the flow of different features in forward propagation, and the importance of different features to the output. After an active discussion, three reviewers leaned towards acceptance, while two leaned towards borderline rejection. The reviewers praised the good presentation of the paper, the novelty of the proposed feature complexity metric, and the thoroughness of the related literature. The reviewers were concerned about the generalizability of the findings as the paper only conducted experiments on the ResNet-50 model, and the lack of comparison between the proposed metric with other feature complexity metrics. One reviewer was also concerned about significance of the findings, while another reviewer held an opposite view that the paper added significant depth to existing knowledge. During the rebuttal period, the authors discussed about the limitations of only studying the ResNet-50 model, and compared between the proposed metric and other metrics (e.g., the CKA metric). The reviewers feel most of their concerns are addressed after the rebuttal period.

Reviews: review: paper_summary: paper_summary: This paper proposes a new measure of feature complexity based on an information-theoretic metric, the $\nu$-information metric. Utilizing this complexity measure, the paper shows (1) visualization of features of different complexities, (2) simple features are propagated through the residual connections to reach the final layer, (3) simple features are learned earlier than complex features during training, and (4) simple features usually have a higher importance (weight) to the output score of the model.
strength_weakness: strength_weakness: My major concerns for this paper are its coherence and significance.

1.	The paper lacks coherence because multiple components (e.g., metrics, algorithms) introduced in this paper do not come from the same theoretical framework. For example, the notion of feature complexity in this paper (the $\nu$-information metric) is drawn from information theory, while the method for extracting features (the Craft method) is based on non-negative matrix factorization and has little connection with information theory. The same issue applies to the feature visualization method for demonstrating features of different complexities, and CKA metric, and the importance metric $\Gamma(z_i)$. These metrics/algorithms are borrowed from different theoretical frameworks and contexts, so it is in doubt whether they can be used together. I would appreciate it if the authors are able to re-organize all components of the paper under a coherent framework, e.g., the information theoretic perspective (if at all possible).

2.	My second concern is about the paper’s significance, since many claims have been discussed in previous works and appears non-novel. The conclusion that simple features are usually color/edge detectors that are located in earlier layers is not surprising, nor do the conjecture that simple features tend to propagate through the residual connection. For the conclusion that simple features are learned earlier than complex features, there are many studies leading to this conclusion, from the perspective of spectral bias [cite1], game-theoretic interactions [cite2], or frequency [cite3]. Overall speaking, I do not learn new insights from the paper. Nevertheless, as mentioned in the 1st point in weaknesses, if the authors are able to re-organize the paper from the purely information-theoretic perspective, it will be a more intriguing work.

3.	Using K-means clustering to aggregate features into meta-features may lead to incorrect clustering results. K-means clustering is known for its poor performance for data clusters that are not circular-shaped and are of discrepant sizes. I am not sure how features are distributed in this paper’s experiments, but more advanced clustering strategies such as the spectral clustering or hierarchical clustering can be applied to mitigate this issue.

[cite1] Rahaman et al. On the Spectral Bias of Neural Networks. ICML, 2019.

[cite2] Liu et al. Towards the Difficulty for a Deep Neural Network to Learn Concepts of Different Complexities. NeurIPS, 2023.

[cite3] Xu et al. Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks. ICLR, 2020.
questions: questions: 1.	The paper claims that simple features tend to propagate through the residual connection, while complex features tend to propagate through the main branch of the network. Then a natural question arises: how are simple and complex features propagate in CNNs without residual connections (e.g., AlexNet, VGG)?

2.	About the visualization of simple and complex features in Appendix B. The feature *fences* is among the most simple features, while the features *whiskers* and *dotted texture* are among the most complex features. However, I do not see an essential difference between the features *fences*, *whiskers*, and *dotted texture* from the sample images in Figure 7 and Figure 8. For example, *fences* and *dotted texture* both seem to consist of lines that cross over each other to form holes. Similarly, the *whiskers* feature is also composed of lines, although the lines do not cross over each other but appear parallel. Why is *fences* a simple feature, but *whiskers* and *dotted texture* complex features?

Another question is that previous studies show that CNNs are usually biased towards textures, i.e., they tend to learn textures as a shortcut solution, but why is the *dotted texture* here measured to be a complex feature?
limitations: limitations: NA
paper_summary: paper_summary: The paper introduces a novel metric for quantifying feature complexity in deep learning models, specifically focusing on an ImageNet-trained ResNet50 model. This V-information-based metric captures whether a feature requires complex computational transformations for extraction.  The study addresses four key questions:
(1) The appearance of features as a function of complexity.
(2) The learning timeline of these features during training.
(3) The flow of simple and complex features within the network.
(4) The relationship between feature complexity and their importance in the model's decision-making.
The study reveals that simpler features dominate early in training and are transported through the network via residual connections, while more complex features emerge gradually and require more computational effort. Interestingly, the most important features tend to be simpler and become accessible earlier in the network, suggesting a sedimentation process.
strength_weakness: strength_weakness: (1) Generalizability: While the findings are insightful, they are based on a single architecture (ResNet50) and dataset (ImageNet). The results might not generalize to other models or tasks without further validation.

(2) Complexity Metric: The assumption that each layer optimally represents features for downstream linear probes may not hold universally, potentially leading to overestimated complexity scores in some cases. More empirical validation across different models and tasks could strengthen the proposed metric's robustness.

(3) Temporal Analysis: The focus on specific epochs (e.g., epoch 90) may overlook important dynamics occurring at intermediate stages of training. A more granular analysis could provide a clearer picture of feature evolution, i.e., Can you analyze the changes in importance during the training process？
questions: questions: (1) How would the proposed V-information complexity metric perform on different neural network architectures, such as Transformers or GANs?
(2) Can the authors provide more details on how the complexity metric correlates with other existing complexity measures in literature? Which one provides better insights?
(3) How does the model's simplicity bias, as observed in the study, impact its generalization performance on unseen data?
(4) Are there specific strategies that could be employed to mitigate the simplicity bias and encourage the learning of more complex, yet important features?
(5) Does the conclusion of section 4 still hold on networks outside ResNet? How to explain the architecture of the transformer regarding the residual connection?
(6) Why does the horizontal axis in Figure 4 show several blocks instead of all blocks?
limitations: limitations: While the paper provides significant insights into feature complexity in deep learning models, there are several limitations that need to be addressed:

Model and Dataset Scope: The study focuses solely on a ResNet50 model trained on the ImageNet dataset. This raises concerns about the generalizability of the findings to other neural network architectures (e.g., Transformers, GANs) and different types of datasets. Future work should validate the proposed metric across a variety of models and tasks to ensure broader applicability.

Assumption on Optimal Representation: The complexity metric relies on the assumption that each layer in the network provides an optimal representation for downstream linear probes. However, this assumption may not always hold true, potentially leading to overestimated complexity scores. Further empirical validation and adjustments to the metric may be necessary to account for cases where this assumption is violated.
paper_summary: paper_summary: The paper introduces a metric based on V-information to measure feature complexity in deep learning models. Using ResNet50 trained on ImageNet,  it explores feature spectrum, training dynamics, network flow, and decision impact.

The study also highlights the role of simplicity bias and the evolution of feature importance in neural networks.
strength_weakness: strength_weakness: 1) Not Easy To Follow On Math: The paper's mathematical sections are difficult to follow. Clearer and intuitive explanations would improve accessibility.
2) Limited Model Diversity: The study focuses solely on ResNet50, ignoring modern architectures like depth-wise separable CNNs.
questions: questions: 1) The paper states that both $z$ and $D^*$ are positive due to the use of Non-Negative Matrix Factorization, which aligns with the nature of ReLUs. Could you elaborate on why negative values for $D^*$ wouldn't work, despite the positive nature of ReLU? It's not clear why allowing $z$ and $D^*$ to be negative would be problematic, especially considering that $(-z)(-D^*) = zD^*$.

2) In line 126, $z$ is defined as $z \in \mathbb{R}$. Given that $z$ is described as positive elsewhere, wouldn't $z \in \mathbb{R^+}$ be more precise? Or are you intentionally allowing negative values for $z$?

3) The nature of $z_1$, $z_2$, and $z_3$ in Figure 1 is unclear. If these represent the $i$-th element of $z$, they would be scalars, which doesn't seem to align with what's depicted. Could you clarify what these variables represent?

4) Figure 1 shows feature visualization across different layers, and the paper mentions earlier layers elsewhere. However, line 622 states that feature extraction was done only for the penultimate layer. Could you explain this and clarify how earlier layer information was obtained if extraction was limited to the penultimate layer?
limitations: limitations: The authors have addressed limitations in the appendix.
paper_summary: paper_summary: The current work proposes a method to assess the complexity of features in a representation space in terms of usable information.  The measure of complexity is related to how far back in the layers of a trained network one can find information about the feature that is recoverable by a linear decoder.  Equipped with a measure of complexity per feature, the authors grab 10k distributed features in the penultimate layer of a ResNet50 and then examine various relationships such as the growth in complexity over the course of training, the relative importance of simple features in determining the output, and the “flow” of simple features through the residual backbone of the network.
strength_weakness: strength_weakness: Many of the analyses are questionable, in my opinion, making the work seem to have a strong premise with a lackluster follow up.  I will be happy to be rebutted on these points with justification and clarifications during the discussion period.

Specifically, the “what” and “where” analyses are odd to me.  My main gripes are below, with smaller points in the "questions" section.
- Regarding the qualitative exploration of dictionary vectors, and their grouping into clusters (for “what”): is there a good reason to think relative positions mean anything in feature space, and therefore that clustering is meaningful?  Were the dictionary vectors normalized beforehand, or else did the variable magnitude affect the UMAP and clustering?  Why were 150 clusters used?  Did a sweep over k or any other analysis indicate that the vectors are indeed clustered to some degree, and 150 is an appropriate choice?  Unless I missed it, it’s not explained how the meta-clusters were labeled -- was it just visual inspection, and the remaining 120 clusters simply weren’t interpretable?
- Regarding the use of CKA for “where”: The proposed complexity measure is already based on how far back in the processing the feature is still recognizable to a linear probe.  Why show CKA instead of the usable information as a function of depth for simple vs complex features?  It’s not intuitive to me to infer anything about the similarity of the “residual” branch aka simply the feature vec one layer deeper, compared to that of the delta (“main branch”), which is not used as a feature by the network.  It seems far more straightforward to look at any changes in the usable information as a function of depth as arising from the delta at that layer.  Why not just show the curves for each of the features?  The curves would also show the way in which usable information is growing with depth -- is it gradual, or stepped?  All said, it’s hard to find much insight from the results of Fig 4, and a straightforward alternative exists.
questions: questions: - In Appendix A: “The dictionary D was designed to encapsulate 10 concepts per class”.  Does this mean the NMF was run separately for each class, and then the dictionaries were all combined?  If that’s the case, are there multiple copies of some features (e.g. “wheel” being found for every vehicle class)?
- How were the visualizations of Fig 1 produced, particularly the earlier manifestations z2 and z3?  How should we interpret a feature’s identity in an earlier layer when it is not recognizable in that earlier layer (and therefore a function of many features)?
- Fig 6: What are we supposed to infer from the epoch 1 results?  If everything is still essentially random at this stage of training, then we're just seeing meaningless values plotted against each other, right?
- The inhibitory/excitatory results (Fig 12) are striking -- **every single inhibitory feature has complexity less than ~0.5?**  While the other findings are rough trends extracted from point scatter, this finding is relegated to the appendices and given minimal attention?

Questions out of interest rather than needing to fill a gap in the paper:
- Regarding the higher importance of simpler features: are they simply present more often, or are they more influential?  I wonder if there's a frequency-complexity relationship that would be insightful.
- Did you do any of the same analyses with the individual neurons in the penultimate layer (i.e. comparing local vs distributed)?
limitations: limitations: There is a reasonable discussion of limitations in the Appendices.
paper_summary: paper_summary: This paper proposes a method to measure the complexity of features extracted by deep learning models. This method is based on $\mathcal{V}$-information, an extension of Shannon's mutual information that takes the computational capabilities of a decoder into account. The proposed measure of feature complexity is inversely related to the cumulative $\mathcal{V}$-information across a model's layers, the intuition being that features that become available at earlier layers will accumulate a larger $\mathcal{V}$-information, while features that only become available late will have a smaller $\mathcal{V}$-information.
Features are extracted from a standard, ImageNet-trained ResNet50, from which an overcomplete dictionary is learned. These features are then clustered, and the mean complexity score for each feature is computed. Feature clusters with low, intermediate and high complexity scores are visualized, revealing that low-complexity features tend to be related to uniform colors or low-frequency information, intermediate-complexity features to local shapes such as eyes and noses and high-complexity features to highly structured shapes, such as insect legs.
Using the CKA between the features in the dictionary and the activations at different layers within the main branch and residual branch of the network, the authors find that Easy features tend to emerge in early layers and then copied to later layers through the residual branch, while Complex features steadily increase throughout layers in both branches.
Analyzing the dynamics of different features' emergence throughout training, complex features are found to emerge later than simple ones. Interestingly, analyzing the interplay between features' complexity and their importance in influencing the network's outputs, the authors find that (a) more important features tend to be less complex, and (b) the complexity of important features is reduced throughout training, suggesting that the network might be "compressing" the more important features.
strength_weakness: strength_weakness: - The proposed measure is closely related to the accuracy with which a feature can be linearly decoded from the layers of a network. I believe the paper would benefit from an explicit discussion of what, exactly, is the additional information provided by the proposed method which cannot be gained from directly looking at the "raw" decoding accuracy.
- For simplicity, the authors restrict their analysis to a single ResNet model. While this is an understandable choice, I am not sure about what the implications are for models which do not include residual connections. The residual stream is found to be central in "teleporting" simple features to later layers. The authors should explicitly discuss what they predict the pattern of results would be for networks without residual connections. For example, would simple features be overall less relevant to the network's responses? Or would the network implicitly implement a residual-like stream? The limitations of using this architecture exclusively are briefly acknowledged in the Limitations section, but the specific role of residual connections is not discussed there.
- The proposed measure is based on a loose assumption that the decodability of features tends to increase across layers. However, it is possible that certain features are _only_ available in early layers (for example, certain low-level image features might be discarded). These features would have a low cumulative $\mathcal{V}$-information, and might thus be erroneously classified as high-complexity. The authors should explicitly discuss whether this is a concern at all, and for what reasons.
- The plot in Figure 5B shows that more important features tend to become less complex as training progresses. As the authors have access to the features which are subject to this process, and those which are not, it would be extremely interesting to visualize them, showing what exactly is happening - are the same features being extracted at earlier layers? Or are they actually changing, and starting to resemble simpler features such as colors or edges? The answer to this question would clarify more precisely what the proposed complexity measure is capturing. Is it possible that features we visually evaluate as "complex" can be extracted at early layers if they are important to the network's task? Or is layer depth always correlated with visual complexity?
questions: questions: No questions, beyond the requests for clarification listed in the weaknesses section.
limitations: limitations: Limitations are discussed adequately. Please refer to the weaknesses section for comments about limitations which I believe were not sufficiently discussed.

Comments: title: Thank you for increasing your score
comment: >  the main point [...] is to justify that all these algorithms and metrics can used together and do not conflict with each other
> Is Craft truly compatible with the proposed v-information metric? 

Can you clarify what you mean by "conflicting with each other" ? None of these frameworks operate with mutually exclusive hypotheses.  

> These components are expected to be organized in a coherent framework

This is rather ambitious, as it would require a comprehensive understanding of all phenomena arising in deep learning training. We believe our work provides a first step into the direction of identifying promising components of a "coherent framework".  

> why choose the Craft method to extract features but not other methods?
> The same question can be asked for other components: the feature visualization method, the CKA metric, the importance metric , etc

All these methods are published, and have demonstrated their effectiveness in deep learning benchmarks, including human experiments. 

We observe consistent results using different tools and theories. This diversity is a strength, not a weakness: this is precisely what makes us confident in the robustness of our observations. As all results and metrics point toward the same direction, we are confident to not overfit a single method.
I thank the authors for their rebuttal. Their answers are very helpful to clarify the few points I was unsure about, or to acknowledge when open questions still remain (as in the role of residual connections and the tracking of features during training). Just for clarification, in my point about decoding accuracy I was indeed referring to MSE. I apologize for using the "accuracy" terminology which suggests a discrete classification, that was probably confusing.

Given that my review was very positive in the first place, and my concerns were quite minor, I will keep the same score.
I would like to thank the authors for their response. The explanation for using the K-means method, and the complexity of the *fences* and *dotted texture* feature seem reasonable. Considering this effort, I have raised my score to 4.

However, I cannot give a higher score because my concern for the coherence and significance of the paper remains. To further clarify the coherence issue, the main point of "multiple components introduced in this paper do not come from the same theoretical framework" is to justify that all these algorithms and metrics can used together and do not conflict with each other. For example, why choose the Craft method to extract features but not other methods? Is Craft truly compatible with the proposed $\nu$-information metric? The same question can be asked for other components: the feature visualization method, the CKA metric, the importance metric $\Gamma(z_i)$, etc. These components are expected to be organized in a coherent framework, or at least be justified for the particular choices, rather than appear like a combination of engineering techniques. I hope this paper can be more impactful in doing so.
Thank you for your feedback! We're glad our responses helped clarify the analysis. We'll make sure the comparison with CKA is included in the appendix as suggested. We appreciate your support and the score increase. Thanks again, and we hope the final version meets your expectations!
Dear reviewers,

The discussion period will end soon. Please read the rebuttal from the authors and participate in the discussion, as soon as possible.

Thank you.

Best,

AC
I thank the authors for their rebuttal and their time. Their clarification were indeed insightful for better understanding the paper.

After reading other discussions, and reading the paper one more time, I'm even more unsure of my assessment. I will stand by my current rating as I think the authors put a lot of effort in this paper, and their results are reasonable, but I will downgrade my confidence to 1.

I hope the best for the authors.

Best regards,
Thank you for the responses.  I would have liked to see in a rebuttal pdf the usable information vs depth comparison to CKA ("we have included these additional findings in the appendix as suggested"), but I'll take your word for it, and look forward to checking it out in the published version.  

All told, this paper (refreshingly) gave me a lot to think about, and I'm more convinced after the response about the soundness of the analyses.  I'll raise my score to a 7; good luck with acceptance.