On the Joint Interaction of Models, Data, and Features
Authors: Yiding Jiang
Christina Baek
J Zico Kolter
Venue: ICLR 2024 oral (2024)
Keywords: Generalization
feature learning
empirical phenomena
Primary area: visualization or interpretation of learned representations
TL;DR: We propose a framework for feature learning that can explain previously not understood phenommena in deep learning.

Abstract: Learning features from data is one of the defining characteristics of deep learning,
but the theoretical understanding of the role features play in deep learning is still in
early development. To address this gap, we introduce a new tool, the interaction
tensor, for empirically analyzing the interaction between data and model through
features. With the interaction tensor, we make several key observations about
how features are distributed in data and how models with different random seeds
learn different features. Based on these observations, we propose a conceptual
framework for feature learning. Under this framework, the expected accuracy for a
single hypothesis and agreement for a pair of hypotheses can both be derived in
closed form. We demonstrate that the proposed framework can explain empirically
observed phenomena, including the recently discovered Generalization Disagreement Equality (GDE) that allows for estimating the generalization error with only
unlabeled data. Further, our theory also provides explicit construction of natural
data distributions that break the GDE. Thus, we believe this work provides valuable
new insight into our understanding of feature learning.

Published as a conference paper at ICLR 2024
ON THE JOINT INTERACTION OF MODELS, DATA, AND FEATURES
Yiding Jiang
Christina Baek Camegii Mellon University kbaek0cs cmu. eddu
J. Zico Kolter Carnegie Mellon University Bosch Center for Al zko terêcs cmc edu
Cameeiie Mellon University yiding c&cs. cmu. edu
ABSTRACT
Learning features from data is one of the defining characteristics of deep learning, but the theoretical understanding of the role features play in deep learning is still in early development. To address this gap, we introduce a new tool, the interaction tensor, for empirically analyzing the interaction between data and model through features. With the interaction tensor, we make several key observations about how features are distributed in data and how models with different random seeds learn different features. Based these observations, we propose a conceptual framework for feature learning. Under this framework, the expected accuracy for a single hypothesis and agreement for a pair of hypotheses can both be derived in closed form. We demonstrate that the proposed framework can explain empirically observed phenomena, including the recently discovered Generalization Disagree- ment Equality (GDE) that allows for estimating the generalization error with only unlabeled data. Further, our theory also provides explicit construction of natural data distributions that break the GDE, Thus, we believe this work provides valuable new insight into our understanding of feature learning.
INTRODUCTION
It is commonly said that deep learning performs feature learning, whereby the models extract useful patterns from the data and use the patterns to make predictions. Most successful applications of deep learning today involve first training the models on large amount of data and then fine-tuning the pre-trained model on downstream tasks (Chen al., 2020; Brown et al., 2020; Radford et al., 2021). Their success suggests the models are learning useful and transferable knowledge from the data that allows them to solve similar tasks more efficiently. Experimentally, many different works (Nguyen et al., 2016; Zeiler and Fergus, 2014; Bau et al., 2017; Olah et al., 2017: 2018; Li et al., 2015) have studied various aspects of the features learned by deep neural networks. These works help the community gain a better intuitive understanding of the mechanisms underpinning deep learning as well as improve the interpretability of deep models. However, to the best of our knowledge, the theoretical understanding of the role features play deep learning still under-explored For example, the popular neural tangent kernel (NTK) (Jacot et al., 2018) 18) framework analyzes deep learning as kernel regression with features defined by the model's gradient at initialization, which is independent of the training data.
While it may be intuitive to think of defining features quantifying the "unit" of information in data that models use to make predictions, the community has yet reach a consensus on the exact definition of features in deep learning beyond toy models. Nonetheless, is undeniable that the models have learned something from the data. In fact, the same models trained with different random seeds would learn different information that leads to different predictions (Lakshminaray anaan et al., 2017). This phenomenon has important downstream consequences for ensembling randomly initialized networks including better generalization (Allen-Zhu and Li, 2020), calibration (Lakshminarayanan et al., 2017), and the Generalization Disagreement Equality (GDE) (Jiang et al., 2022) where the expected test accuracy is equal to the expected agreement deep ensembles.
We postulate that a good definition of features should be fine- aie-igrinedd enough to differentiate the difference in the knowledge of different models. To this end, we attempt define such a construct that allows us to analyze the similarities and differences of information learned by different models, while also remaining amenable to quantitative and theoretical analysis. Through this definition,

---

Published as a conference paper at ICLR 2024
Least features
Most features
Airplane
Car
Cat
Dog
Horse
Figure I: Visualization of images with the least features leatures (left) and the most features (right) for classes of CIFAR 10 under our feature definition (defined in Section 3.1). Each row corresponds to one class of CIFAR 10 (zoom in for better viewing quality). We can see that the images with the least features semantically similar to each other whereas the images with the most features much more diverse and contain unusual instances of the class or objects in rare viewpoints. More examples for all classes can be found in Figure 12 of the appendix.
can gain insights into the behavior of models and the underlying mechanisms of feature learning. Notably, we show that GDE arises immediately as a consequence of how neural networks learn appropriately defined features. This phenomenon was previously explained by assuming calibration of the underlying ensemble, which is often a strong assumption to make.
We begin with an empirical investigation of feature learning, using a natural definition of features on real data (Figure 1) that allows to easily compare information about the data distribution learned by different models and a construction we propose called the interaction tensor. We define features to be the projection onto the principal components of different models' last-layer activation. The interaction tensor then jointly models the features learned by multiple models and across multiple data points. Inspecting this tensor constructed on collections of models, we find that the model of data presented in Allen-Zhi and Li (2020) is not conceptually reflective of the actual observed phenomena, suggesting that an alternative model is needed.
Based on these observations, we propose an alternative (still simplified) model of feature learning, which better captures the above phenomenon. Specifically, we posit a framework where features come in two types: dominant (more frequentt and rare (less frequent). which captures the observed heavy-tailed nature of features. We also assume that data points either contain small number of dominant features or a large number of rare features and that models learn features according to their frequency in the data set; this captures the observed phenomenon where data points with predictions. Under this model, we can analytically derive
fewer features receive expressions for the accuracy and agreement of resulting classifiers. Despite the simplification, show that our framework naturally captures the phenomenon of GDE without assuming calibration. Instead, we show that GDE arises immediately as consequence of the distributional properties of features in natural data. Finally, we demonstrate that the framework can make accurate predictions about the effects of merging classes and changing data distribution on GDE and calibration, leading to the construction of natural data distributions that break GDE.
Thus, we believe overall that our framework of feature learning shows promise as an additional useful and valuable conceptual tool in understanding how deep learning works. Note that we do not attempt to derive how our model can arise mechanistically via optimization, but we believe that this can be considered strength of our approach. Similar to natural sciences and econometrics, the empirical phenomena of deep learning can be phenomenologically understood "on their own terms" at many different layers of abstraction, and our model provides one such formalism that comports with observed behavior at the level of observed feature learning phenomena.
2 RELATED WORKS
Feature learning. Representation learning (Bengio al., 2013) the practice of discovering useful features from raw data directly instead of using hand-crafted features. Deep learning is the de facto approach for learning features from large amount of data (Chen et al., 2020; Brown et al.,
2

---

Published as a conference paper at ICLR 2024
2020: Radford et al., 2021), but formal understanding of the process is still lacking. Recent works have started to incorporate feature learning into theoretical analysis (Li et al., 2019; Allen-Zhu and Li, 2020; Yang and Hu, 2021; Karp et al., 2021; Wen and Li, 2021; Allen-Zhu and Li, 2022; Ba et al., 2022). This paper is most immediately related to Allen-Zhu and Li (2020) which proposes the multi-view data structure where there exist two types of data: multi-view data which contain all the features of class and single-view data which contain only one feature. In this work, we investigate whether this structure of features holds in practice by treating features as first-class citizens both empirical investigation and theoretical analysis. Our experimental results reveal a more nuanced perspective on the structure of data and "features". Based on these observations, we propose an alternative conceptual model that better reflects how features, data, and models behave in reality. We analyze the generalization property of the model and also its agreement property (Nakkiran and Bansal, 2020). We also show that this feature learning model provides an alternative condition under which the curious GDE phenomena observed in Jiang al. (2022) can arise.
Ensemble and Generalization Disagreement Equality. Deep ensembles (Lakshminarayanan al., 2017), trained with different initializations on the same dataset, have demonstrated superior performance over classical methods such as bagging and Bayesian approaches (Breiman, 1 1996; Welling and Teh, 2011; Gal and Ghahramani, 2016). One of the notable attributes of deep ensembles is their well-calibrated nature- - making predictions with appropriate confidence levels (Murphy and Epstein, 1967). Jiang et al. (2022) explored the implications of this calibration by showing that test accuracy can be estimated using the agreement of two independent models (Nakkiran and Bansal, 2020), a phenomenon that the authors called the Generalization Disagreement Equality (GDE). However, calibration an open problem (Fort et al., 2019). To address this gap, this work shows that the phenomenon of GDE can arise within feature learning context, without making explicit calibration assumptions.
strong assumption why deep ensembles are calibrated is largely
Understanding representations. One line of work tries to understand deep learning with a more empirical approach. Many try understand the features by visualizing what aspects of the input data they correspond to (Nguyen et al., 2016; Zeiler and Fergus, 2014; Bau et al., 2017; Olah et al., 2017; 2018). Another line of work attempts to compare representations of different models (Li et al., 2015; Raghu et al., 2017; Morcos et al., 2018; Kornblith et al., 2019). We demonstrate that simple PCA can reduce the redundancies in high-dimensional representation and find a parsimonious set of features that the model relies on to make predictions. Our feature clustering algorithm may be seen as generalization of pair-wise matching proposed by Li et al. (2015).
3 CONNECTING MODELS AND DATA WITH FEATURES
In this section, we describe the procedure for constructing the interaction tensor Ω E {0. 1|MxNxT The first axis corresponds to M models, the second axis corresponds to N data, and the last axis corresponds to T features. If the nh data point contains the (th feature and the th model has learned the Th feature, then Pmont would be 1. This tensor captures how features are distributed (see Figure 6 for an illustration). First, we identify features within each model of ensembles. Then, we cluster the features and construct the interaction tensor with the identified feature clusters.
Notations. Let x denote point in X, the input space, and y E [C] denote the label, where [C] is the set of labels, [1, C). Let 9 be the data distribution over X' x [C]. We use (z. y) to denote samples from the random variable following 9. Let f X + [C] be a parameterized function, f(x) = (s' ()))) s X' + Rd maps the input to a d-dimensional representation, and v/ Rd + [C] maps the representation to class. For a collection of models (f1...., we use confidence to denote their average one-hot predictions for the ground truth label.
3.1 PRINCIPAL COMPONENTS OF ACTIVATION AS FEATURES
Many works (Li et al., 2015; Olah et al., 2017) study the representations of deep neural networks by treating individual neurons as features or the most elementary unit of the representation, but these rep- resentations are high- dimennsiioii vectors, which contain redundant information. Furthermore, sin- gle feature may be distributed across multiple neurons. Instead, ideal features should be parsimonious and can capture the dependencies between different coordinates of the representations. These criteria can be fulfilled by dimensionality reduction methods. We choose principal component analysis (PCA)
3

---

Published as a conference paper at ICLR 2024
which captures the linear dependencies between different coordinates of the representations and can be efficiently computed with stochastic algorithms (similar to Härkönen et al. (2020)). While this pro- cedure can be applied to any layer, we use the last layer representation to avoid non-linear interaction through superposition (Elhage et al., 2022). Concretely, given a neural network and a set of data points
X = [xx(0, _,) x(A) we use & E R xd be the matrix that contains all of yp (z(() as its rows. The singular value decomposition (SVD) yields + UEVT where the columns of V Rdxd contain the principal components of Φ. We use the top K principal components V.K and project the
representations R Φ - <V VV (z(1)), V. (x(²) ...... Vikip (z(A))] For notation simplicity, we will use e(x)) to denote V. 1p (a) and Um, to denote the 1thh entry of of
the m model's v(z). Intuitively, we can interpret the principal components as feature, or more
concretely, orthogonal subspaces that the model uses to classify any given data points in X.
3.2 CONSTRUCTING THE INTERACTION TENSOR
Clustering features of different models. Given M models, f2..ff..... ....,.... fas): we can compute the projected representation for each network of the M models, { TTTT T. T For a single model fm and its 1tth feature,
compute its mean and variance:
Mm.k k (a.))~ [U),,(x)], of - ((za))= (um,((z) Mm )²
(I)
For models (f) and their respective th and 6thh features, we can define their correlation to be: ((t.))(a.b)) A E(x.g)= [(vra(=)) ms,a) (1),(()) - (os,a o.
(2)
This can seen as performing the procedure of Li al. (2015) with PCA projected representations. We use K., E []1, 1 to denote the collection of all pair-wise correlation values between the features of f; and fj. and A E [-1, x to denote the collection of all the correlation matrices between every pair of models. A centralized list of notations is provided in Appendix A.
With A, we can identify unique feature clusters in the MK features learned by all models. To account for the arbitrary direction of correlation in A, take the absolute value and use threshold, Toom E (0, 1), to determine whether two features should be considered as the same feature. We use greedy clustering algorithm (Algorithm 1) to match the features with one another, as k-partite matching is known to be NP-complete for
2 (Garey and Johnson, 1979). After running the clustering algorithm, each feature is assigned to one of T clusters (where T < .), and we treat every feature in single cluster as the same feature. The greedy algorithm is efficient and does not generate a fixed number of clusters, which is desirable in cases where some features have low unique cluster. More sophisticated
correlations with other features and should be isolated algorithms, such as graph cut, could be used, but we find the greedy algorithm sufficient for our purposes. See Algorithm 1 and Appendix E.2 for details on the algorithm and hyperparameters.
Matching features to data points Once the features of all models are clustered, we can identify which features are present in each data point of X. First, we normalize each individual Um. by its -oo-no,, which ensures that all features are between 0 and 1. We will denote the now-normalized @P poroj We then pick another threshold, "aata E (0, 1), that decides whether feature is present data point. Concretely, if the t entry of the th row F larger than "data. we assign the nt data point in X the feature cluster containing the m!h model's 1tth feature. In other words, if the mh model's is k feature belongs the (th cluster, we say the nh data point contains the fhh feature. In Figure 1, we visualize the data points with the most and least number of features.
Aggregating Information. After thresholding, we have enough information to construct the inter- action tensor. Each entry indicates whether the tt feature is present in both the th model and nh data point. In the next section, we will inspect various aspects of the interaction tensor S and other experimental artifacts to understand how the models learn features from the data.
4 EXPERIMENTS AND OBSERVATIONS
Experimental setup. We use a collection of M = 20 ResNet18 (He et al., 2016a) trained on the CIFAR-10 10 dataset (Krizhevsky et al., 2009) following the experimental up of Jiang al.
4

---

Published as a conference paper at ICLR 2024
feature frequency over training
feature frequency (45k)
log featune density by confidence (45k)
high coniidecc confidence
-2.0
3.4
v8 grab (base
2.5
3.2
oetumences
3.0
3.0
2.0
2.0
-10
22
8
3
5
feature (162)
feature (1o2)
festure (1e2)
(a)
(b)
(c)
Figure 2: (a) Feature frequency over the course of training. The red curve represents the feature frequency at the initialization. (b) Features how often they appear in the dataset. The features are sorted by frequency and the distribution appears to be long-tailed. (c) Feature frequency by different confidence levels. cowfideeeee data tend to have more low-frequency features.
vs. fealure count. (45k)
data vs. n models w/ feat (45k)
shered feature vs. shared error (45k)
17.5
160
modells with featuee
number offeaturs
15.0
125
12.5
0.41
100
10.0
0.40
75
7.5
0.38
25
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0
2
3
10
12
14
confidance
of data with featura (1o3)
shared fealures
(a)
(b)
(c)
Figure 3: (a) Density estimation of confidence number of features over data. High-confidence data tend to have fewer features. (b) Scatter plot of the number of data with a feature vs. the number of models with that feature. The strong positive correlation suggests that the more data has a certain feature, the more likely the model will learn that feature. (c) Number of shared features shared error. The lower bound of shared error monotonically increases with the number of shared features.
(2022). The models are trained on random 45000 subsets of the whole training set (45k). In addition, we repeat the experiments on CIFAR- 10 using random 10000 subsets (10k) and the SVHN dataset (Netzer et al., 2011) using random 45000 subsets of the whole training set (SVHN). We compute A and Ω the test set using the output of the penultimate layer (i.e., is the final linear layer). The experimental details can be found in Appendix F. We primarily show 45k here and leave the 1 10k and SVHN results to Appendix F, both of which are similar to the 45k results qualitatively.
Observation Feature frequency is long-tailed. (0.1) We use the interaction tensor Ω to compute the frequency of each of the T features in the dataset. First, sum Ω over the model axis and then clip the values to 1 (since we are only interested the relation between data and features here). Then, we sum over the data axis to obtain the number of data points that contain each individual feature. The features are sorted by their frequencies, which reveals long-taii tailed distribution (Figure 2b). Furthermore, this distribution is consequence of learning. In Figure 2a, we compare the feature frequency computed from 20 models over the first 30 epochs of training (i.e., 600 checkpoints) with the feature distribution for untrained models. We observe that untrained models have much higher frequencies for tail features (low frequency). After even single epoch, the frequency of tail features decreases significantly and then stays relatively stable over training. At the head of the distribution, the models first learn a large number of features and then prune the features as training continues, eventually converging to a fixed distribution with small number of features.
The ensemble tends to be more confident on data points with fewer features, and data points with lower confidence tend to have more features with low density. (O.2) Another question that we want to understand is how the features interact with the confidence of the ensemble. In Figure 3a, we show the joint density plot of the ensemble's confidence for a data point and the number of features the data point has for 45k. We can see that low-confidence data tend to have more features, whereas high-confidence data points tend to have fewer features. This finding
Observation 2:
5

---

Published as a conference paper at ICLR 2024
contradicts the model of Allen- and Li (2020) in which if a data point is multi-view (i.e., contains more features), all members of the ensembles will classify it correctly (i.e., high confidence). A plausible explanation for this observation is that there is small sub-population of features that are learned by large number of models. Based on O.I, we postulate that these features are learned by more models because they appear with higher probability. Furthermore, we plot the log density of features' in high-confidence data and all other data points (low-confidence) in Figure 2c. We see that the low-confidence data tend to have more features with low density in Figure 2b. One explanation is that the features in the tail are responsible for different models making different predictions.
Observation 3: Number of models with a certain feature is positively correlated with the feature's frequency. (O.3) The interaction tensor also reveals how the number of data points containing a certain feature relates to the number of models that have learned that feature. This relationship can shed light on how models learn features of different frequencies in practice. In Figure 3b, we observe that the number of models with any given feature has a strong positive correlation with the frequency of that feature appearing in the data (linear / super-linear). This implies that the more a feature appears in the data, the more likely model will pick it up. We hypothesize that the feature learning procedure can be phenomenologicaly approximated by a sampling process where the probability of learning feature is related to how often that feature appears in the data.
Observation 4: Models with similar features make similar mistakes. (0.4) Another natural
hypothesis is that models with similar features should make similar mistakes. For every pair of models, we compare how many features they share to how often they make the same mistake relative to the average number of mistakes both models make (Figure 3c). We can that the lower bound of shared error is almost monotonically increasing with the value of shared features. On the other hand, when models share a smaller number of features, the shared errors have much larger variance, which indicates their errors are less correlated and therefore more random. This effect is more amplified for different architectures. We show this result on more than 20 diverse architectures in Appendix F.6.
Finally, we provide more analysis on the effect of using PCA for clustering in Appendix F.2 and different hyperparameters for the interaction tensor in Appendix F.10. We also explore the properties of features found under our definition in Figure and Appendix F.3, In Appendix F. we investigate the feature distributions at different depths and discuss the interpretation of these features.
5 A COMBINATORIAL FRAMEWORK OF FEATURE LEARNING
In this section, we present a new framework of feature learning for binary classification based on the insights from the experiments. We saw in O.I that the distribution of features is long-tailed. This means a relatively small number of unique features constitute a large proportion of all the features in the data. To facilitate analysis, we will assume there are two types of features, dominant features features and rare features, where the dominant features appear with much higher probability than rare features. Further, we observed that data with high confidence tend to have much fewer features than the ones with high confidence (O.2). To model this behavior, we will assume that there are two types of data points: dominant data and rare data. The former contains small number of dominant features and the latter contains a larger number of rare features. This is another simplification based on O.2, which shows that high-confidence data tend to have fewer higu-frey features.
Definitions and additional notations. Before describing the full model, we first define the pa- rameters of the model as well as some additional notations: Pa: the proportion of all data that are dominant, pr: the proportion of all data that are rare (this parameter is equal to 1 - Pa), c: the total capaciit dominant features available in the data for one class, t,: the total number of rare features available in the data for one class, na: the total number of dominant features single dominant data point has (nd < tz), and n,: the total number of rare features a single rare data point has (n, S t,). We will use T (-) to denote the set of all features a model or a feature has (an expanded list in Appendix A).
model capacity which represents how many features single model can learn, ta: the total number of
Data generating process. We can see the data generating process as the following sampling procedure. First, we decide which class the data point belongs to. We consider a class-balanced
'The features plotted in the same order as Figure 2c (hence the jaggedness). and the density is obtained by normalizing with the total number of features in both high confidence and low confidence groups of data.

---

Published as a conference paper at ICLR 2024
binary classification problem each class occurs with equal probability of 3. Then, we decide whether the data point is dominant or rare. This is equivalent to sampling from a Bernoulli distri- bution, Ber(pa). If the data point is dominant, we sample ni dominant features uniformly without replacement. Vice versa, if the data point rare, we sample n, dominant features uniformly without replacement. It is easy to verify that the proportion of dominant data points and features is Pa and the proportion of rare data points and features pr-
How the models learn. We saw in O.3 that the frequency of features occurring in different models is positively correlated with the frequency at which the features occur in the data. We can model the learning process as another learns a feature is proportional to the frequency at which the feature occurs in the data. Under this assumption, in expectation, Cd = 32Pac of the features in a single model would be dominant features for a single class, and e, = ?p,cc of the features for a single class would be rare. We can further simplify this process by assuming that the model will always sample dominant features for each class and o, rare features for each class?.
process where the probability that a model
How the models make predictions. For a data point x and a model f. we assume that the model will correctly classify if the overlap between the features of and the features of f is not empty (similar to assumptions in Allen-Zie -Zhu and Li (2020)). Otherwise, the model will perform random guess. The expected error that single model f makes on single datum pair (2, y) is thus err (f 2. y) = } (V() n ((x) 23. Further, given pair of models (J,g) and single datum pair (I, v). there are three distinct behaviors for how they will make predictions. (1) The two models will always agree with each other if both of them share feature with x, since both will classify x correctly (i.e., if D (f) n [(x) > 0 and [g) n M (z)) > 0). (2) If the models both do not share any features with z. then by the previous assumptions, the models will make random guesses (see Appendix D.2 for why this is justified); however, if the models share features with each other, their random guesses will not be independent of each other (0.4). We hypothesize that how two models agree with each other is a function of k. the number of features they share, and c. the model capacity. We capture this intuition with an agreement function, Ç N x N - [0, 1]. which returns the probability that two models will agree based on how many features they share relative to the full model capacity. This function is crucial for understanding how models make mistakes. (3) Finally, if the models do not share any features with each other with x, both models will perform independent random guesses, in which case they will agree 50% of the time.
It is natural to ask how reasonable the simplifications are. In Appendix D. we discuss these simpli- fications (e.g., random guess and number of classes) in detail and provide comparison between this framework and Allen-Zhu and Li (2020). We encourage interested readers to read this section. Still, we will see that this relatively simplified model readily offers interesting insights into observed phenomena and can make surprisingly accurate predictions about the results of experiments a priori.
5.1 ANALYTICAL FORMS OF ACCURACY AND AGREEMENT
Using this model, the closed-form form of expected accuracy, Acc, and expected agreement rates, Agr, can be derived through combinatorics. All propositions are proven in Appendix C. Proposition 5.1. The expected accuracy over the model distribution and data distribution is:
(1a-ad)
Acc
1 2
md
+
1
(mp)
Proposition 5.2. Let (-) = O when n < 0, < or < r, and let:
(1a c4)
(1p-en)
+
(dd-nd)a
(p-na-ed) (z)(7))))
( (*amre)
Ce
(T) ("ren (o
(2(k) Dd
Pr
then the expected agreement between an i.i.d pair (f.g) drawn.for the model distribution is:
Ag n (c(H. 2)) 92()).
Both or and ca are rounded to the nearest integer such that the total number of features in model is still
7

---

Published as a conference paper at ICLR 2024
DNoking
abiating
ablating & a
3.50
0.50
3.00
0.52
4.7
0.81
0.76
0.05
600
200
930
ablatiing
ablating Og
ablating fyy
ablating fyy Rg
0.00
0.75
0.76
0.70
0.66
10
20 00
85
40
10 0.88
10
22
80
0.5
10
20
50
Figure 4: Numerical simulations using the analytical forms of accuracy and agreement. Each plot in the left three columns ablates a single parameter in the framework. The right column shows the effect of coupling two parameters, namely (z) and (M,, na). Changing 4, and n, alone deviates from GDE, but they are coupled with dominant features, we can recover GDE (approximately).
Further, we discuss irreducible error and other properties of this framework in Appendix and discuss potential connection between feature learning and data scaling (Hestness et al., 2017).
5.2 NUMERICAL SIMULATION
We now study the properties of the analytical forms of the expected agreement and the expected accuracy. Instead of bounding their difference, we will use numerical simulation to characterize their properties and differences. The model has 6 free parameters, namely, Pa; c, Ld, fr. Rdd ny We first pick set of initial values and then vary each value to study the behavior of the model. Unless specified otherwise, the initial values used for all simulations are pa 0.7, 20, tu 20, t, 180, n2 5, n, 10. Further, we pick ((k, e) 0.91 > 0}. This reflects the intuition that two models share any features, then with high probability they would agree with each other. This reasonable if we assume all the models in the same hypothesis distribution are naturally more inclined to agree with each other (0.4 ..4 and Figure 3c show that for a 10-class classification the shared error least 35%). We study the properties and effects of Ç in Appendix F.7.
In the left 3 columns of Figure 4, we vary each parameter of the framework over a wide range of values. We observe that for both Pa and the agreement closely tracks each other for large portion of the parameter values. This suggests that the difference between generalization error and agreement is robust to how much of the data has dominant features and the size of the model. Further, we observe both accuracy and agreement saturate the model capacity, c, increases. This is equivalent to increasing the model capacity with an infinite amount of training data. This consistent with prior works model scaling (TTn and Le, 2019) which suggests that model size may be related to how many features the model can learn. On the other hand, the behaviors of agreement and accuracy appear to be more sensitive to the other parameters that describe the relationship between total numbers of existing features and how often these features appear in a single data point, in particular f, and n,, quantities that govern the distribution of rare data. Intuitively. the rare features and data represent the part of data distribution that appears in the tail of the data distribution and require memorization to learn (Feldman, 2020).
The observations about n, and t, suggest that GDE requires some distributional assumptions the features and data in our framework (also in Jiang et al. (2022)). One possible hypothesis is that the relationship between 4, and la and the relationship between ni, and na follow the Pareto principle (Pareto, 1964) (given the long-tailed behavior in 0.1). To verify the effect of this hypothesis, we vary n, and t, while keeping n. and ny, proportional to them, that is, We choose between these quantities is constant, agreement once again tracks the accuracy closely, indicating that the relationship between dominant and rare data is central the origin of GDE in this model.
[ann] and tu ]. 0.2 and show the results in the right column of Figure 4. Notice that the ratio
8

---

Published as a conference paper at ICLR 2024
Table 1: Average accuracy and agreement on datasets with different interventions. K represents different number of superclasses and K 10 is the original CIFARIO. G2 presents the data in the (i 1)-1)am x thth to x 20th percentile of blue intensity. The difference between accuracy and agreement is approximately the same for different K's (thus GDE holds) but not for different G,'s.
K 2
K = 3
K = 5
K = 10
G1
G2
G3
G4
G3
Accuracy
0.80
0.84
0.78
0.81
0.62
0.60
0.59
0.66
0.70
Agrocment
0.85
0.88
0.83
0.85
0.70
0.67
0.69
0.71
0.75
Difference
0.05
0.04
0.04
0.05
0.08
0.07
0.10
0.05
0.05
6
FROM DESCRIPTION TO PREDICTION
The proposed theoretical model makes a series of simplifications. We now demonstrate its predic- tive power of what actually happens in deep learning under specific interventions the following experiments on GDE are conducted after we derived the theoretical framework and to the best of our knowledge have never been done in prior works. In other words, our model has not been specifically adjusted to account for the results of these experiments, but rather the experiments are designed based on the framework. Results for both experiments are shown in Table (with uncertainty in Table 2) and the experimental details are in Appendix F.8 and F.9.
The first experiment considers merging classes. We observed in Section 5.2 that for GDE to hold approximately, the features distribution needs specific properties, namely, fy X ta and n, x nd. If features do not interfere with each other significantly, our framework predicts that merging classes into superclasses should not change the ratios and thus would not break the GDE. We merge the classes of CIFAR 10 into different superclasses and run the same learning algorithms as Section 4 on the new data (6 random seeds). The aggremett difference does not change significantly across different partitions as predicted, even though the accuracy and agreement are different.
This result suggests that breaking the GDE requires intervening on the covariate distribution in order to change Lz, Thus, our second set of experiments considers re-partitioning data. In particular, we sort CIFAR 10 images by the proportion of blue in their total color intensity and partition them into 5 equally sized groups with increasing blue intensity. We observed that the differences of different data partitions are drastically different, corroborating the prediction made by the theoretical framework. Furthermore, we see that group has the largest difference between accuracy and agreement which according to our theoretical framework suggests that the total number of rare features is larger. Through visual inspection (Figure 17), we can see that in group I, the examples seem more visually complex and diverse, which could lead to a larger number of rare features (i.e., larger t,)). It is worth noting that, unlike the setting of Kirsch and Gal (2022), each group is still i.i.d. Therefore, the violation of GDE immediately implies that the ensemble is not calibrated on the data partition (Theorem 4.2 of Jiang al. (2022)). We believe this is the first direct, non-adversarial construction of natural datasets where a deep ensemble is not well-calibrated in-distribution from dataset which the deep ensemble is usually well-calibrated.
CONCLUSION
We investigate distributions of features in data and how neural networks perform feature learning. Based on the empirical observations, we propose new framework for understanding feature learning. We show that the proposed framework is more reflective of reality and can explain other phenomena in deep learning, notably GDE, without making any assumption about calibration. We believe this work provides new insight into our understanding of feature learning and data distribution in deep learning. The proposed framework could be useful for studying other phenomena related to agreement and ensembles such as calibration (Jiang et al., 2022), phenomena related to distribution shift such as aceuracy-on-then -in (Miller et al., 2021) and agreement-on -ot-e---ieeeeeeeeeee (Baek et al., 2022), and transfer learning. We discuss our framework's 's limitations and future directions in Appendix D.3. The new empirical tools we introduced can be valuable for other empirical investigations beyond the scope of this work. While recognizing that our framework, like all models, has its simplifications, we hope the intriguing results will catalyze more future works to study feature learning in this direction.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We would like to thank Vaishnavh Nagarajan, Samuel Sokota, Elan Rosenfeld, Saurabh Garg, Jeremy Cohen, and Zixin Wen for the helpful discussion. We also thank Victor Akinwande, Zhili Feng, and Josh Williams for their feedhack on an early draft of this work. Yiding and Christina were partially supported by funding from the Bosch Center for Artificial Intelligence. Yiding is also supported by the Google PhD fellowship.
REFERENCES
Z. Allen-Zhu and Y. Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:20/2.098 1, 2020.
Z. Allen-Zhu and Y. Li. Feature purification: How adversarial training performs robust deep learning. In 2021 IEEE 62nd Anmual Symposium on Foundations of Computer Science (FOCS), pages 977-988. IEEE, 2022.
J. Ba, M. A. Erdogdu, T. Suzuki, Z. Wang, D. Wu, and G. Yang. nigh-dimensional asymptotics of fea- ture learning: How one gradient step improves the representation. arXiv preprint :Xiv:2205.0144, 2022.
C. Baek, Y. Jiang, A. Raghunathan, and Z. Kolter. Agreement-on-the-linee Predicting the perfor- mance of neural networks under distribution shift. Advances in Neural Information Processing Systems, 35:19244-19289, 2022.
D. Bau, B. Zhou, A. Khosla, Oliva, and Torralba. Network dissection: Quantifying inter- pretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541-6544 2017.
Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8): 2013.
L. Breiman. Bagging predictors. Machine learning. 24(2):123-144 140, 1996.
T. Brown, B. Mann, N. Ryder, M. Subbiah, D. Kaplan, Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:187 177111779991 2020.
N. Carlini, U. Erlingsson, and N. Papernot. Prototypical examples in deep learning: Metrics, characteristics, and utility. 2019.
T. Chen, Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
A. Damian, J. Lee, and M. Soltanolkotabi. Neural networks can learn representations with gradient descent. Conference on Learning Theory, pages 5413-5452, PMLR, 2022.
Y. Dandi, F. Krzakala, B. Loureiro, L. Pesce, and L. Stephan. How two-layer neural networks learn, one (giant) step at a time. In NeurlPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023.
N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodde R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, and C. Olah. Toy models of superposition. Transformer Circuits Thread, 2022. https://trnsformer sircuits.pub2202220 model//nnex.hhml
V. Feldman. Does learning require memorization? short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 954-959, 2020.
S. Fort, H. Hu, and B. Lakshminarayanan. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.022
2019.
Y. Gal and Z. Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning- In international conference machine learning, pages 1050-1059. PMLR, 2016.
10

---

Published as a conference paper at ICLR 2024
M. R. Garey and D. S. Johnson. Computers and intractabilityy, volume 174. freeman San Francisco, 1979.
E. Härkönen, A. Hertzmann, J. Lehtinen, and S. Paris. Ganspace: Discovering interpretable gan controls. Advances in Neural Information Processing Systems, 33:9841-9853 2020.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016a.
K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks, 2016b. URL http:/ //t/xxxivvv//////////v/////////a org/ / abs /tt.//1633.0000 cite arxiv:1603.05027oommee. ECCV 2016 camera-ready.
J. Hestness, S. Narang, N. Ardalani. G. Diamos, H. Jun, H. Kianinejad, M. Patwary, M. Ali, Y. Yang. and Y. Zhou. Deep learning scaling is predictable, empirically. arXiv preprint 17X1::7712.00000 2017.
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. ArXiv, 0as/1704.0486// 2017.
J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu. Squeezzeanndeexitation networks. IEEE Transactions on Pattern Analysis and Machine Intelligence,
2020.
G. Huang, 7. Liu, and K. Q. Weinberger. Densely connected convolutional networks. 2017 IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2261-2269, 2017.
F. N. landola, M. W. Moskewicz, K. Ashraf, S. Han, W. J. Dally, and K. Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and ; mlm model size. ArXiv, abs/ abs/1602.073300, 2016.
A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Y. Jiang, V. Nagarajan, C. Baek, J. Z. Kolter. Assessing generalization of SGD via disagreement. In International Conference Learning Representations, 2022. URL https: / net / forum? id 1YGGCEAQhx..
Z. Jiang, C. Zhang, K. Talwar, and M. C. Mozer. Characterizing structural regularities of labeled data in overparameterized models. arXiv preprint (rXXix:0002.03206 2020.
S. Karp, E. Winston, Y. Li, and A. Singh. Local signal adaptivity: Provable feature learning in neural networks beyond kernels. Advances in Neural Information Processing Systems, 34, 2021.
A. Kirsch and Y. Gal. A note on" assessing generalization of sgd via disagreement". arXiv preprint 0tt1:22022.18551. 2022.
S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519-3529, PMLR, 2019.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, 2017.
Y. Li, J. Yosinski, Clune, H. Lipson, J. E. Hopcroft, et al. Convergent learning: Do different neural networks learn the same representations? In FE@ NIP,, pages 196-212, 2015.
Y. Li, C. Wei, and T. Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. Advances in Neural Information Processing Systems, 32, 2019.
C. Liu, B. Zoph, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. L. Yuille, J. Huang, and K. P. Murphy. Progressive neural architecture search. In ECCV, 2018.

---

Published as a conference paper at ICLR 2024
J. P. Miller, R. Taori, A. Raghunathan, S. Sagawa, P. W. Koh, V. Shankar, P. Liang, Y. Carmon, and L. Schmidt. Accuracy on the line: on the strong correlation between cuf-of-distribution and in-distribution generalization. In International Conference on Machine Learning, pages 7721-7735. PMLR. 2021.
B. Moniri, D. Lee, H. Hassani, and E. Dobriban. A theory of non-linear feature learning with one gradient step in two-layer neural networks. arXiv preprint arXiv:2310.07891, 2023.
A. Morcos, M. Raghu, and Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in Neural Information Processing Systems, 31, 2018.
A. H. Murphy and E. S. Epstein. Verification of probabilistic predictions: A brief review. Journal of Applied Meteorology and Climatoloyy, 6(5):748-753) 1967.
P. Nakkiran and Y. Bansal. Distributional generalization: A new kind of generalization. arXiv preprint arXiv:2009.0800 2020.
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
A. Nguyen, A. Dosovitskiy, J. Yosinski, T. Brox, and J. Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in neural information processing systems, 29, 2016.
E. Nichani, A. Damian, and J. D. Lee. Provable guarantees for nonlinear feature learning in three-layer neural networks. arXiv preprint arXiv:2305.0065 066966 2023.
C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization. Distill, 2017. doi: 10.23915/distill. http:://iisilllppbb2////ttttrere visualiztin.......inn....
C. Olah, A. Satyanarayan, I. Johnson, S. Carter, Schubert, K. Ye, and A. Mordvint- sev. The building blocks of interpretability. Distill, 2018. doi: 1/23911/distill.000000 http://ddiiiil..ubbb// 8/building-blocks.
V. Papyan, X. Han, and D. L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):4662-24466 2020.
V. Pareto. Cours d' L'économii politique, volume 1. Librairie Droz, 1964.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
M. Raghu, J. Gilmer, J. Yosinski, and Sohl-Dickstein. Sveca: Singular vector canonical correlation
analysis for deep learning dynamics and interpretability. Advances neural information processing systems, 30, 2017.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: /409. 2014.
M. Tan and Q Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on muchine learning, pages 6105-6114 14. PMLR, 2019.
Z. Wang, A. Engel, A. Sarwate, I. Dumitriu, and T. Chiang. Spectral evolution and invariance in linear-width neural networks. arXiv preprint wwXiv:22/1.065006 2022.
M. Welling and W. Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 681 681-688. Citeseer, 2011.
Z. Wen and Y. Li. Toward understanding the feature learning process of self-supervised contrastive learning. International Conference on Machine Learning, pages 11112-11 PMLR, 2021.
S. Xie, R. B. Girshick. P. Dollár, Tu, and K. He. Aggregated residual transformations for deep neural networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5987-5995, 2017.
12

---

Published as a conference paper at ICLR 2024
J. Xu, Y. Pan, X. Pan, S. H. Hoi, Z. Yi, and Z. Xu. Regnet: Self-regulated network for image classification, IEEE transactions on neural networks and learning systems, PP, 2022.
G. Yang and E. J. Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In International Conference on Machine Learning, pages 1727-11737. PMLR, 2021.
M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833. Springer, 2014.
X. Zhang, X. Zhou, M. Lin, and Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6848-6856, 2018
13

---

Published as a conference paper at ICLR 2024
A LIST OF NOTATIONS
A.1 CONCEPTUAL FRAMEWORK
A.1.1 MAIN PARAMETERS OF THE TOY MODEL
Pai the proportion of all data that are dominant.
py: the proportion of all data that rare. This parameter is equal to 1 Pd-
c: the total model capacity. represents how many features single model can learn.
La the total number of dominant features available the data for one class.
t the total number of rare features available in the data for one class.
nd: the total number of dominant features single dominant data point has. na td.
ny: the total number of rare features single rare data point has. n, S tr-
A.1.2 OTHER NOTATIONS OF THE TOY MODEL
(-): function that returns the set of features learned by model ff or contained in data point æ. The intersection of (-( of different models and data determines how the model makes mistakes in our framework.
err(ʃ y): the error function for model f on the data (21,z).
G:: The total capacity the model has for rare features. c, = $.p.c.
ca: The total capacity the model has for dominant features. G. = Pac.
Ç: the agreement function that dictates the probability with which two models that do not share features will agree.
A.2 INTERACTION TENSOR
+ Nxd, Matrix that contains the representation of all data points for single model.
dd The principal components of Φ.
$ Pryj E Nxk; Matrix that contains the projection of the representation of all data points.
(x) I: the coefficient the representation of in the 7th model projected the ah principal component.
Hi,a E R: the mean of U; (x) over the data distribution.
of E R: the variance of (;,a(z) over the data distribution.
P(ss).(a)b) E R: the Pearson correlation between (xx( and (2).
Ki E 1|=1,1]A*k; the feature correlation matrix of two models i and j.
A [-1, [ x the collection of all pairwise feature correlation matrices for all M models. other words, contains the K., for all pair of models.
B IRREDUCIBLE ERROR AND DATA SCALING
In this section, discuss the sources of irreducible error in our framework. Concretely, there two sources of irreducible error:
1. Inductive bias mismatch: these are errors that arise from the fact the models fundamentally cannor learn some of the features present in the data via conventional training, e.g., stochastic gradient descent.
2. Finite sample error: these are the errors that arise from insufficient sample size, where the models do not observe all the features in the data.
14

---

Published as a conference paper at ICLR 2024
In both cases, the errors result from the models being unable to learn all the 2(t + t,)) features in the support
will refer to the percentage of all features that are present the training data as coverage and use B to denote it. When irreducible enror occurs, in the best case, the best possible model can only learn up to Eetd dominant features and B,t rare features for each class. Further, while we have previously assumed that there are always more features than the model capacity will make the mild but new assumption: if the model's capacity is larger than the coverage, the model will sample noise for the remaining capacity. Concretely, the model may be memorizing noise patterns in the data that do not help generalization similar to (Allen-Zhh and Li, 2020), We can characterize the expected accuracy when the irreducible error occurs (Proof in Appendix C.3).
Lemma B.I. Under the proposed framework, with coverage of Ga and Be, the expected accuracy is upper-bounded by:
(11 ((1-Ba)es) 1d Pr-
(1--9))
1
Acc S Pd
(5)
2
br
Lemma B.I provides an upper bound on the expected accuracy under this framework when the models cannot learn all the features. To test the validity of this hypothesis, using training sets of different sizes. Specifically, we use training set size 5% increment from 5% to 100% on CIFAR 10 and ResNet18. 88. In Figure 5a, we show the the upperbound in Equation 5 as function of coverage 3 (same value for both B, and Ba). In Figure 5b, show the test accuracy as the function of training set size.
simulate different coverage by
coverage expected acc
data fraction vs test acc
1.00
0.8
0.95
0.90
0.7
0.85
0.6
0.80
0.5
0.70
0.85
0.3
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.0
0.2
0.4
0.6
0.8
1.0
coverage
training
(a)
(b)
Figure 5: Plots of the predicted accuracy as function of coverage (left) and real model accuracy as function of the percent of training data (right).
Note that the lemma describes the average-case test error rather than the worst-case test error that classical bounds based on uniform convergence describe. Figure 5 shows that varying coverage can approximate the behavior of scaling dataset size (Hestness et al., 2017). Nonetheless, we see that some discrepancies between the two plots remain. Most notable is the fact that the test accuracy seems to increase at faster rate than the accuracy described by the framework when the dataset is small. This difference exists likely because the relationship between training dataset size and coverage is not linear. In particular, coverage increases faster when the dataset size is small but saturates after the dataset size becomes large. One explanation for this phenomenon is that the models learn features differently in the presence of different dataset sizes. Our framework currently does account for this effect but it is promising direction for future works.
C
FULL PROOF
In this section, we provide the full proof of the theoretical results. For convenience, we repeat the claims here.
15

---

Published as a conference paper at ICLR 2024
C.I EXPECTED ACCURACY
Proposition C.1. Under the proposed model, the expected accuracy over the model distribution and data distribution is:
(1d-
Acc pd
1
Rd
Pr
1 2
2
km
Proof. We are interested in computing the expected accuracy over the entire data distribution y and the entire hypothesis distribution A. It is simpler to think about the average error instead of accuracy (the prediction is random when the model and data do not share any features so the error is ))
E A & [e,r(,,,,))]]
(6)
(1()) - 21
= ESNEA
E E(x
(7)
PP (V(( (())) =
(8)
P sampled and sampled do not share any features
(9)
To avoid notational clutter, we will use and x to denote sampled f and sampled x. The probability of interest is thus:
I f and x do not share any features ] = 1 - f and x share at least features dominant, and
(10)
We first partition the event space into two parts: dominant, we want to compute IP f and x do not share any features x is dominant J. Since all f's have equal probability of being sampled (as they contain the same numbers of problematically indistinguishable features), the probability is equivalent to:
rare. Suppose that is
P f and x do not share any features is dominant
(11)
P x does not have fixed set of features x is dominant
(12)
(fa-Cd)
na
(13)
km
This the configuration of x that does not contain the ca features f. Analogously, we can compute:
P f and do not share any features is rare
74
Given the assumption about how models make mistakes, the expected for both parts of the event space:
Acc.1
[ and x do not share any features
x is dominant
+ P and x share least features
x is dominant
(14)
1 t4
11 (aa
1
(15)
2
2
12
nd
Analogously we repeat the computation for rare data,
Acc, = 2
(16)
We now compute the expected accuracy the entire event space, Acc = PdAccd Acc,
(17)
(t4 (ta-cd)
= Pa
1
+ Pr
1
(18)
2
2
m
17
16

---

Published as a conference paper at ICLR 2024
C.2 EXPECTED AGREEMENT
Proposition C.2. Under the proposed model, let (*) = when n < 0, < 0 or < and further
define:
2
(r
91 Pd
1
Ad
ti
+ Pr
1
(cd)
ni
(2(k) = Pd
od
ca-a
(15-
(ca)
a+tbkk
Cd
(r
(1a-cd)
(cr)
+ Pr
c--b
n,
b+b==
Cr
43 Q1 (k)),
the expected agreement between an i.i.d pair of model (f.9) drawn for the model distribution over the data distribution is:
Agr +
C(k, ck)
Proof. We are interested computing the expected disagreement of (f.g) ~ FA x FA over the data
distribution 2. Based on the features in and g. We partition the event space into subsets:
A: f and g both share features with
B: f and do not share any features with x but share features with each other.
{1ON + N (v())
( ( ( o)
C: The rest of the event space. In these events, we have either:
and do not share any features with each other. only one of f and g share features with x.
Case A. Since f and g are independent and identically distributed, it suffices to compute the probability of one of them not sharing any features with We further partition the event space into two part conditioned on whether the data point is dominant or rare (this is possible because x is independent from f and g). Following the same logic as Equation 11:
P P( n (x) # is dominant]
(19)
=1 P (xx) = Ø x dominant
(20)
(21)
Analogously,
(n Rr
P(P()) 1
rare
(22)
(1r)
17

---

Published as a conference paper at ICLR 2024
By the independence of f and g:
PP| A x dominant]
(23)
= PP (f) n (x) +
x is dominant J
(24)
(d-ca)
Thtt
(25)
Ld
and similarly,
cr nr
PPA x is rare |
=
1
(26)
Putting everything together:
(r-cr)
Gr- 2
Q1 = P[A] Pd
1--
mr
+ Pr
1
(27)
Case B. Again, partition the event space based dominant and rare data. Then further partition the even space based on h; the number of features that f and share with each other. First, we compute the probability that both f and g do not share any features with By independence and equation 11:
P(P()) (x) Ø and ((() n (x) Ø x is dominant
(28)
PP((()) Ø x is dominant !?
(29)
(
(30)
Conditioned on that æ is dominant and that ff and g do not share any features with æ, we now compute the probability where f and share exactly & features. By symmetry. this probability is equal to the probability of sampling that shares exactly ic features with a fixed f. Since f and g cannot share any feature with x, the total number of dominant features available is tu nd. This event space can be further partitioned into disjoint events where g shares exactly a dominant features and b rare features with f for (a, b) E {(0, k), (1, 1- l), 1)...., (k- 1, 1), (k: 0) Since always samples ca dominant features and e, rare features, the processes independent from each other and respectively follow bypergeometric distributions (i.e., marble picking problem):
IP ([)((() n (g)) = k (f) n ((x) Ø and ((g) n (x) Ø and x is dominant
(2)) (a--n--a))
(7) (2:3)
Σ a+bbmk
(w=-
The first term in the summation is the density of the hypergeometric distribution for sampling allowed dominant features, and the second term is the hypergcometric distribution for sampling b allowed rare features.
The same reasoning process can be applied to when x is rare by modifying the available number of rare features 4, nt and keep the available number of dominant features as bx:
(h--cr)²
P(P(f) n (x) = and (gg) n (x) = Ø x rare
(31)
(+r
P[ V [g) = l (f( (x) Ø and (g) n (x) 2 and x is rare
(32)
(5x) (*r
c.-
(33)
a+bb=k
er
18

---

Published as a conference paper at ICLR 2024
Putting everything together, arrive at the probability:
22 (k) = (() n (g)) = k and (f())
n (x)) 28 and V(g) 2] (34)
(a- 2
Cdt
(1-R8-
Pd
(35)
(11)²
nd)
La+bbk
(dd) (a-caa
(Gr)
+ Pr
ca-b
(36)
(--
a
Note that there may be cases where the combination is undefined (e.g., la nd cd < 0 or
ta - nd Cd Ca - a). These cases means that the configurations are impossible to exist, so their corresponding probabilities are O. We will define (") 0 when n < 0, r < 0 or n < to handle
these cases. The total probability of B is equal to the sum of (kk) from k 1 to e since that is equivalent of the event (f) V (g) O:
P[B]
(k)
(37)
k=1
Case C. This event is the complement of A B so:
Ø3 P[C] = P[A] - P[B] 21
(k))..
(38)
R=1
In A, we know the models agree with probability 1. In C, either both models will make random guess or one model will make a random guess and other will classify x correctly. In both cases, they will agree with probability 2. In B, we assumed that the probability agreement is modulated by the agreement function (Section 5). Combining these agreement conditions with the probability of A, B, C gives:
Agr = 1 P[A] PIC +
Σ (2()))(k)))
(39)
1
Q1 +
2
+
Σ 42 (kk)))(k,c).
(40)
Replacing 4s with 1 - 01 - Ei == (2(k) and simplify yields the final results.
C.3 COVERAGE LEMMA
Lemma C.3. Under the proposed framework, with coverage of Ba and Br, the expected accuracy is upper-bounded by:
1 (1
(1-Auitt)) (laaaata))
(1-8,Ma)
Acc S Pd
na
+ Pr
(41)
2
2
Proof. Lets call the set of all features I' and set of features available for the models to learn I. We can naturally partition them based on dominant and rare features - Ta is the set of all dominant features and r, is the set all rare features. By the coverage assumption [f,| = 8 [T and [ = Ta|Ta]-
Notice that having different numbers of features available to the models and data means that the distributions of model sharing features with conditioned on the data is no longer the identical for different data. The conditional probability changes depending on how many features of the data point is not in F. On the other hand, conditional probability of data point sharing features with fixed model is the same for all models, because N C r No matter what features are in (J), the
19

---

Published as a conference paper at ICLR 2024
probability that sampled data point does not share any dominant features with is ((um))t) /(.)² Recall that ca and c, represent how many features the model can learn which is upperbounded by Bata and Byby. Since (") is monotonically increasing in ni:
(1-
(1=Ba)ii) -
ld Cdl nd
Bata 2 cd - La Bald cd
cta- -
(42)
the same can be derived for rare data. Substituting in the expression for accuracy from Equation 3,
1
1
Acc Pd
1
Ad
+ pr
1
Ry
(43)
2
2
np
(1-AA)(d)
(1-2-)47
1
S Pd
1
nd
+ Pr
1
(44)
2
md
D FURTHER DISCUSSIONS OF THE THEORETICAL MODEL
D.1 COMPARISON TO PRIOR AND CONCURRENT WORKS
An important difference between this model and the multi-view model from Allen-Zhu and Li (2020) is that our model does not treat all features as having the same learning difficulty (i.e., probability of being learned). Indeed, the experiments in Section show that features demonstrate a wide range of behaviors in terms of how often they occur in the data and how they interact with the models. Another notable difference is that in Allen-Zhu and Li (2020), the multi-view portion of the dataset contains all the features. In reality, the "easy" part of the data that large portion of the models classifies correctly actually contains much fewer features. These observations suggest that having different types of features may be a more accurate description of nature. Nonetheless, we do not describe the exact mechanism of how feature learning actually happens under our model since we not assuming any particular hypothesis class. Consequently. we do not use the same definition as Allen-Zhu and Li (2020) they adopt a very simplified model of features (i.e., orthogonal vectors in the input space). The spirit of our model of feature learning is close to that of Allen-Zie -Zhu and Li (2020) and we believe similar iterative analysis can be applied to our model.
There is also a recent line of theoretical works that depart from the random feature model and try to incorporate feature learning into the analysis (Damian et al., 2022; Wang et al., 2022; Moniri al., 2023; Nichani et al., 2023; Dandi et al., 2023). These works generally assume the data is an isotropic Gaussian and the target function is a 2-layer neural network. The feature is usually a linear function of the input and sometimes special training algorithms needed (c.g., layer- training). A notable development Nichani et al. (2023) which analyzed 3-layer neural network which learns non-linear features although it still requires layer-wise training. In general, there is still a gap between the assumptions made in these works and real models and, to the best of our knowledge, they do not account for ensemble or the possibility of learning different features. The goal of our work is not to discuss the merits or limitations of these works but study what actually happens in practice on real dataset and real architectures. Due to the complex nature of deep learning, we believe this more scientific approach is good complement to the existing theoretical works.
is also natural to question whether the simplification where a single feature is sufficient for deter- mining the class is sensible. We believe that this simplification is realistic for a binary classification problem and that using more features in determining the true class may make the model more expres- sive but should not fundamentally alter the behavior of the system. Further, the true data distributions are evidently more complex- features and data can lie on a continuous spectrum between "dominant" and "rare" (Figure 2b and 2c). These changes can be incorporated into the framework by modifying the distribution of features but doing so can increase the complexity of the analysis and require tail-bounds to characterize the system's ss behavior.
dominant data can contain rare features, and, vice versa. In fact, both
'eere assume the capacity is smaller than the number of available features. If the capacity is larger, then model will leamn all available features and the bound is tight.
20

---

Published as a conference paper at ICLR 2024
D.2 SOURCES OF RANDOMNESS
Another assumption we made is that when the model f does not share any feature with a data point æ, the model will make random guess. At first look, this seems like strong assumption that requires the model to make a perfectly random guess. However, recall that we are computing the expectation over the model distribution and the data distribution rather than a single fixed data point. For single model f. its prediction effectively random if its average prediction over all the distribution of data that do not share features with ʃ is at the chance:
F =y y (f) (x) Ø] E [1((()) y 1 (
Ø =1 (45)
This means that f can be completely deterministic long as its accuracy over all the data that doesn't share feature with random chance. This in fact the only sensible outcome we assume that features are indeed what the models use to make predictions. In this case, the source of randomness comes from the data, (x, 2.
We now analyze the case where we hold single data point (r, y) fixed and generate the source of randomness from the training algorithm f ~ FA (once again, the individual model can be completely deterministic). When the data point is one with which f does not share features, we cannot expect the models to make independent predictions since the models have similar inductive bias and can make predictions correlated manner depending on x (e.g., noise in z):
P [f(() (f) n v (x) = Ø = Sr [1((()) y} ((f) Ø +
4
(46)
Consequently, the agreement between a pair of models will not be random over the data distribution, and this is exactly what the agreement function tries model.
P [f()) = g(z) (T()) n (x) = Ø and (g) n (z) 2) o]= A(FA.=)
(47)
In the most general case, Ç is function of the hypothesis distribution and a data point æ, but the ones we used in the main text assume that Ç is function of the model's features, since what type of data x is irrelevant if neither models have the features to predict
also drop that dependency.
D.3 LIMITATIONS
While our theoretical framework is able to explain some previously poorly understood phenomena, several limitations still exist. We have discussed various limitations of our framework throughout the paper but for case reading, we will discuss all of them here.
Optimization analysis One of the important limitations of this work is that it only describes what kind of features are present in the data and end up being learned by different models, but does not offer any insight on how these features are learned through gradient descent mechanistically. Modeling the optimization procedure is evidently challenging task (although this is true for deep learning in general). One promising avenue is via techniques similar to Allen-Zhu and Li (2020).
Binary dichotomy of features The current conceptual model assumes there are only two types of features, which facilitate the analysis. Evidently, Figure 2b shows that the features in fact live on spectrum of frequency of appearance. Future work could resolve this by assuming long-tailed distributions (e.g., Zipf's distribution) instead of binary distributions to make the model closer to reality.
Binary classification discuss steps towards extending it to multi-class in Appendix D.4.
The current conceptual model only analyzes a binary classification. We
D.4 EXTENSION TO MULTI-CLASS
In order to extend this framework to multi-class, we would first have to decide on how the model makes predictions based on the features it has learned and the features present in the data. In this setting, perfect prediction based on a single feature may no longer be enough since different classes can share features. Instead, one may need to introduce a new function for the probability of correct classification based the number of shared features between the model and the data point or the
21

---

Published as a conference paper at ICLR 2024
probability of making mistake based the features. This also means that cannot longer assume the model will make a random guess since there are more than one possible wrong class and how the model makes prediction will depend on the features they share with these wrong classes. Mathematically, this means that Ç is no longer independent of the data point z. The desired quantities are still computable through combinatorics but the added complexity could make the derivation much more complicated and an analytical expression may may be attainable, though the problem may be amenable through tail-bounds.
E EXPERIMENTAL DETAILS
In our experiments, we use two collections of A: = 20 ResNet18 (He et al., 2016a) trained on the CIFAR-10 dataset (Krizhevsky al., 2009) following the experimental set up of Jiang et al. (2022). The first collection of models is trained on random 10000 subsets of the whole training set (1 (10k), and the second collection of models is trained on random 45000 subsets of the whole training set (45k). On average, the 1 10k models achieve 67.9% test accuracy and the 45k models achieve 84.5% test accuracy. In addition, 45000 subsets of the whole training set (SVHN). The training details are outlined in Appendix and E.2. We compute A and 52 the test set using the output of the penultimate layer (i.e., ti the final linear layer). For clustering features, we choose & 50 for the number of principal components to use, Yeor to be the 90th percentile of A, and Yitta to be the 90th percentile of all entries in T,T0 i= 1, ................................ M. After clustering, obtain T 680 feature clusters for 45k. We primarily show 45k here and leave the 1.k and SVHN results to Appendix F, which are similar to the results qualitatively.
repeat the same process for SVHN dataset Netzer al. (2011) using random
E.1 TRAINING DETAILS
For the ResNet18 experiments, we follow the same procedures as Jiang et al. (2022) which uses the same architecture of ResNet18 as He et al. (2016a). We train the 20 models with:
initial learning rate: 0.1
weight decay: 0.0001
minibatch size: 100
data augmentation: No
The models 45k samples 45000 data points from the training without replacement. Likewise,
the models 10k samples 10000 data points from the training set without replacement.
E.2 CLUSTERING ALGORITHM
Algorithm iterates over all entries A and assigns each feature to a cluster if its correlation with the members of the cluster exceeds Toorr otherwise, the algorithm creates new cluster for that feature. One notable property of the greedy clustering algorithm is that does not generate fixed number of clusters. This is desirable in this case because if feature does not have a high correlation with any other features, we would like to isolate it as a unique cluster rather than grouping together with other features.
For the number of principal components, recommend picking the number where after projecting every activation vector onto the principal components, the linear layer can classify the projected representation with approximately the same accuracy as the representation before projection. In our setting, 50 principal components could retain 100% of the original performance. For Your we found that the qualitative results are not very sensitive to different values. We experiment with different values of Yeor in Appendix F.10 and observed similar results.
E.3 HARDWARE
All experiments in the paper done single Nvidia RTX 2080's and RTX A6000's.
22

---

Published as a conference paper at ICLR 2024
Algorithm I ClusterFeetuures
1: Input: A [M, M, k, k]. Yoor
2: Assignment [M, k] +- new empty matrix
3: Maximum [M, k] < new matrix filled with -1
4: CurrentFeature
5: for to and to m do
6: if Assignment [i, j] is not empty then
7:
Skip the next ]
8: Assignment [i, 11 + CurrentFeature
9:
fo k do
10:
CorrMat + A Ai,,
11:
FeatureRow + CorrMat [j,
q do
13:
if Feaauurreo [q] > Maximum (p, q] and FeatureRow [q] > Ycor then
Assignment qp CurrentFeature
15:
Maximum [p, q] f FeatureRow [q]
16:
CurrentFeature + CurrentFeature
17:
Return Assignment
F ADDITIONAL FIGURES, SIMULATIONS, AND EXPERIMENTS
F.1 INTERACTION TENSOR
I
I
12
12
N
I
I
I
I,N
N
model (M
I
I 2,2
1 2,3
I
2,N
I
3,3
1
J.N
M.N
A.N
feature (T)
1 MM.N
M.1
M.
M.3
data (N)
Figure 6: An illustration of the interaction tensor, Ω. The three axes correspond to model, data, and features. An entry I is if both data n and model m contains feature / and is otherwise.
F.2 EFFECT OF PCA
In Figure 8, we show the correlation matrices, K,ji for different model pairs from 10k and 45k. The first column shows the self-correlatio matrix between the features of the same model. Both matrices are effectively diagonal which indicates that the principal components represent features with no redundant information. This contrasts with Li et al. (2015) where the self colf-correlatio matrices have many off-diagonal entries. Off-diagonal entries for the self-correlation matrix indicate that either there is redundant information or a single feature is distributed across multiple neurons, which is not desirable for studying unique features. Another interesting effect of using PCA projected features is that the features are naturally "aligned" because the principal components are already sorted by the amount of variance they can explain. We can see that the correlation matrices' entries (especially towards the top features) are naturally more concentrated towards the diagonal. Furthermore, models with more data and higher test accuracy (4 5k) have more near diagonal entries. This indicates that the models in 5k have learned nearly the same top features. This observation is consistent with Li et al. (2015); Morcos et al. (2018) 8) which find that better models tend to learn more similar representations.
23

---

Published as a conference paper at ICLR 2024
Set Me--omaaiinn 2000
Cameiian 1150
Sefccmmeen Cth4
Figure 8: Self-correlation (K..) where i j) and correlation (Ki, where i {) j) for random models from 1 10k and 4kk. Both self-correlation matrices contain only diagonal entries, indicating that the features of the same model contain no redundant information. On the other hand, the correlation between models in 4 4k exhibits more structure than the models in 1 10k, with the non-zero entries more concentrated around the diagonal (zoom in for better visuals). This suggests that the features of models learned in 45k contain more similar information compared to models learned 10k.
==2
==3
K 5
X 10
G1
G2
G3
G4
Gs
0.80+0.01
0.84+
0.78+
0.81+
0.62+ 0.02
0.60= 0.01
0.50+
0.66+
0.70=
Ayz
0.85
0.88 0.05
0.83 0.06
0.85 0.07
0.70+ 0.15
0.67 0.15
0.69
0.71+
0.75 0.11
Dirk
0.05
0.04
0.04
0.05
0.08
0.07
0,10
0.05
0.05
Table 2: Average accuracy and agreement on datasets with different interventions. K represents different number of superclasses and K 10 is the original CIFARIO. G, presents the data in the (i - 1) x 20th to x 20th percentile of blue intensity. The difference between accuracy and agreement is approximately the same for different K's (thus GDE holds) but not for different G ss The uncertainty denotes standard deviation.
F.3 EMPIRICAL PROPERTIES OF PCA FEATURES
Features are semantically meaningful As shown Figure 1, our feature definitions are semanti- cally meaningful and can be used for identifying common prototypes and rare images in each class. new experiments on the density of feature in each group? Rare images contain more rare features. This property of the defined features also allows us to find semantically similar images in the dataset. To do so, we first define similarity metric between images:
2 V (2)1 (x2)
s ((x 222
(48)
x + (22)
This function intuitively computes the overlap of features between two images normalized by their total number of features. For any given image æ can compute the similarity of æ and the entire dataset and find the ones with the highest similarities. In Figure 9, we show nearest neighbors of random sample of images. We can see that our metric is able to identify semantically similar neighbors for each image even if the images are not close in pixel space.
Note that for the second row of Figure 9, the first 7 neighbors have 100% feature overlap. The property of our definition of feature may be of independent interest to other applications.
Individual features do not correspond to particular classes It may tempting to think that individual features may correspond to individual classes. In the extreme case, this would reduce to neural collapse (Papyan et al., 2020) (which only happens after the model has been trained for an extremely long time). We find that this is not the case. Instead, individual features do not correspond to any particular classes (Figure 10). To illustrate this point further, we plot the frequency which the top features appear in each class, and observed that the dominant features often appear in many different classes with different frequencies and would be missing from only one two classes (Figure 11). This suggests that individual features can represent multiple "concepts" in the data but combinations of several features are much more interpretable (Figure 9).
This is perhaps not too surprising since in general we cannot expect the models learn features that humans consider to be good features. After all, the appeal for using neural networks is the difficulty of designing hand-engineered features. Future works could investigate these observations further.
24

---

Published as a conference paper at ICLR 2024
2,0000
1,0000
1.0000
1.0000
0.4590
0.600
5- U.4555
5- 0.C500
- U.4.00
3.0.0157
- 0.122
5- D.UINS
6- 0.402
1.0000
5370
5614
1.0000
5- 0.4976
5- 0.4715
5- 0.4645
5- 0,4332
5- 0.4040
5-004473
5 0-253
5- 0.244
5- 0-4237
1.0000
0.5070
0.5063
0.5000
0.4229
0:4819
0,4776
04772
0.722
0.4658
S 1-0000
5= 0.6829
0.542
5 0.605L
5= 0,6000
5 0. 6000
5 0,5009
5 0.5789
E= 0.5788
5 0.5714
1.0000
757
4= 0191
== 0176
K= B 7087
= 7077
==0 7013
= 0-0.8644
+-0 0.678
6 0.375
Figure 9: Nearest neighbors of random images measured by feature overlap. The leftmost image the base image and the row contains nearest neighbors. The similarity score shown above each image. We can see that feature overlap can reliable capture the semantic similarities between different images even if the distance in pixels space is large. This phenomenon particularly obvious in the second row where we can see that several distinct images of frogs have 100% feature overlap.
Features capture prototypical examples. We also observed (Figure 12) that our definition of features can recover the notion of prototypical examples observed in Carlini et al. (2019); Jiang al. (2020). In particular, the images with the least features seem correspond to the prototypical examples (images where the objects are presented in canonical way) whereas the images with the most features seem correspond non-prototypical examples (images where the objects are presented in a rare way). This means that these prototypical examples usually contain much fewer (dominant) features whereas the non non-prototypical examples contain much more rare features. Exploring these connections would be an interesting future direction.
25

---

Published as a conference paper at ICLR 2024
Samp Images for tap teatures
ler1
NANO
Text
NA
NESE
NEIK
MAC
Figure 10: Random data points containing individual features. The top figure shows the sample images for dominant features and the bottom figure shows sample images for rare features. Neither shows obvious patterns, although images with dominant features do seem to be visually less complex.
26

---

Published as a conference paper at ICLR 2024
Class distiontion feature
foot 5997
Fest 11.
5623
Feat 12
5527
1ett 5347
Net14 +-00
Nett 46337
36 + 3003
Prst + 2095
x 2014
2et 2733
Pett x 2570
10
1
Figure 11: Frequency top feature each class. is the feature's 's total number of occurrences.
F.4 CIFAR-10 10 10K SUBSET EXPERIMENTS
For 10k models, we see that the observations are largely consistent with the observations of 45. It is worth noting that in Figure 13e, the shared errors are generally smaller than Figure 3c and the shared errors also exhibit more variance. This may due to the fact that when the models have low performance, their agreement behaves more randomly rather than how the agreement of 45k behaves. We will see in Appendix F.6 that when the collection of models have different architecture, an even strong correlation between number of shared features and amount of shared error is observed.
27

---

Published as a conference paper at ICLR 2024
Visualization of images with the least features (top) and the most features (bottom) for each class of CIFAR 10 under our feature definition (defined in Section 3.1). Each row corresponds to one class of CIFAR 10 (zoom in for better viewing quality)..
Figure 12:
F.5 SVHN EXPERIMENTS
For models trained on SVHN, we observe similar phenomena from the other experimental settings on CIFAR-10. Some notable differences include that there are much fewer low-confidence data points compared CIFAR-I0 10 likely because the performance of ResNet18 is higher SVHN and the models classify most test points correctly. The absolute occurrences different features are higher because SVHN has more test data than CIFAR-10.
28

---

Published as a conference paper at ICLR 2024
(10K)
feature frequency (10K)
160
foature darsilyy by confidencaa (104)
number of
hight curlidanss
-2.0
100
80
2.5
80
6
2009
-3.0
40
20
-3.5
4.0
-02
0.0
0.2
0.4
0.5
0.5
1,0
1/2
2
3
0
confidence
Seature [1x00)
Saaum (1m2)
(a)
(b)
(c)
data modell wi feat (10k)
17.5
shared feature vs, shared enror (10K)
models wit
10.0
7.6
5.0
2.5
0.30
0.0
0.29
of data (163)
shared features
12
(d)
(c)
Figure 13: Same set of plots for 10k models.
conf feature
feature frequency
feature frequency over training
4.25
occumenoos (163)
foatures rum
150
1.5
3.50
1000
3.28
3.00
60
0.0
0.2
0.4
0.6
0.8
1.0
0
3
6
0.0
0.2
0.4
0.6
0.8
1,0
confidence
feature (1e2)
feature (162)
(a)
(b)
(c)
n data vs. models w/ feat
shared feature vs. shared error
log fesilure densily by conficience
17.5
-2.0
High conridercc
lne confilernoa
12.5
15.0
shared error
0.46
0.44
-2.5
10.0
7.5
0.40
5.0
log
25
-4.0
0.0
0.0
0.5
1.0
1.5
2.0
(102)
# of data with feature (1a3)
shared features
(d)
(e)
(f)
Figure 14: Same set of plots for SVHN models.
F.6 SHARED FEATURES AND SHARED ERROR FOR DIFFERENT ARCHITECTURES
We see earlier that when the architectures are the same, the models naturally tend to agree with each other more. For both 45k and 1 10k, even the smallest shared error is much greater than chance. Here we will use a wide range different architectures trained of CIFAR 10 to test the validty of our hypothesis that more shared features lead to more shared error. Figure 15 shows the number of shared features plotted against the shared error between each pair of models. Similar to Figure 3c, the shared error is almost monotonically increasing as function of the number of shared features. If two models share large number of features, they would tend to share high proportion of errors. If two models share a moderate number of features (4 to 11), the distribution of shared error once again appears random. However, unlike the case of same architecture, when architectures share
29

---

Published as a conference paper at ICLR 2024
on architactures
0.5
0.4
0.3
20
features
Figure 15: Shared feature against shared error for models with different architectures.
small number of features (1 3), their shared errors tend to concentrate much smaller values. This observation indicates that while having high number of shared features generally leads to models making similar mistakes, having low number of shared features does not mean two models will have low number of shared error. Rather, different sets of features can still make similar mistakes. The architectures we test include:
PreActResNet18 (He al., 2016b)
DenseNet121 (Huang al., 2017)
PreActResNet34 (He et al., 2016b)
DenseNet169 (Huang et al., 2017)
PreActResNee50 (He et al., 2016b)
ShuffleNetV2 with scale factor (Zhang et al., 2018)
VGGI1 (Simonyan and Zisserman,
2014)
ShuftleNetV2 with scale factor 1.5
(Zhang et al., 2018)
VGG13 (Simonyan and Zisserman, 2014)
ShuffleNetV2 with scale factor 0.5
(Zhang al., 2018)
VGG16 (Simonyan and Zisserman, 2014)
ShuffleNerG2 (Zhang al., 2018)
RegNet X200 (Xu et al., 2022)
SENet18 (Hu et al., 2020)
RegNet X400 (Xu al., 2022)
SqueezeNet (landola al., 2016)
ResNet34 (He al., 2016a)
EfficientNedBO (Tan and Le, 2019)
ResNet50 (He et al., 2016a) 6a)
PNASNetA (Liu et al., 2018)
ResNet101 (He et al., 201
PNSNetA large (Liu et al., 2018)
ResNeX129 (Xie al., 2017)
MobileNet V2 (Howard al., 2017)
All models we use are from testbed created by (Miller et al., 2021).
30

---

Published as a conference paper at ICLR 2024
F.7 DIFFERENT CHOICES OF
In this section, we investigate the effect of agreement function Ç on GDE. Instead of plotting accuracy and agreement separately, we show the difference between accuracy and agreement:
Diff
Acc Arg
The closer the difference is O, the closer the system is satisfying GDE exactly. We test three types of agreement functions, each with an adjustable parameter:
constant: This agreement function assumes that if two models share one or more features, then they have a constant probability 7) E [0.5, 1] of agreeing.
Scoast (k, c, 1) = "
In the main text, we use 77 0.9.
2. proportional: This agreement function assumes that the probability of agreement is directly proportional to how many features two models share relative to the full model capacity. The constant of proportionality is 17 E (0, co) and the probability is clipped to 1.
Grrop (k, 1)) = min (
3. step: This agreement function assumes that there is a threshold 1) E N. If the number of shared features is above 7), then the probability of agreement is 1. Otherwise, the probability of agreement is some constant [0.5, 1].
Saep (k, c, 77, 0) 0 1 1 S n) 1 n)
For these simulation, we use = 0.8 since has a much greater effect on GDE.
We vary the values of 17 for each agreement function and show the results for different values Pa- coupled ny, and coupled ty, tu in Figure 16. Each row corresponds to a different agreement function and from top bottom constant, pettttanttppppppooo proportional and step.
For constant (top row), 7 ranges from 0.5 to 0.95. We see that for pa, R2 x Vl, the range of variation in difference is consistently small when pa is sufficiently large. t, x ta deviates from this behavior where different 77's behave more differently as pa increases. For this scenario, we see that larger are closer to GDE.
For p proportionn annal, ranges from to 2.8. We see that the difference is generally large for all values of the parameters. Suggesting that Sprop may not be good approximation for how models agree in practice.
For step, 1) an integer that ranges from to 9. We see that the differences are more robust to different values of than the other agreement functions. This suggests Sstp could be a good approximation to how models agree in practice. This observation consistent with Figure 3c, Figure 13e, and Figure 15 - when models do not share many features, the shared error (therefore. agreement) is spread out but have similar expected values; when models share large number of features, the probability of agreement increases significantly.
An important observation from these simulations is the importance of ta and 6,. These quantities can be interpreted as proxies for the complexities the entire dataset. t, particular represents the patterns in the data that are rare. The larger t, is, the more diverse or noisy the dataset is. According our framework, this quantity
have large impact the behaviors of accuracy and agreement.
Another important observation is that Pa needs to be sufficiently large for GDE to hold strongly. This is roughly equivalent requiring the feature distribution be tailed, which is true in practice.
31

---

Published as a conference paper at ICLR 2024
ConSE ogrwentt MAOME
00051 sigho 101E Aandon Pv
cons agecnert SHOW F A
C2T00 www.:ooroeoo...oo FueeAe C.
3.50
3.13
8.0
4015
20
070 www.aaaee
8.16
606
600
15p www.sponnrttttco fandion n-4.D.
1.00
3.11
3.15
1.0
6:94
3123
3.50
0.2
810
3.0
401
Figure 16: Difference between the analytical agreement and accuracy for different choices of agreement functions Ç and different values of parameters. From top bottom are constant, proporticnal, and step.
F.8 MERGING CLASSES
For this experiment, we merge different classes of CIFAR 10 to form superclasses. Since the individual images
modified, we expect the majority of features that identify individual classes to also identify the superclasses well (although there may be interferences between features of different classes). Thus, we would expect the ratio between features stay approximately constant.
The specific superclasses are:
2 superclass: (airplane, automobile, bird, cat, deer), (dog, frog, horse, ship, truck]
3 superclass: {airplane, automobile, bird), fcat, deer, dog), (frog, horse, ship. truck)
* 5 superclass: (airplane, automobile), {bird, cat). {deer, dog), {frog, horse), {ship, truck)
Finally, the classes case corresponds to the regular CIFAR classification, The experiments are repeated for 6 random seeds.
F.9 PARTITIONING DATA
For this experiment, partition CIFAR based the intensity of blue pixels. More concretely, let x E [0, 32xx22x3 be an image where the last channel is the RGB value of the pixel. We compute its blue intensity, b, as:
Σ
32
32
b(x)
5==1
1j=1
(49)
E
32
32
Intuitively, this value captures how much the blue channel "weighs" in the whole image. We compute this value for all æ; and compute the CDF, F, of b(æ;) over the training dataset X {æo, L1, #60000). Then we partition the data into groups:
G, = (x E X 0.2i < F(b(=)) < 0.2(i + 1)),
(50)
for E {0, 2, 3,4). For the test data, we partition according to the CDF of the training data, i.e., training and test use the same threshold. In Figure 17, we show random samples of images from each partition based on the blue intensity. We can see that in group O. the examples seem more visually complex and diverse, which could lead to a larger number of rare features (i.e., larger 4,,, the total number of rare features). The experiments repeated for 6 random seeds.
32

---

Published as a conference paper at ICLR 2024
USAA D
SHAR
Figure 17: Visualization based on data partitioning based on blue intensity.
F.10 EMPIRICAL OBSERVATIONS UNDER DIFFERENT HYPERPARAMTERS
The construction of the interaction tensor involves several hyperparameters. In this section, we study how sensitive the empirical observations are the changes in hyperparameters. Specifically, we study the effect of Yoorr which decides whether two principal components of two models are capturing the same feature, and k the number of principal components used for each model. We show the results for Yoor = 0.5 and Your = 0.7 in Figure 18 and Figure 19, and the results for = 30 in Figure 20.
We observe that the most significant impact of Your is on the total number of features because having a higher threshold would mean that more features would be counted as different features if they do not have highly correlated responses to data; this would increase the total number of features. Interestingly, the qualitative observations remain the same which means the conclusion is robust to the choice of these hyperparamters.
Similarly, too many principal components would inflate the number of rare features (features the tail). Empirically, this would affect Observation by inflating the length of the tail (but still long-tailed), but it should not affect Observation 2-4. For 2, the high-confidence data would still have few dominant features. For Observation 3, since these inflated rare features do not usually appear in more than one model and are not useful for classification, this observation is not changed either. For Observation 4, following the previous reasoning, models with similar features would still make similar mistakes
Note that in our experiments, 30 is the minimum number of principal components required to retain the full accuracy on CIFAR although this can change depending on the dataset (note that the plots are almost identical to Figure 2 and Figure 3). Since more principal components only affect the rare features, the effect of moderate overcounting is not significant. For the main experiments, we chose 50 principal components that are sufficient for retaining the full classification accuracy on the test data but not large.
33

---

Published as a conference paper at ICLR 2024
feature frequency (45k)
log feature density by confidence (45K)
con' vs. festure count (45K)
-1.5
ligh cortidence
occrren (1e3)
log prob (basn 10)
confidence
features
95
-2.5
8
-3.0
10
-3.5
23
0.
2.0
2.5
-4.0
0.0
1,0
1.5
2.5
0.0
02
0.4
0.6
0.8
1.0
feeture (1e2)
feature (1e2)
confidence
(a)
(b)
(c)
data vs. n models w/ feat (45k)
shared feature vs. shared error (45k)
mnallls wilh fealure
0.43
20
15
0:40
0.39
0.38
0.37
15 shared Teatures
# of data with a feature (1m3)
(d)
(c)
Figure 18: Ycorr 0.5
feature frequency (45k)
log feature density by confidencce (45k)
conf vs. feature count (45k)
-1.5
high confidence confidence
00curTe000 (1e5)
log proh (booe 10t
features
-2.5
80
8
8
-3.5
2
0.0
0.2
0.4
0.6
featune (1e2)
feeture (1a2)
confidence
(b)
(c)
data vs. n models w/ feal (45k)
shared feature vs. shared error (45k)
20.0
17.5
15.0
0:11
with
0.40
0.39
0.38
5.0
25
0.36
0.0
3
6
5
10
16
18
of data with feature (1x3)
shared fealures
(d)
(e)
Figure 19: Yoor 0.7
F.11 PROGRESSION OF FEATURES AT DIFFERENT DEPTH DEPTH
In the main text, studied the distribution of features at the input of the final linear of the model. As we mentioned earlier, this choice is to avoid superposition (Elhage et al., 2022) where model leverages the non-line loneeiiyy to model multiple features using few features. This phenomena cannot happen at the final layer since only the linear component of the representation can be used make decision. In general, we want to emphasize that features are not always well-defined object. In all but the simplest cases
say what the feature of an input is concretely.
34

---

Published as a conference paper at ICLR 2024
feature frequency (45k)
can' count (45k)
log feature density by confidence (45k)
80
-1.5
ligh contidencos
0000Tre nouureences [1e3]
log prob (baun 10)
confidence
number features
8
-2.0
-2.6
-3.0
-3.5
0.5
200
2.5
0.0
0.5
1.0
1.5
2.0
255
0.0
0.2
0.4
0.8
0.8
1.0
feeture (1e2)
feature (1e2)
confidence
(a)
(b)
(c)
n data vs. n models w/ foat (45k)
shared feature vs. shared emror (45k)
of modele feature
17.5
0.43
15.0
12.5
0.41
0.40
0.39
0.38
25
0.37
0.0
0
5
6
12
14
of data with feature (1n3)
shared features
(d)
(c)
Figure 20: 45k using 30 principal components per model.
In fact, it is not particularly productive to talk about features without talking about the model used to classify the data. One may be interested in resorting human interpretation but the human cognition is in itself classifier with features, and a machine learning model may or may not have the inductive bias to automatically learn the same features. For example, humans may think the face of puppy is the defining feature of a puppy, but a neural network may consider the fur or paws as the most important features.
In deep neural network, we often say that the model learns hierarchical features. These features are combined to form more complicated features (Olah et al., 2017). However, this also means that not all information present in these features is necessary for performing classification. From this perspective, the final layer features may be viewed as the "greatest lower bound" of this feature hierarchy as the model can no longer compress the feature beyond the final layer and whatever features present at this layer is the feature used to predict the classes. As we can see in Figure 10 and Figure 11, these features do not correspond to human perception in an obvious way and are thus not highly interpretable.
That being said, looking at the feature distribution at different depths of the model could reveal interesting observations about the evolution of features within the model at macro level. The ResNet18 architccture we used can be broken down into 4 residual blocks. We refer to the features at the output of each block as blockn where is the index of the layer (c.g.. the results in the main paper would be bloch look) 4). Figure 21, 22 and 23, we present the feature distribution and associated artifacts for bl 1lcc1 block2 and block3. 3. We observe that O.1, O.3 and O.4 generally still hold across different depth. However, the separation in data with differrntt confidence (O.2) tends to happen at deeper layers. In the earlier layers, the features of data at different confidence levels are not distinguishable, but this separation becomes more pronounced deeper layers (subplot (b) and (c)). In other words, the representations for data with high confidence become increasingly similar and compressed the deeper they are into the model. The rare data that are less compressible are encoded with additional dedicated features (rare features) that are not used for the dominant data.
35

---

Published as a conference paper at ICLR 2024
feature frequency (45k)
log feature densily by confidence (15k)
vs. count (45k)
175
high contidencee
oczurrences (163)
confidaroo
number features
125
19 pibb (base
-2.4
100
-5
-2.8
8
8
0.0
0.2
0.4
0.0
0.8
1.0
feature (162)
fealure (1e2)
confidence
(a)
(b)
(c)
data models w/ feat (15K)
shared festure shared (15k)
17.5
of models with feature
15.0
0:43
0:42
enter
0.41
10.0
0:40
7.5
0.39
0.38
25
0.37
0.38
00
of data with (1a3)
shared features
(d)
(e)
Figure 21: Features distribution of 45k depth1.
feature frequency (45k)
feature density by contidence (45k)
cont va. teature count (4bk)
-20
high confidence
0020111118 (163)
bg gr00 (baae 10
low contidence
features
125
-2.4
78
-2.9
8
--3.0
-3.2
feature (1e2)
(1c2)
(a)
(b)
(c)
data vs. models foat (45k)
shared testure vs. shared error (49K)
8 models a isolure
14
0:43
12
0.42
16
0.41
6
0.40
6
0.39
0.38
25
50 75 1000 12.5 15.0 17.5
of data with a foature (1c3)
shared features
(d)
Figure 22: Features distribution of 45k at depth2.
36

---

Published as a conference paper at ICLR 2024
feature Trequency (45k)
feature density by confidence (45k)
conf vs. feature count (15k)
-2.2
high confidencea contidence
S [163)
pioh (bssn 10)
number of features
-2.4
-2.9
8
-3.0
0.0
02
0.4
0.6
0.8
feature (1e2)
feature (1e2)
confidence
(b)
(c)
data vs. models wi feat (45k)
shared festure vs. shared emor (15K)
of models with fealure
0:43
12
10
arror
0.41
0:40
0.39
0.38
0.37
of data with a teature (163)
shared features
(d)
(e)
Figure 23: Features distribution of 45k h3
37


Reviews:
[{"review_id": "CmYGaqyrCP", "reviewer": "Reviewer_AyvM", "summary": "The paper proposes the interaction tensor which is a binary 3-dimensional tensor T describing the presence of certain features in both model and data point, i.e., if $T_{tmn} = 1$ implies that $t_{th}$ feature is present in both $m_{th}$ model and $n_{th}$ data point. The construction of the interaction tensor is based on the correlation analysis between PCA-reduced features of penultimate layers of the collection of models on the given dataset. Authors utilize this interaction tensor to empirically analyze properties of feature learning and propose a feature learning model based on their observations. Using this feature learning model they focus on analysing recently proposed Generalization Disagreement Equality and given which conditions GDE can arise.", "questions": "1. Given that construction of interaction tensor depends on thresholding ($\\gamma_{corr}$ and $\\gamma_{data}$), how important are these hyperparameters? How to properly set them?\n2. Currently theoretical framework analyzes binary classification, does the analysis extend to multi-class classification?\n3. Does the framework allow for introducing distribution shifts and etc?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Understanding how models learn is an important topic in modern deep learning. Authors build the new framework to describe feature learning from the different perspective which allows to describe recently observed phenomenas. I think that this new perspective provides a valuable contribution to the community and can facilitate further developments in this area. In addition, I personally liked the construction of a natural dataset on which deep ensemble is not well-calibrated in-distribution and where GDE fails.", "weaknesses": "Honestly, I don't see obvious weaknesses of the proposed framework and study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aJUc38hbU2", "reviewer": "Reviewer_PxMC", "summary": "This paper focuses on the interaction between data and model through feature to understand deep learning from the feature learning perspective. Based on their observations, they propose a framework to characterize the quality and process of feature learning, with theoretical support. Some empirical results are provided to validate their approach.\n.", "questions": "See weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) To catch on feature learning process during deep learning is a key problem in community.\n2) Paper provides a practical framework with solid theoretical analysis. \n3) Empirical results on different datasets are provided. And clear experimental details are listed.", "weaknesses": "Figure 1 is a bit vague. It is recommended to replace it with a clearer version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RUsLYJEhvF", "reviewer": "Reviewer_MLur", "summary": "This paper introduces the interaction tensor, for empirically analyzing the interaction between data and model through features. Based on some observations using this tensor, they propose a very simple toy model (a combinatorial model) for feature learning. They show that this model also exhibits Generalization Disagreement Equality (GDE). Finally, the authors use their model to provide data distributions that break GDE in real world experiments.", "questions": "- Looking at figure 3 (a) and 3 (c), it seems that the observations made from them are not that significant. Am I missing something? How does the choice of the thresholds affect your observations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The problem of feature learning is quite important and there has been a lot of attempts for gaining a better theoretical understanding of it in recent years.\n\n- The authors are able to come up with a toy model that shows GDE which is a very important phenomenon (and it does need the explicit assumption of calibration).\n\n- The model that the authors propose is simple and it can be analyzed fully.\n\n- The paper is very well-written.", "weaknesses": "- I definitely agree with \"the empirical phenomena of deep learning can be understood at many different layers of\nabstraction\". However, I think the model proposed in this paper is too simplistic. The implicit biases of deep learning are core to some of the merging phenomenon that we see these days and the models that the authors propose fails to capture that. I also think that a good toy model should leave the door open to generalizations and getting closer to real world practice (for example, for the random features model of deep learning, there is a very obvious way to move towards making it more realistic). But the models that the authors propose is too abstract and it is not clear what simplifications are made to the real problem to arrive at the proposed model.\n\n- Although the model is based on some observations using the interaction tensor, I still find the model to be not very well motivated. Any insights on how this can relate to the training of deep nets?\n\n- It is not very clear how the authors set the hyperparameters in their model (e.g., the thresholds).\n\n- The notations are a bit confusing (i, j, k, etc.). I suggest authors avoid using these generic letters. Also, the paper will benefit greatly from a figure that summarizes all the notations (p_d, p_r, etc.). It will also help explain the method.\n\n- The authors \"prove\" GDE in their model without the explicit assumption of calibration. But the model is very abstract/high-level and I'm not sure if the assumptions that they make are stronger or weaker than calibration.\n\n- The theoretical understanding of feature learning is not as rudimentary as the authors claim. For example,\n\n[1] Alex Damian, Jason Lee, and Mahdi Soltanolkotabi. Neural networks can learn representations with gradient descent, 2022.\n\n[2] Zhichao Wang, Andrew Engel, Anand Sarwate, Ioana Dumitriu, and Tony Chiang. Spectral evolution and invariance in linear-width neural networks, 2022.\n\n[3] Eshaan Nichani, Alex Damian, and Jason D Lee. Provable guarantees for nonlinear feature learning in three-layer neural networks, 2023.\n\n\n[4] Yatin Dandi, Florent Krzakala, Bruno Loureiro, Luca Pesce, and Ludovic Stephan. Learning two-layer neural networks, one (giant) step at a time, 2023.\n\n[5] Behrad Moniri, Donghwan Lee, Hamed Hassani, and Edgar Dobriban, A theory of non-linear feature learning with one gradient step in two-layer neural networks, 2023.\n\nand many more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qWHw7eZQQW", "reviewer": "Reviewer_PinD", "summary": "The authors propose a novel framework for analyzing feature learning in deep NNs, from the perspective of observed phenomena occurring in trained networks. The learnt features in a trained model are defined as the principal components of the last layer's activations, and the proposed interaction tensor examines these feature representations of a dataset for a set of models. The interaction tensor is used to empirically validate several common-sense intuitions, such as feature distribution being long-tailed, which then leads to a further abstracted framework for combinatorial-style analysis. This framework is able to theoretically demonstrate empirical phenomena from the observations as well as prior works such as GDE.", "questions": "- While simplifying features and datapoints to be either dominant *or* rare is good enough for the section 6 experiments, have you considered modeling the rare-dominant variation as a spectrum instead of a binary? For instance, looking at figure 2b, I'm not sure where I would want to draw a line to separate the rare from the dominant features. Even putting the line somewhere around x=6.5, there is still a relatively large deal of variation of frequency of occurrence in both the rare and the dominant feature types", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Due to the increasingly complex nature of deep networks, the concept of studying feature learning's observed phenomena like a natural science is a useful supplement to more derivation-based frameworks such as NTK\n- The combinatoric analysis is a creative and original perspective for studying feature learning. While the simplifications might seem a little overly-simplistic at first, the experiments in section 6 and observations in section 4 serve as adequate motivation\n- The resulting findings of the framework not only validate but add nuance to understanding of prior phenomena such as GDE", "weaknesses": "- Like any theoretical framework, many assumptions went into the combinatorial analysis. In particular, this framework assumes features and datapoints are each either dominant *or* rare, a dominant/rare datapoint always has the same number of dominant/rare features, etc\n- The framework is less useful for understanding the learning process of networks, such as why some runs might collapse while others successfully learn the desired features\n- Some minor typos: section 5, data generating process paragraph's second-to-last sentence samples $n_r$ rare, not dominant, features. Appendix C, equation line 34 the two $\\not = \\emptyset $ could maybe instead be $= \\emptyset$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)