Learned feature representations are biased by complexity, learning order, position, and more
Authors: Andrew Kyle Lampinen
Stephanie C.Y. Chan
Katherine Hermann
Venue: Accepted by TMLR ()

Abstract: Representation learning, and interpreting learned representations, are key areas of focus in machine learning and neuroscience. Both fields generally use representations as a means to understand or improve a system's computations. In this work, however, we explore surprising dissociations between representation and computation that may pose challenges for such efforts. We create datasets in which we attempt to match the computational role that different features play, while manipulating other properties of the features or the data. We train various deep learning architectures to compute these multiple abstract features about their inputs. We find that their learned feature representations are systematically biased towards representing some features more strongly than others, depending upon extraneous properties such as feature complexity, the order in which features are learned, and the distribution of features over the inputs. For example, features that are simpler to compute or learned first tend to be represented more strongly and densely than features that are more complex or learned later, even if all features are learned equally well. We also explore how these biases are affected by architectures, optimizers, and training regimes (e.g., in transformers, features decoded earlier in the output sequence also tend to be represented more strongly). Our results help to characterize the inductive biases of gradient-based representation learning. We then illustrate the downstream effects of these biases on various commonly-used methods for analyzing or intervening on representations. These results highlight a key challenge for interpretability---or for comparing the representations of models and brains---disentangling extraneous biases from the computationally important aspects of a system's internal representations.

Published in Transactions on Machine Learning Research (9/2024)
Learned feature representations are biased by complexity, learning order, position, and more
Andrew Kyle Lampinen Google Deep
kampinen@google.com
Stephanic C. Y. Chan Google DeepMind
scychan
Katherine Hermann Google Deed
hermannk@@google..om
Reviewed on OpenReview: https: // optpevveew....ew...mmoo net/ forum? id aY2nsgE97a
Abstract
Representation learning, and interpreting learned representations, key areas of focus in machine lcarning and neuroscience. Both fields generally use reprosentations a mcans to understand or improve a system's 's computations. In this work, however, we explore surprising dissociations between representation and computation that may pose challenges for such efforts. We create datasets in which we attempt to match the computational role that different features play, while manipulating other properties of the features or the data. We train various deep learning architectures compute these multiple abstract features about their inputs. We find that their learned feature representations are systematically biased lowards representing some features more strongly than others, depending upon extraneous properties such as feature complexity, the order in which features are learned, and the distribution of features over the inputs. For example, features that are simpler to compute or learned first tend to be represented more strongly and densely than features that are more complex or learned later, even if all leatures are learned equally well. We also explore how these biases are affected by architectures, optimizers, and training regimes (e.g., in transformers, features decoded carlier in the output sequence also tend to be represented more strongly). Our results help to characterize the inductive biases of gradient-based representation learning. We then illustrate the downstream effects of these biases on various commonly-used methods for analyzing or intervening on representations. These results highlight key challenge for interpretability- or for comparing the representations of models and brains disentangling extraneous biases from the computationally important aspects of system's 's internal representations.
1 Introduction
A key goal of machine learning research learning representations that allow effectively performing down- stream tasks (Bengio et al. 2013). A growing subfield aims to interpret the mechanistic role that these representations play in the models behaviors (Geiger et al., 2021 Olsson et al. 2022 Geiger et al. 2024 Wu et al. 2024), or to alter those representa-
Nanda et al.] 2023 Conmy et al.) 2023 Merullo et al. 2023
tions to improve model alignment, interpretability, or generalization (Sucholutsky el al.) 2023 Bricken el al. 2023a; Zou et al. 2023). Likewise, neuroscience studies the neural representations that a system develops and how they relate its behavior (Churchland & Sejnowski 1990: Baker et al.] 2022). Each field focuses on representations as a means to understand or improve system's computations that is, its more abstract patterns of behavior task, and how those behaviors implemented.

---

Published in Transactions on Machine Learning Research (9/2024)
Train
Easy feature
Hard feature
Analyze Representations
Downstream Implications
Outputs:
classify
Representations biased towards features with certain properties?
o O
features
Looks like this network does easy things...
Manipulate task & data properties
70%
5%
+
... and it matches the brain!
Input
Experiment Domains
Branch# #
Deepest
Shape match
Color
Linear Sum%2
First 3
"([[] [}>}))
Boolean (MLPs)
Language (Transformers)
Vision (CNNs)
An overview of our approach. (top left) We train networks to compute multiple features of an imput, on controlled datasets where we systematically manipulate various properties of the input and target output distribution; for example, to compute easier- and harder-to-computee features of the input. (top center) We study how these properties bias the feature representations the model learns for example, lowards representing easier-lo-computte features more strongly. (lop right) We explore the impacts of those biases on downstream areas like interpretability and cognitive neuroscience. (bottom) We consider these phenomena across range of experiment domains, including boolean functions and various language and vision tasks, using standard architectures for each domain.
Figure 1:
However, the relationship between representation and computation is nontrivial. Thus, using representations understand or improve computation depends fundamentally on the linking assumptions we make about their relationship. Here, attempt to shed empirical light this linking, by characterizing how computational role, and various aspects of the features and extraneous details of the learning process, affect the representations that deep neural networks learn.
To do so, we train various models-Miie abstract foatures about an input, through separate outputs (Fig. D. This approach provides a controlled setting in which each abstract input feature plays an equivalent computational role: contributing to predicting exactly one output value. this controlled setting, we manipulate various properties like feature complexity, the order in which features trained, and their prevalence and distribution within the inputs. We show that, although these propertics do not substantially affect the computational role that each feature plays in the model's input-oulput behavior, they substantially bias the structure of the model's learned representations.
ResNets and Transformers to compute multiple independent
In particular, we find that:
Features that are simpler compute (c.g. lincarly computable from the input) tend be more strongly represented by the model than complex (e.g. non-linear) features that play an equivalent computational role in the model's outputs.
2

---

Published in Transactions on Machine Learning Research (9/2024)
Features that are learned earlier tend to be represented more strongly than those that learned later. Thus, learned representations are substantially path-dependent (cf. Lubana et al. 2023 if a model is pretrained to compute feature A, then trained to additionally compute feature B, it will have different representations than it was trained on feature B before feature A, or if it was trained on both and simult even with similar behavior on Irain and test sets.
Other properties like the prevalence of a feature in the training dataset, or the output position in which the model is trained to report it, can likewise bias feature representations.
These broad patterns of biases are common across range of models and optimizers; they seem to be rather general patterns of gradient-based deep representation learning rather than artifacts of a particular architecture. However, we note there are complex interactions between architecture, optimizer, and task, which can shift or even reverse the biases in some cases.
These feature representation biases have downstream implications for interpreting learned represen- lations, comparing them between different systems, or trying to alter them. Conclusions or results may be biased by relatively extraneous details of the learning process.
In Sections & we set these results in the broader context of related work, such prior work the computational- and feature-level inductive biases of deep learning, and challenges of interpretability. Our results may have implications for both artificial intelligence and cognitive (neuro)science.
2 Definitions & methods
Our goal is to study how feature representations may differ, even when the features play similar computational roles. But what is a feature or a representation? Many of the terms we use are heavily overloaded, or even philosophically disputed. Thus, we begin by defining the way we will use them here.
Input stimulus: a raw imput I provided to a model, for example a tokenized string, a vector of boolean values, or an image.
Feature: an abstract, task-relevant property of an input; e.g., whether "to be or not to be" appears within a sentence, or whether XOR of the first and third boolean value is true. Thus, a feature can defined as the result of applying some function to the input: feature value = fI). Generally, the features discuss are those model is explicitly trained to output.
Computation:
the behavior (i.e. input-output mapping) of system, described at the abstract level of the task-relevant features. For example, if train a vision model report the shape and color of an object in an image, and it does is computing the features shape and color. That is, the task of the model would be to output the vector [fahape (I), fcotor (7) where the feature functions identify the respective values.
a held-out test set of images, say that the model
Computational role of fcature: the downstream contribution of fcature to the desired output be- havior, once that feature's value has been computed. For example, suppose the network is trained to label sentences with whether they contain the word "cake", and whether they are grammatical, through separate outputs (Ifaake JTrammaamati putational role (direct output) once they are computed, even though computing the value of one feature might be much easier than the other.
We say these features play an equivalent com-
Representation:
a distributed activity pattern across neural units within (part of) a model. We will denote the representation the model produces for an input as d(f).
Feature Representation: the portion of a representation (e.g. a vector subspace) that can be predicted from knowing the value of a particular feature for the current input. (Note that we do not here stipulate that this subspace must be causally implicated the model's behavior, unlike, c.g., Caoj 2022, We make this choice for practical reasons, avoid verifying causality throughout training. However, we do verify the causal role of the representations in some cases in 6..
3

---

Published in Transactions on Machine Learning Research (9/2024)
Representation variance explained by feature: the proportion of the total variance across repre- sentations produced by a set of stimuli that is accounted for solely by knowing that feature's value for each stimulus; i.e, the R² of a linear regression from the feature values for each stimulus onto the representations produced for those imput stimuli. See Methods below for a more formal definition. This is a method for quantifying the "strength" of a feature in the representation.
In section we justify some of these choices empirically, for example showing that the representations are causally implicated in each feature's input-output mapping, illustrating how the representation variance explained by feature impacts interpretability methods and RSA, and examining how feature variance in the representations affects downstream learning.
2.1 Methods
In Fig. [ [e show overview of our approach. We primarily train networks output classifications of multiple features, through separate output units (e.g. from MLP), sequence (from a Transformer). We construct the datasets such that the features are statistically independent from one another. In this setting, we know the "correct" computation that the model should learn. We train models on large enough datasets, and for long enough, that they achieve high accuracy (>95%, and often 100%) on the held out test set for all features (except where explicitly noted the text). Thus, we can ensure that the models are computing cach feature a way that broadly generalizes in accordance with our understanding of that feature.
We investigate whether particular properties of the features change the way that those features repre- sented. These properties included feature complexity, feature prevalence in the training dataset, or feature position in the output sequence. We therefore create families of training datasets which systematically ma- nipulate these properties. We also create held-out validation and test datasets corresponding to each training datasct, for analysis and to ensure that the model generalizes as expected. The train/validain /aaaaavaallioottteeee datascts are sampled IID.
To analyze the results, extract internal representations from the trained model for each stimulus in the validation test set, and then inspect those representations are driven by each feature. Our main analyses focus on the variance explained for each feature. To compute this quantity, we fit linear regressions whose inputs are the value of a feature, and whose outputs are the activations of each unit in the model's 's representations. We measure the variance explained by a feature as the R² resulting from the corresponding regression, when the regression fit using the validation set and tested on the test set.
More formally, if ((T.) E R denotes the model's learned representation vector for an input 1 and f(I)) is the corresponding feature value, then via linear regression find the optimal linear predictor W7 of the model's representations on the validation set V from that feature:
W7 = min E [IIW - & (T.) [13]
W NET
We then assess the variance explained by that feature relative to the Lotal variance in the model's represen- tations on the test set T:
E 1 [W7J(7) - & 11 G,T
R²
=
11)
where the total variance ErerT ||ø [12 is simply the variance of the representation units across all inputs in the datasct When comparing representation variance explained over the course of training, we normalize each timepoint relative to the total variance at the end of training (i.c., in the equation above, we replace (7,) ||2 in the denominator with the final value [|or (I.) l|a at all timepoints).
the total variance term (T,) to plot all values on a consistent scale.
With the mean representation across the dataset subtracted out, for notational convenience when writing the variance. "Not that when compare multiple features, this measure could in principle "overcount" variance by attributing the same variance the representations both features. However, because we have constructed the datasets such that the features are statistically independent, we do have this issue in practice.
4

---

Published in Transactions on Machine Learning Research (9/2024)
Representation variance explained (% of final)
Test accuracy (%)
100
Value
50
feature
25
easy
hard
10000
20000
30000
0
10000
20000
30000
Training steps
Figure 2: Test accuracy (left) and representation variance (right) over learning in an MLP trained to output easy and hard features. The casy feature (reading out a single input) is Iearned quite rapidly, but the hard feature (sum of 4 inputs mod 2) is learned slowly; nevertheless, by the end of training the model achieves perfect accuracy for both features more of the penultimate layer represcntation variance than the hard feature. It is particularly interesting to note that long after the easy feature has been mastered, the representation variance altributable to it continues to grow, especially when the harder feature performance is most rapidly improving. (Representation variance is plotted normalized by the total representation variance at the end of training, to show the gradual increase over time. See appendix(0ffor an unnormalized version. Lines are averages across 10 seeds, intervals are 95%-CTs.)
held-out test set. Despite this, the easy feature occupies substantially
3 MLPs on binary features
We begin with some simple experiments on MLPs trained to compute binary features from vectors of 32 binary inputs. This initial sotting motivated by the original results of Hermann & Lampinen (2020) on RSA, and provides a simple domain where the complexity of relationships between input features and their outputs is clearly defined.
3.1 An initial look at complexity bias
We first train a 4-layer MLP to simultaneously output the values of two features embedded amongst other noisy inputs: (1) casy, linear feature (reading out the value of an input unit) (2) harder feature (the sum mod 2 of non-overlapping set of 4 inputs) That is, suppose the input vector I is composed of & boolean inputs 1 = [A, W, X, Y, N.. E {0, 1 where the first 5 will be relevant to one of the features and the remaining N are purely noise that the model must learn to ignore. Then the features are defined respectively as:
fEasy (I) A SHard (W + + + mod 2
We study the representations of these two features at the penultimate layer of the model, immediately before the output logits.
In Fig. 2 we show the basic pattern of results, which illustrates many of the phenomena that we study below. In the left panel, we plot model accuracy on held-out test set over training for each feature (see Appendix B.1 for the corresponding loss curves). The easy linear feature is learned quite rapidly, almost instantly on
The nonlincar feature harder by varicty of overlapping measures, including the minimum depth and width ReLU network that can compute the feature, the number of updates required learn the datasset size required learn this feature in way that generalizes,
5

---

Published in Transactions on Machine Learning Research (9/2024)
Representation variance explained (%)
60
Easy Feature Value
-6
0
40
feature easy hard
-10
-12
Simultaneous Pretrain easy
Pretrain hard Training order
-12.5 -10.0
-7.
-5.0
-25
0.0
2.5
5.0
Figure 4: Penultimate representation variance explained by the easy (lincar) and hard (sum of 4 inputs mod 2) features, across different. train-
Figure 3: The first two principal components of the penultimate representations of MLP af- ter learning the easy (linear) and hard (sum % 2) features (sce Fig. 2). The first PCs of the representations are structured into two clusters defined entirely by the value of the easy feature (colors); the hard feature (shapes) does not sub- stantially impact the top PCs.
ing paradigms. Pretraining to output the easy feature results in very similar representations to training the features simultaneously, since the easy feature is learned first anyway. Pretraining the hard feature closes the gap somewhat, but the easy feature still dominates. (Bars are aver- ages across 10 seeds, intervals are 95%-CTs.)
the scale of the plot By contrast, the hard feature is learned more slowly. In the right hand panel, we plot the portion of the variance in the penultimate representations of the model explained by each feature. Of course, because the easy feature is learned early in training, it initially begins to occupy more variance in the model fcature representations. As the harder feature begins to be learned, it too occupies some variance in the penultimate representations. However, the easy feature dominates the hard one; it occupies far more variance. In fact, even though the easy feature is learned and generalized extremely early in training, as the hard feature begins to be learned, there appears to be some cooperative pressure that actually inflates the total variance occupied by the casy feature far beyond its initial value (note the inflection point in the casy feature variance explained). Thus, throughout training, the penultimate representations of the model are driven mostly by the easy feature, and learning the hard feature actually increases this dominance 5
To visualize this result more intuitively, in Fig. 3 we show the representations the end of training for one of the models, projected down to the top two principal components (PCs). The representations are entirely structured into two clusters defined by the value of the easy feature; there is little impact of the harder feature. Of course, because the hard feature is represented by the model in generalizable way, it must be encoded in the representations somewhere, and indeed in $6.2.1 we show that it is captured in higher principal components, depending on which other features are being learned. However, simply from looking at the redured--iimenssionlltty representations shown here, might seemn that the model is doing something quite simple, locused only on the value of the easy feature.
3.2 Is the effect due to feature complexity or simply learning order?
Where does the gap in representational variance between the easy and complex features originate? it due the effect of feature complexity in itself, or the fact that the easy feature is learned earlier? To disentangle
'In fact, the easy lincar feature linearly decodable with 208% test accuracy from the penultimate representations of untrained uetwork.
WW provide simple
colab
demonstrating
this
result
httpp://gistt. www.gggtt......agpggeegeeeeeeeee wwt.//apimmnnnnddd...

---

Published in Transactions on Machine Learning Research (9/2024)
the contributions of these two factors, in this section experimentally manipulate the training order-either training both features simultaneously, as above, or pretraining one feature to convergence before beginning to train the other. By pretraining the more difficult feature, we ensure that it is learned first, and disentangle the effect of learning order from the effect of difficulty.
In Fig- we show the results. Pretraining the easy feature does not substantially change the results, which is sensible since the easy feature is learned first regardless. Pretraining the hard feature does close the gap somewhat, but the easy feature still dominates the representations. See Appx. B.3 for learning curves.
3.3 Varying features and accounting for many ways they can be represented
If learning order alone does not explain the dominance of the easy feature, what aspects of fealure complexity might? To explore this, we trained models on a larger set of hard features, ranging in difficulty from simple AND of groups of inputs to more complex ones. In analyzing these models considered another property of noulinear features: there are more reasonable ways to compute them. For example, XOR(X,Y) can be decomposed as AND(OR(X.Y) NOT YN)( or as OR(AND(X, NOT Y), AND(NOT X, Y)), which yield different patterns of representations (cf. Fig. A.14 in Hermann & Lampinen 2020). By contrast, all simple ways of encoding the linear feature are linearly equivalent. Could it be that the model is strongly representing components of the hard feature, even if it not representing the hard feature itself?
To test this possibility, we measured the variance explained by all patterns of inputs relevant to the hard feature. That is, enumerated all the possible hard-feature-relvvar imput patterns and then created one-hot embedding for each of them. We then regressed from this set of labels onto the representations, The advantage to this analysis is that any function of the hard inputs alone is linear in this input-pattern space it is essentially a labular function representation where there is a separate entry for each possible input pattern- and thus it can account for any linear and non-linear patterns of encoding the hard input, as long as they do not mix in other irrelevant inputs. (Note that the casy feature inherently captures all relevant input patterns, so there no need to perform an equivalent analysis for the easy feature.)
The results of this analysis are shown in Fig. 5 In sum, accounting for the variance of all imput patterns does not by itself explain the gap between easy and hard features; if the features are trained simultaneously the gap remains quite large. However, if the hard feature is pretrained, and the analyses are performed with respect to all hard input patterns, the gap vanishes, and there is even a slight bias towards the hard feature in some cases. Thus, the easy-hard gap seems to be explained by a combination of hard features being learned later, and the fact that there are more ways to represent harder fcatures.
3.4 Additional evaluations of complexity biases on unit level, and with additional features
We present some additional evaluations in full detail in the supplement, but summarize them here.
Per-unit analyses: We find similar qualitative biases, as well as interesting individual effects, when ana- lyzing each unit in the penultimate layer individually (Appendix B.4.
Sparsity: We also find representation sparsity biases: more complex features tend to produce sparser representations (Appendix 24
More than two features: Finally, in Appendix B.6 we show that there are qualitatively similar, graded biases when the models are trained simultancously on more features sampled from a set of varying difficulty.
3.5 Feature prevalence also biases representations
In the above experiments, all features were balanced to appear equally often in the dataset. However, distri- butions of properties in naturalistic data tend to be skewed (e.g. Piantadosi, 2014), and these distributional properties can shape the solutions that models learn (e.g. Chan et al. 2022). Thus, we explored whether feature prevalence affects the representational biases we observed above.
To do so, we constructed datasets containing two linear features (casy), and two sum-mod-2 features (hard), where the first feature of each type was present (ie. had label of 1) in 50% of the data (common), but
7

---

Published in Transactions on Machine Learning Research (9/2024)
Representatin variance explained (%)
Simultaneous
Pretrain easy
Pretrain hard
0.6
0.4
feature
easy
hard
hard all input patters
0.2
0.0
XOR(OR(WX)) OB(YZ))
ANDIWXXY22)
XOR(ANDIWXX) AND(Y.21)
ANDIXOR(W.X). XOR(Y,2)
XOR(WX)
SUMOWX.Y.Z)992
XOR(OR(WX) ORIY.2))
ANDIWXXY21
XOR(ANDOW.X), AND(Y.21)
ANDIXOR(WLX). XOR(Y,2)
XOR(WXX)
SUMW.X.Y.2)94
XOR(OR(WX). ORTY.2))
ANDIWXY.11
XOR(ANDIWX)), AND(Y.21)
AND(XOR(WXX). XOR(Y,2)
XOR(WX)
SUMWAXYY
Hard feature
Figure 5: Penultimate representation variance explained at the end of training by the casy (lincar) and hard features, as well as all the patterns over the inputs relevant to the hard feature, and across different training paradigms.
Encoder R 2 (%)
Penultimate R 2 (%)
2.0
Representation variance explained
1.5
Represertation vanaace
explained (96)
15
1.0
40
10
0.0
Easy (8h))
Easy (D%)
Hard (515)
Herd (10%)
-0.5
I-exactmatch
eeaccmatch
d F-eacttmaahh s-atactmatch
Z-aituoone
3 "aliiutone
F-aliutton
I-summed2
--aactmatc 2-exactmalch
D-exactmathh
d-exactmatch
2 2-aliiutton aaleuutone
3 mithuun
i-giitutone
--sumodz
Feature
Figure 6: Feature prevalence, as well as difficulty, affects representational vari- ance in MLPs. Featurcs that appear less frequently the training data lend explain less variance the model's penultimate representations than more prevalent features of equal difficulty. (Bars are averages across 32 seeds, in- tervals are 95%-CIs.)
Figure 7: Representation variance explained at the final encoder layer (left), and pre-logits decoder layer (right) for transformers trained on character sequence classification datasets. In the encoder, features explain relatively lit- tle of the overall variance (note vertical axis scale). There are strong effects of number of tokens involved, as well as complexity. In the penultimate decoder layer (right), the features follow the expected complexity ordering, though there appears to be some effect of number of tokens. (Bars are averages across 128 seeds, intervals are 95%-CTs.
8

---

Published in Transactions on Machine Learning Research (9/2024)
the other appeared in only 10% of the data (rare). For example, for the easy rare feature, the corresponding input would be O in 90% of the dataset examples. We analyzed how difficulty and prevalence jointly influence representations.
In Fig. 6 we show the results. Feature prevalence (in addition to difficulty) indeed affects the feature representations. This somewhat intuitiive as more frequently occurring features effectively receive more gradient updates, and thus are learned earlier. However, again the model achieves perfect accuracy and very low loss across all features; thus, it would not be immediately obvious from the model's behavior that one feature should be represented more strongly than another because of frequency artifact in the training distribution. This result provides yet another demonstration that when model that is computing effectively the same thing generalizing to the test set for both the frequent and rare version of each feature it can nevertheless implement those computations in ways that are biased towards one feature or another.
3.6 The effects of architecture and hyperparameters-
weak but complex interactions
How general are our findings? In Appx. B.7 we explore the effects of MLP architecture and other hyper- paramcters. We summarize the results here. First, deeper MLPs tend to show larger reprosentational gaps between easy and hard features; model width has little effect. Despite the fact that nonlinearities strongly influence the structure of learned representations (Saxe et al. 2019 Alleman al. 2024h, our general pattern of findings is robust to choice of nonlinearity (tanh or rectifier).
There is, however, a surprising interaction between dropout and choice of optimizer at convergence. Without dropout, all optimizers show similar effects, with large gaps ween easy and hard features. However, we show in Appx. [37.] that dropout yields strikingly different effects with different optimizers at convergence; while SGD and AdaGrad show consistent gaps between easy and hard features, other common optimizers like Adam show qualitatively different effects with dropout, and in fact show a slight hard feature bias at moderate dropout rates. However, this is only true convergence- earlier in training, even when the hard features have been learned and generalize quile well, the easy features remain dominant. Furthermore, we also show that transformers trained on more complex datasets (see below) show more consistent feature representation biases across dropout levels.
Optimizers and dropout:
4 Transformers on sequence tasks
The majority of models employed in modern deep learning are not MLPs. In recent years, Transformer (Vaswani et al. 2017) models have come to dominate, in language and beyond. We therefore assessed whether transformers show similar effects to those we observed in simpler architectures, and whether the additional properties of languago-like data for examplc, positions and structure of tokens within a soquence- likewise bias feature representations.
To do so, we instantiated several increasingly language-like training regimes. The simplest are character- level sequence-to-sequence language datasets that bear some similarity to those used above, but with few additional properties. These datasets involve mapping a sequence of letters to sequence of classifications of non-overlapping subsequences of the original sequence. In particular, we consider three types of features. The simplest is exact-match vs. non-match to a particular letter sequence (analogous to the linear feature above), for example if the sequence is 'A,,C'' and 0 otherwise. We also tested two increasingly nonlinear features: matching all-but-one of the letters in the sequence (XOR-like), and the sum of matches modulo 2 (analogous to the most complex fcature above). In addition, we evaluate the effect of the number of letters in the larget sequence, and the position within the outpul sequence. Specifically, we created datasets containing 8 features (exact match of 1-4 token sequences, all-but-one for 2-4 token subsequences, and sum- num-mod-2 for 4 token subsequences), along with distractor characters in between. We sampled the feature order randomly for each dataset.
We trained encoder-decoder transformers on 128 of these datasets and evaluated the representation variance explained in the last representation of the encoders (after concatenating all token representations), and the variance explained immediately before the logit layer of the decoder when outputting that feature. That is, unlike in the MLP experiments above, in the decoder the representations of different features are longer
9

---

Published in Transactions on Machine Learning Research (9/2024)
Encoder 2 (%)
Penultimate 2 (%)
Representatinn variance explained (%)
2.0
70
feature
1.5
1-exactmatch
2-exactmatch
1.0
3-exa 3-eaachmatch
60
4 4-exa ctmatch
2 2-alibutone
0.5
3-allbutone
4-allbutone
50
4 4-summod2
0.0
-0.5
4
8
2
4
6
Feature position in output sequence
Figure 8: Position in the output sequence affects representation variance explained at the pre-logits decoder layer (right), but has little effect at the final encoder layer (left), for transformers trained on character sequence classification datasets. Features that appear earlier in the sequence are generally represented more strongly. (Lines are averages across 128 feature orderings, errorbars are 95%-Cls.)
being analyzed simultaneously the same representations, instead they analyzed at different positions in the output sequence.
In Fig. we show the results. There are several interesting phenomena to note. First, none of the features explains very much of the overall variance in the penultimate encoder representations. (Note that the variance explained can be negative due to failure to generalize, because we fit the regressions using a validation set and compute variance explained on the test set.) Nevertheless, there are clear biases based on both complexity and the number of tokens involved in computing the feature; generally features involving more tokens are represented more strongly, while more complex features represented less strongly.
In the penultimate decoder layer, we observe results that are qualitatively similar to those observed in MLPs; simpler features explain more variance than more complex ones. Number of tokens does still have some effect, but the pattern less clear. Moreover, we observe substantial output order effects on representations in the decoder (Fig. in general, features earlier in the sequence occupy more variance.
Sequential training: sequential training of one feature at time also yields interesting biases, which shift over training in distinct ways for the encoder and decoder (Appx. B.8
4.1 Dyck languages: structural and token-level properties dominate in different model components
We next trained transformers on set of more complex tasks involving processing hierarchical structures from string inputs. More specifically, these tasks are based on the Dyck languages strings of balanced brackets. These languages have been used in a variety of prior works studying generalization and interpretability in RNNs (Hewitt al. 2020) and Transformers (Yao et al. 2021] Murty et al., 2023 Friedman et al.] 2023a Like several prior works, we create tasks basod on the versions of these languages with bounded depth (Hewitt et al. 2020)). We sample strings from (20k-1)0,10); that is, balanced strings over 20 brackets with nesting depth most 10.
However, in contrast to prior works (which have mostly focused on language modeling), we train transformers to output a variety of classifications of these sentences, including structure-indepeedent token-level "seman- tic" features (a binary classification of the first bracket type), analogous token-level but strueture-deeendent features (classifying the type of the first bracket at maximum depth), or purely structural features (the max- imum depth, number of root brackets, and maximum branching factor) of the sequence. These features do not capture the full structure of the sentences, but allow for more controlled experiments and analyses than
10

---

Published in Transactions on Machine Learning Research (9/2024)
Encoder R/2 (%)
Penultimate R (%)
Representation varlance explained (%)
2.0
E controlled
Ponuttmate 8 controlled)
60
Rapresertation vanaanos exploined (%6)
1.5
40
1.0
0.5
20
0.0
esent-or-pine -iist
nested -ar--------eee de-eeeuettaal
valence
NI Wi-siingutrr
N2 W2-singutar
N1 -i-vat
N2-ralue
present-er-paa
nested- varr-----uuuee -equenttiaal
valerice
Ni Wi-singular
N2 #2-singular
N1 Ri-value
N2 N2-value
A0 token cless
Despest token cless
Max depan
Frost nodies
Max trancning
First token class
Desppett token cless
Max depan
Front nodies
Max branciing
Feature
Feature
Figure 10: Representation variance explained at the final encoder layer (left), and pre-logits de- coder layer (right) for transformers trained to identify semantic and syntactic features of sen- tences sampled from semi-naturalistic gram- mar. In the encoder, structural syntactic fea- tures dominate, but there is notable variabil- ity among the other features. At the output, there are strong effects of number of possible out- put values for a feature features like identifying the noun that plays particular role within the sentence (yellow) are much more strongly repre- sented than abstractions about that noun that only take two values (such as whether it is plu- ral or singular, light purple). (Bars are averages across 32 seeds, intervals are 95%-CIs.)
Figure 9: Representation variance explained at the final encoder layer (left), and pre-logits de- coder layer (right) for transformers trained to identify semantic and structural features of sen- tences from Dyck-(20,10). Again, there is a bias towards simpler features at the penultimate layer; at the encoder layer, however, the most variance is explained by depth. (Because so- quence length correlates with features, this plot controls for sequence length; raw results ported in Fig. 35 Bars are averages across 32 seeds, intervals are 95%-CIs.)
doing full language modelling. As these structural tasks are more challenging, the models do not achieve perfect generalization performance the held-out test set in every case, which allows us to assess the impact of imperfect generalization on the results.
We show the basic effect of feature type representation variance in Fig. At the pemultimate decoder layer, substantially more variance is explained by the simplest, non-structural, feature compared to the fea- tures involving structure. At the encoder layer, however, the pattern is nearly the opposite most variance is explained by a structural feature (the maximun nesting depth of the sequence). Though the task formu- lation is different, this finding is broadly in keeping with prior results about how early layers of transformers tend to strongly represent depth these structures Yao al., 2021 Friedman et al., 2023a)). Overall, we again observe substantial biases in the representations of the features, but the nature of those biases shifts across the components of the model. Nevertheless, even aggregating across different components of the model, there are clear biases towards some features over others.
In Appendix B.9 we present some supplemental analyses of these results, including showing that the repre- sentation variance explained by each feature across seeds does not appear to be strongly correlated with the test accuracy of that feature.
4.2 More language-like datasets reveal an additional output-range bias
We next evaluated transformers on of datasets that are more like natural language: sentences sampled from grammar that yields real English sentences with complex syntactic structures (center embedding
11

---

Published in Transactions on Machine Learning Research (9/2024)
sequential dependencies), and semantic properties (such valence and noun properties). See Appendix A.2.2 for details. We trained transformers to output various judgements about these sentences, such as identifying the sentence structure, or reporting which noun is the main subject. We analyze the representations with respect to these features.
In Fig. 10 we show the results. There are several interesting patterns. At the final encoder layer, the repre- sentations are again dominated by the syntactic feature (sentence structure). However, at the penultimate decoder layer, the variance is dominated by two features that involve reporting the identity of the nouns that play different roles in the sentence. Reporting more general features of these noums, such as whether they are plural or singular, does not lead to as high of representational variance. Likewise, reporting valence of the sentence (which can be positive, neutral, or negative) carries more variance than reporting whether the sentence in present or past tense (two values). Thus, the number of possible output values a feature seems reason-
can take may strongly affect the variance its representations carry near the output-wwie able in retrospect, given that cach possible value that can be output is effectively a separate output logit. These patterns are also reflected in the encoder, but a lesser extent. These results show that biases still persist in slightly more naturalistic settings, and can also be driven by the number of possible outputs associated with feature.
5
Vision
5 pattial pool 2 (88)
Penultimate R 72 (56)
pattil pool 2 (8)
Penultimate R/2 (%)
Representation variance explained (%)
Representation variance explained (%)
6
10
shape
color
color
shape
shape
Feature
Feature
(a) Non-residual CNNs
(b) ResNets
Figure 11: Representation variance explained the final pooling layer, and penultimate MLP layer for non-residual CNNs and ResNets trained to classify colors, textures, and shapes of objects in images. (a) non-residual CNNs show strong representational bias towards color the easiest feature at the spatial pooling layer, which persists with reduced magnitude at the penultimate layer. (b) ResNets instead show a strong bias towards shape representations, which again somewhat weaker at the penultimate layer. (Bars are averages across 32 seeds, intervals are 95%-CTs.)
We round out our experiments by considering a third large class of problems woolems-comppuerr vision and some architectures typically used to solve them. Specifically, we construct synthetic datasets of images that show objects with distinct colors, shapes, and textures (examples provided Appendix A.2.3). We train ResNets (He et al. 2016) and other CNNs without residual connections to classify all of these features simultaneously architectural details in Appendix A.1 These experiments are again inspired by prior work finding that features like color are casier for models to learn, and will dominate when multiple features compete to predict label Hermann & Lampinen 2020). Here, we investigate the models' internal representations of these features when all features are learned.
We find that non-residual CNNs exhibit a strong bias towards the simplest feature (color, which can be decoded from any single pixel) the final spatial pooling layer, and somewhat weaker bias at the penul- timate classifier layer (Fig. 11a This is in keeping with the general pattern observed above, that models
12

---

Published in Transactions on Machine Learning Research (9/2024)
da Resprseentaioon variance explained (%)
Representation variance explained (%)
Spattal pool 72 (%)
Penuttmate R 18)
Spatial pool R/2 (*2)
Ponultimato R/2 (%)
20
20
10
0
(hinary)
toture (binary)
shape (binary)
colors match?
latures match?
shapes match?
color (hinary)
anture (tinary)
shape (sinary)
colors match?
footures match?
stapes match?
color (binary)
Iesture (tinary)
shape (binary)
colors matsh?
festuran match?
shapoe match??
color (tinary)
locture (binary)
shape (tinary)
colore metch??
testuree match??
ahapes match??
F Feature
Feature
(a) Non-residual CNNs
(b) ResNets
Figure 12: Representation variance explained at the final pooling layer, and penultimate MLP layer for non- residual CNNs and ResNets trained to classify properties of one object, and relations among the properties of the other two, in images with three objects. In this setting, both non-residual CNNs and ResNets show more similar biases, with higher variance explained by the basic features than the relational ones, and higher variance explained by color/texture than shape. Intriguingly, these biases are mostly attenuated by the penultimate layer in this sctting, for both model classes. (Bars are averages across 32 sceds, intervals are 95%-CIs.)
are biased towards representing simpler fcatures more strongly, as well as with prior work. Intriguingly, however, ResNets exhibit the opposite bias-wite representations that strongly favor shape (Fig. 11b]. This is despite the fact that color remains the easiest feature, and shape the most difficult (slowest) feature to learn for both models (Fig. 38 as in Hermann & Lampinen 2020
One possibility that this bias towards shape may stem from the interaction of the residual connections and the fact that shape requires integrating information over larger areas. Similarly, in the transformer experiments 4 we noted relatively stronger biases in the encoders which have residual connections to the input features that require integrating over larger input areas. However, we do not see shifts in the easy-hard feature biases of residually connected MLPs (Fig. 27). Taken together, these findings suggest that residual connections across layers may shift the biases of models with respect to extent in space or a sequence. Note, however, that these biases also interact with other features of the input and output structure, as we will see in the next section.
5.1 Relational vision tasks show similar complexity biases
All the visual features considered above involve computations that are arguably simpler (disjunctive template matching) than the relational features (like XOR or grammatical structure) that considered in our ML. and Transformer experiments. Thus, we performed some experiments where we trained models on images with three objects, and asked them to classify the properties of one object, but to report the relations (same or different) among the properties of the other two objects. We reduced the basic category classifications to binary labels (arbitrary splits of each type), to equalize the number of classes of basic and relational categories (as we observed above that the number of classes affects variance).
In Fig. I2 we show the results. As would be expected, the easier basic feature representations tend to carry more variance than the harder relational ones at the final spatial pooling layer in both architectures. Surprisingly, both model classes show consistent biases for color and texture over shape, for the basic features. All biases are very weak by the penultimate layer; moreso than we observed above. In general, the biases are weaker in this setting than with more output classes; together with a supplemental experiment (Appendix Fig. 39) showing that binary classifications show a weaker bias in the sctting of the original classifications above, these results suggest that feature biases may be stronger with larger output spaces, which accords with transformer results showing that number of labels itself can be strong biasing factor.
13

---

Published in Transactions on Machine Learning Research (9/2024)
Why do the basic feature biases of ResNets change direction in these tasks, compared to those above? In follow-up experiments (Appendix 1. we show that switching to binary classification, and the addition of other objects to the image, may both contribute to reversing these biases. These results highlight the complex interactions that affect representational biases. However, the bias towards basic over relational Teatures seems relatively more consistent.
6 Why should we care? Evaluating downstream impacts
Computing representational variance explained by features is not a standard analysis. However, the variance explained by the features has various consequences. Some of these consequences are mathematically neces- sary, and we illustrate them purely for clarity, but some we merely observe empirically. In this section, we illustrate these downstream implications in causal interventions, interpretability, representational similarity analyses, and use of representations for downstream lasks.
6.1 The representations identified are causally sufficient and specific
The above analyses are correlational. However, most philosophical accounts (e.g. Cao 2022
Baker et al.
2022) define representations partly in terms of their causal role in computation. An increasing number of approaches in interpretability similarly causal interventions to achieve more faithful interpretations of model internal representations (e.g- Geiger et al. 2021 Meng et al. 2022 Geiger et al. 2024). Here, verily in one case that the representations identified in our analyses indeed play a causal role in the computations.
To do so, we take an MLP model (from the Fig. |expermmnts) at the end of training, manipulate its internal representations, and evaluate the causal effect on its output behavior.
To manipulate model representations of feature, we take the representation pattern predicted when that feature is present absent and subtract those two patterns to create present-absent difference vector. We then add (or subtract) scalar multiple of this difference vector the representation of each stimulus in the test dataset, and evaluate whether by doing so we can flip the label the model predicts for that fcature on that stimulus. This approach is equivalent to approaches that create steering vectors from activation differences (e.g. Turner et al. 2023).
We show the results in Fig. 13
we intervene along the subspace defined by the casy fcature difference vector, we can flip the models labels of the easy feature for all items, without altering any of its labels for the hard feature. Similarly, if we intervene along the direction of the hard feature difference vector, flip the hard labels without altering the easy ones (unless we make an extremely large intervention). Thus, the representations identified in our regressions are causally sufficient and specific.
Moreover, these results provide another way to see that the hard features occupy less variance than the easy feature much smaller interventions on the representations suffice to flip the models' hard feature labels. These results might also have downstream consequences for each feature's robustness to perturbation.
6.2 Impact on model simplification for interpretability
Many approaches to mechanistic interpretability involve simplifying the model's representations, for example visualizing their top principal components (PCs). However, these simplifications may exacerbate feature biases; if the top PCs are biased towards certain features, lower-variance features may be missed in the interpretation. Thus, these representation simplifications may not faithfully capture the computations of the original model.
We describe one way in which this could occur here, motivated by the experiments of Friedman et al. 2023a That work shows that certain simplifications may be less faithful -itto--disttibution and thus interpretations basod on these simplifications may not give generalizable understanding of a model's computations. For example, keeping only the top principal components of a transformer model's keys and queries can yield accurate model behavior in distribution, but fail to generalize appropriately to more challenging test sets,
14

---

Published in Transactions on Machine Learning Research (9/2024)
100
Feature evaluated
75
casy autsparn
easy
50
hard
Sewwsre on
Labels fllpped (%)
25
100
Accuracy (%)
75
Saateee pura Patepaen
Uo Sosoouua
50
50
feature
25
easy
hard
10.0 100.0 R Representation intervention magnitude (L2)
0.1
1.0
Representation SVD components kept
3
If we keep only the top-k prin- cipal components from a model's represen- tations, the easy feature will be preserved with smaller k than the hard feature.
Figure 14:
Figure 13: The representations identified by analy- ses are causally responsible for the model's behavior. II we intervene along the subspace defined by the each fea- ture, we can flip the model's labels for that feature for every item without altering any of its labels for the other feature. Smaller interventions are sufficient to flip hard feature labels because the hard feature occupies less rep- resentational variance. (Vertical dashed line shows the difference vector magnitude from the regression, ie.. the intervention that we expect would flip all labels.)
suggesting that the original model may use subtler features in more challenging cases, that the simplifications omit.
Though our experimental settings are somewhat different, our results hint at a possible basis for those phenomena. In particular, the features required to solve the harder OOD splits are more complex, they may occupy relatively less variance, and therefore be more readily lost when keeping only the top principal components. We show a simple demonstration of this from one of our MLP models in Fig. 14 if we keep only two principal components from the models representations, the model perfectly, while hard feature accuracy remains at chance until we keep at least four principal components. (In Appx. B..1 we show that dropping the top PCs yiclds the opposite pattern, with hard features proving more robust).
still identify the easy feature
6.2.1 Impact on feature visualization
Given the above observations, clearly low-dimensional visualization may be biased towards certain features. We showed a simple example of this in Fig. 3] in which the top PCs are entirely clustered by the easy feature. In Appx. B.12 we elaborate these results, showing that the hard feature appears in the next PC. However, if models are trained with more and more casy features, the hard feature is pushed into higher and higher PCs. Visualizing with (-SNE yields somewhat better results with only a single easy feature due to t-SNE's nonlinear embedding and emphasis on neighbors- but once enough easy features are added it similarly becomes difficult to identify the hard feature in low-dimensional t-SNE plots as well, while the easy features show extremely clear clusters. Thus, feature visualizations may be inherently biased towards simpler features, those that are learned earlier, etc.
15

---

Published in Transactions on Machine Learning Research (9/2024)
(SUM82 2
(SUM%2, ANDIXOR XOR
(S SUM% XOR
1S U (SUM%) %2 Linear
AND
JANDIXOR XOR SUM%2) ANDIXOR XOR
(AND(XOR,
Network B tasks
(ANDI XOR,XOR XOR)) AND
JANDIXOR, XOR XOR
(ANDIXOR, XOR Linear
(XOR. SUM%
[XOR, ANDIXOR XOR
XXR, XOR
(XOR, AND
(XOR, Linear
IAN SUM%
[AND, ANDIXOR
IAND.
(AND, Linear)
MUM
(Linear, AND(XOR, XOR
(Linear, XOR
AND
(Linear, Linear)
(Linear, Lincan)
AND)
(Linew, XOR
MOXXOR.K (ORX
(Linear, SUM%
(AND. Linean
(AND. AND)
(AND XOR
(AND. ANDIXOR, (ORXXOR)
(ANI. SUM%?
(XOR, Linear
(XOR. AND)
(XOR XOR)
(XOR, ANDIXOB KOR. XOR)
(KOR, SUM%))
BOOKOO XOR Linear)
UNNEXOR XOR). AND)
XOR FORLXOO XOR), XOR)
YOROR AND XORJ)
INDDOOO XOR). SUM%))
SUMMIZ (nserr)
USUMAR2 AND)
UAMMS XOR)
ANDDXOII XORI
(SU SUMMM2 SUMM))
(Linear, ANDIXOR
Representational similarity
(SUMSSZ
(ANDIXOR
1.00
0.75
(SUM)2
0.50
(AND
0.00
Network A tasks
Figure 15: Representational similarities among MLPs trained to compute different pairs of features. The feature representation biases produce strong biases in which networks appear similar; in some cases, networks may appear similar to ones that computing entirely different features than they do to another network computing the same features. (Results computed across 5 seeds per feature pair, using Euclidean distances and Pearson correlation.)
6.3 Impact on Representational Similarity Analysis
Representational Similarity Analysis (RSA; Kriegeskorte et al. 2008) is technique for comparing two sys- tems (c.g. a model and the brain) based on the (dis)similarity structure of their internal represontations. The basic idea is that while two systems' representations are not directly comparable, their similarity structures are. RSA and related techniques (e-g- CKA; Kornblith al. 2019) are commonly used in neuroscience compare models brains, and in Al for model analysis (Sucholutsky et al., 2023). However, several prior works have noted that RSA may not perfectly reflect computational similarity [Maheswaranathan et al.,
2019 Hermann & Lampinen 2020: Dujmović et al. 2023 Friedman et al. 2023b); our findings likewise highlight the potential for representational similarity to dissociate from computational similarity.
To illustrate this, in Fig. 15 we perform an RSA analysis between many pairs of MLP networks. Each MT. trained to output two features of varying difficulty. Tdeally, the RSA would be more similar on the diagonal (for networks trained perform the same two tasks) than off the diagonal, and those on the block diagonals (trained with just one feature that the same) would be the feature representation biases lead to substantially more biased structure. For example, two models that are both computing SUM%2 over two sets of features will appear to be less similar to one another than each appears to be a network that is solely computing simple linear features. These results use Euclidean metric for RSA, which is naturally sensitive to variance. However, in Appx. B.13 we show that the results are qualitatively similar with cosine distance as the metric.
similar than those off. However,
These results highlight how RSA may be misleading when representations are driven more by some features than by others.
16

---

Published in Transactions on Machine Learning Research (9/2024)
6.4 Impact on downstream use of representations
A key motivation for representation learning is to use the representations to help solve downstream tasks. Here, we assess how feature biases in the pretrained representations will affect the behavior of the downstream model.
100
Classiffer feature preference (%)
To do so, we take several MLP models trained above, and train (non)linear classifiers top of their rep- resentations, to predict label that is probabilisti- cally related to both casy and hard features inde- pendently. More specifically, we create a range of datasets where the label is independently predicted by the easy and hard features with varying probabil- ity, but where both features are equally predictive. We then test. the extent. to which the classifiers rely cach feature by testing on datascts where the fea- tures are decorrelated, and only one feature predicts the label. This setting is similar to some of the ex- periments of Hermann & Lampinen (2020) and Her- mann et al. (2023), but beginning from multi-task pretrained representations rather than raw inputs.
60
Feature
easy
hard
Classifier hiddens
(128)
[128, 128)
0.6
0,7
0.8
0.9
1.0
pilabelleasy) p(label[hard)
Figure 16: Representational biases affect the general- ization behavior of downstream ML models that build on the representations. This figure shows the results of training classifiers (linear nonlinear) on top of the representations of an MLP model to predict a fea- ture that is stochastically related to both the easy and hard features. both features are highly predictive, the classifiers are strongly biased toward relying on the easier feature. (Results are similar for both linear and nonlincar classifiers, so collapse across them.)
In this setting, we show in Fig. 16 that the biases in the feature representations affect the downstream model behavior. Specifically, we plot the degree to which the classifier outputs "prefer" each feature when they are pitted against one another, across values of feature predictivity. If the features have low predictivity, the classifier relies equally on both features. However, the features' predictivity increases, the classifiers exhibits increasing bias towards the easier feature.
Thus, these biases in feature representation, even in model that learns both features successfully, will affect how downstream models behave.
7 Related work
Implicit inductive biases towards simplicity: Highly over-parameterized deep networks sometimes generalize well even when they could memorize the entire datasct (Zhang al. 2021). To make sense of these observations, variety of works have proposed implicit inductive bias for simplicity, either in the
architectures themselves (Valle-Perez et al. 2018 Huh et al. 2021 Rahaman et al.
2019) or in combination
with gradient-based learning dynamics (Tachet et al. 2018 Arora et al. 2019 Advani et al. 2020 Kalimeris
let al. 2019 Lampinen & Ganguli, 2018 Shah et al. 2020
Lu et al.] 2023 Xu et al. 2023) In particular,
Valle- Perez et al. (2018) emphasize that networks with random parameters are biased towards computing simpler functions, which may contribute to the relative case of lcarning simpler fcaturcs and the greater sparsily in representations of more complex features.
However, most of these prior works have focused the case where the features redundantly serve single task, rather than the case where the network is computing all of them, and have mcasured this bias at the level of the input-oulput behavior, rather than in the network's 's internal representations. Our results suggest that the impact of these biases persists in the network's internal representations even in a setting where it has learned to compute the more complex features equally well. Morcover, these representational biases may even help to amplify other inductive biases because, once some features are more strongly represented, they will be favored in later layers' learning (ef. 66.4 thus, they may be part of the mechanism of the
17

---

Published in Transactions on Machine Learning Research (9/2024)
broader phenomenon, rather than just a consequence. However, there are also cases like the shape bias in ResNets trained on single-object images, despite color being learned first-in which the behavioral and representational biases follow distinct trajectories over learning.
Representational biases towards simple, common features: Several other empirical works have noted that representations can be biased towards capturing simpler features and suppressing irrelevant (or more complex) ones, including [ermann & Lampinen (2020) and more recent work on contrastive learning (Xue et al. 2023), and boolean feature learning QQiu et al. 2024p. Other works have noted feature-level biases in which image features are easiest to learn (e.g. Dasgupta et al. 2022). In addition, several works have noted biases due to prevalence, c.g. that learned image features focus more on prevalent features (Benjamin et al. 2022), cosine similarity of word embeddingg is biased by prevalence (Zhou et al.
2022 or that pruning
selectively harms performance on rare exceptions within a category (Hooker et al. 2019 A more recent study by Morwani et al. (0033) suggests that some representational biases may stem from networks learning maximum- maximum-margin solutions. One carlier work on modularity (Csordás et al. 2021) noted in passing some biases in the proportion of network weighls devoted to two different lasks. However, aside from (hat, most works have focused on the case where multiple features are used to predict the same label, and thus there is effective competition between features. Here we characterize these biases more systematically, in the case that all features are playing an equally-importantt, independent computational role. Moreover, we go beyond complexity to characterize other biases, such as the output position biases in transformers.
A concurrent work from Fel et al. (2024) studies the features learned by ResNets trained on ImageNet (Deng et al., 2009), and finds broadly compatible patterns: simpler features tend to be learned earlier than more complex ones, and simpler features also tend to be more important to the network's decisions- but complex features do still play a role. These results complement ours by exploring a more realistic setting, which correspondingly implies less systematic control over the task or the statistics of the input and input-output distributions.
Shortcut learning: A variety of works have studied the problem of shorteut learning (Geirhos et al. 2020 i.e., that networks may learn to utilize spuriously-coorrllteedd features in the training dataset, and thus fail to generalize to distr--disttibuttin tests. For example, IeageNet models are often biased towards using texture over shape (Geirhos et al. 2018), and this bias is shaped by the choices like augmentations typically used in training (Hermann el al.] 2020 Similarly, a variely of works have shown that (small) language models often rely on semantic heuristics rather than more complex syntactic computations (MeCoy et al. 2019 2020).
In many cases these shortcuts appear simpler than the true solution, which suggests that they might ht arise similarly from the inductive bias towards simplicity (cf. [Shah et al. 2020). Indeed, Hermann et al. (2023) show that models are biased towards preferring features that are more available (e.g. have greater spatial extent). However, again most of these works have considered settings where the features are both relevant to single task, rather than serving independent computations.
Learning disentangled representations: One line of work focuses on learning disentangled represen- tations of inputs (Higgins et al. 2017a by Träuble et al. 2021 Bricken et al. 2023a),, either to improve learning, generalization, or interpretability. Disentangled representations can improve results in some cases (Van Steenkiste et al.) ) but may not always [Montero et al. 2020 2022); again, because the relationship between representation and computation nontrivial. However, these discntangling objectives likely would
shift the feature-level biases we consider.
Challenges of interpretability: Various priors works have noted challenges to interpretability methods
(Adebayo et al., 2018 Ghorbani et al.] 2019 Jain & Wallace 2019 Wiegreffe & Pinter 2019 Adebayo et al.,
2021 Bolukbasi et al. 2021 Zhou et al. 2022 Friedman et al. 2023a b Wen et al.) 2023 often suggesting that saliency or activations can be misleading about what a model is "doing" computationally Our results highlight similar challenges, that perhaps underpin some prior observations. For example, Friedman et al. (2023a) noted that simplifying model internal activations using techniques like PCA could yield interpreta- tions that are less faithful on more challenging test. distributions; as we noted above ($6.2). our results may point to a possible explanation. More generally, many interpretability methods are implicitly predicated the idea that variance in the neural representations effectively conveys importance; our results imply that these methods may be similarly biased.
18

---

Published in Transactions on Machine Learning Research (9/2024)
Analyzing and interpreting representations on algorithmic tasks: In order to circumvent the chal- lenges of interpretability in complex settings, a variety of prior works have analyzed the representations and circuits of models on controlled, algorithmic settings where the computational goal of the model is clearly specified. For example, these tasks include modular arithmetic [Nanda et al., 2023; Liu et al.) 2022 Zhong
et al. 2024 and Dyck languages (Hewill et al., 2020 Yao el al. [021] Murly el al. 2023 Friedman et al.
2023aJ. These algorithmic tasks reveal rich learning dynamics in which more complex, generalizing features are often learned later in a "grokking" stage (Power al. 2022) likely because the complex features that generalize take longer to learn (Xu et al. 2023). However, like the above works representational biases, most of these works have studied the case where features are redundantly predictive of single task to dif- fering degrees, often have confounds of complexity with utility, and have focused on a small range of tasks. Our work complements these by performing controlled studies of representation biases across a wider range of tasks and architectures, in cases when the features being represented are independently useful rather than redundant or confounded- correspondingly at the cost of performing less in-depth analysis of the circuitry for solving any particular problem than in prior works.
Neuroscience: Issues like those we describe here are not unfamiliar in neuroscience; it is commonly acknowl- odged that making inferences from representation to computation is challenging, even with interventional methods (e.g. Jonas & Kording 2017]. For example, it is well-known that neural signals can be strongly driven by "simple" or otherwise-salient variables like movement (Musall et al. 2019). which can interact with the representation of task-relevant variables in complex ways. More generally, representations can be biased towards certain features -ucc as biases towards horizontal and vertical orientations in primary visual cortex (e.g- Levick & Thibos 1982). These biases can yield suboptimal, biased performance in downstream behavioral tasks, that appear to be driven by learning via feature reweighting rather than feature tuning (Laanerad et al. 2024) which is also one explanation for why linear features are easier to learn than nonlin- ear ones in our setting. Other works have highlighted the potential for aliasing between systems computing different functions when using RSA over limited datasets (Dujmović et al., 2023), and conversely representa- tional divergences among different architectures computing similar things (Maheswaranathan et al. 2019). Complexities like these led Kriegeskorte Diedrichsen (2019) to highlight the need to "peel away layers that reflect developmental coincidences, random biological variation, and other epiphenomena without functional relevance, to arrive at the brain's representational core: the neural code used by the brain itself to mind and manipulate the world." We hope our analyses help to provide inspiration for underslanding some of these complexities and coincidences, and more pragmatically the issues that may arise in brain- brain-model comparisons.
8 Discussion
We have attempted to systematically characterize the internal representations of deep lcarning models that are trained to compute multiple features of their input. We found that these representations can be biased towards one feature or another by a variety of factors such as the relative complexities of the features, their positions and extents, their prevalences, and their learning order. Some of these biases interact with architecture and optimizer in complex ways, while others are relatively consistent.
We believe characterizing these biases is valuable for multiple reasons. First, there is inherent scientific value in exploring the complex phenomena of gradient-based representation learning that are not, to the best of our knowledge, fully understood. Some of our observations echo the previously-observed simplicity bias of deep learning architectures, but at a representational rather than behavioral level. These representat ional simplicity biases appear to be partly driven be learning order, and partly by the fact that there are inherently more ways to represent a more nonlinear feature. Second, our findings have important practical implications for interpretability and neuroscience methods that rely on analyzing model representations.
Implications for interpretability: Machine-learning rescarchers often want to interpret model represen- lations to understand the models (Olah et al., 2018 Geiger el al., 2021 Olsson et al. 2022 Bills el al., 2023)-either for scientific reasons, or to attempt to assess their safety and reliability, or to improve model behavior (Zou et al., 2023 Muttenthaler et al.] 2024). However, many of these analyses implicitly or ex- plicitly prioritize features that carry high variance in the representations. This prioritization is explicit in the case of using methods like PCA to analyze model representations. Indeed, as we suggested above, our
19

---

Published in Transactions on Machine Learning Research (9/2024)
results might help explain the fact that simplifying models to interpret them, using methods like PCA, may not generalize as well out of distribution (Friedman et al., 2023a). However, variety of other analyses also implicitly prioritize high-variance components, and may therefore likewise be subject to some bias.
For example, several recent works have proposed to tackle superposition in model activations via sparse dictionary learning (Bricken et al., 2023b Cunningham et al., 2023). These methods use a sparse autoencoder (SAE) with a high-dimensional latent space and a sparsity penalty process the internal representations, which can decompose them to yield more interpretable features. However, because the autoencoder is trained to reconstruct the internal representations via. an E loss, the objective favors reconstructing high-variance features; thus, these interpretability methods will likewise potentially be biased towards particular features and may neglect others that nevertheless play an important role in a model's computations.
Implications for computational neuroscience: Our findings may also be relevant to computational neuroscience. Research this area has increasingly found correspondences between reprosentations in doep learning models trained for vision or language processing, and representations in corresponding brain regions in humans (Yamins et al., 2014 Yamins & DiCarlo 2016 Schrimpf et al. 2018 2021 Tuckute et al 2023; Hosseini et al., 2024). Yet, models have come to be trained on increasingly large datasets, the results have become perplexing. [oowwll et al. (e.g. 2023 note that seemingly-diverse classes of models often end up achieving similarly high degrees of ability to predict (regress) brain representations, even when their representational structure is different either under RSA or more conservative comparison metrics (Khosla & Williams 2023 - Our findings hint at several potential explanations of this phenomenon: if the models differ in representational biases (e.g., if some are more biased towards easy features than others), but ultimately compute the same functions, their representational similarity structure might be dissimilar for reasons that are essentially orthogonal to the computations they perform.
More speculatively, if the braán representations are also biased towards representing simpler features more strongly, then explaining most of the variance in brain representations might not be much of an achievement- the most complex, interesting signals might be in the low-variance components that both regressions and RSA will tend to neglect. However, neural coding in the brain is likely optimized via other learning rules, and for other factors (like energy efficiency; Laughlin| 2001) that may shift its biases.
In either case, representational biases in the systems we are analyzing may lead us to make biased inferences about their relationship at an algorithmic or computational level-either under- or over-estimating their similarity.
proposed three levels of analysis for a system (in neuroscience): the computational level (the problem a system is trying to solve), the algorithmic level (the approach to solving it), and the implementational level (how that approach is implemented on the hardware). Prior work has highlighted the complex relationships between these levels of analysis (Churchland & Sejnowski] 1990 Shea, 2018). Similarly, our results both illustrate the separation between these levels for example, the fact that there are many reprosentation structures that realize the computation of the feature SUM % 2-but also their complex interactions. For example, the representational biases shift which features a model will use for a downstream classification task, thus illustrating how representation structure can influence higher levels of abstraction like the function a model will learn to compute.
Levels of analysis: Marr
1982
It is natural to ask whether our terminology fits these levels of analysis appropriately. In our motivation, we described the different features as playing the same "computational" role, in the sense that the system is trained to output all of them. However, it would be reasonable to say that feature complexity, for example, changes the "computational" role that a feature plays in a model-it depends on where one draws the boundary between the computational and hlmc We have adopted a particular convention for clarity, but we do not think that adjusting this boundary and our terminology would fundamentally change our results or their implications.
Do interventional methods address these challenges? Sexton & Love (002)) likewise note that vari- ance in representations docs not always capture computational importance. The authors therefore propose more directly computational comparisons: replacing the representations in one system (a model) with repre- sentations regressed from another system (a brain). This method does indeed offer much stronger constraints
20

---

Published in Transactions on Machine Learning Research (9/2024)
the system that is intervened upon, which allows for novel insights. However, knowing whether model features in the neural data would require intervening on
has captured all the computationally-impotaa. the brain instead which is usually infeasible. Otherwise, the model might be missing some low-variance features that are nevertheless essential for downstream computations.
The work above is one instance of many interventional methods that have been applied to analyze the computations of deep learning models (Geiger et al., 2021; Olsson et al., 2022 Geiger et al. 2024 Nanda Conmy et al. 2023 Wu et al.] 2024; Shah et al.) 2024 and brains (Deisseroth 2015 Liu
et al.
2023
et al. interpretations of which features are playing a particular role in computation.
2012 Again, by intervening, these methods do yield stronger constraints, and allow stronger causal
However, in practice, the hypotheses lested via these more rigorous interventional met often originate from correlational analyses of representations (e.g. seeing that certain neurons are highly responsive to a feature) or prior knowledge about a task (c.g. computational hypotheses about how the problem should be solved). Our experiments suggest that the first class of approaches may be biased in the kinds of causal hypotheses they generate, e.g. towards hypotheses about simpler features. Similarly, algorithmically that some features should be used to solve task, and find representations that correlate with the feature and causally influence the models representations. However, in either case, it will be difficult to rule out the possibility that other subtle representational urat ures are influencing the system's 's computations in important ways. Even verifying causal sufficiency, as in $3.3 does not eliminate the possibility that some other subtle feature is playing a role in some cases, but is preempted (Mueller 2024 by our strong intervention on the simpler feature.
may believe
Furthermore, our analyses in Section is nonlinearly involved in the network's 's compulations, we need to account for all the ways that feature could be represented-ee even with prior knowledge about which features are relevant to a task, this will be challenging in the general case. Thus, while causal methods are critical to thoroughly testing hypotheses about representation and computation, we do not believe they fully resolve the challenges raised here.
3.3 suggest that to fully identify the contribution of feature that
Do our findings allow us to correct for representational biases? In the case of the simple MLP networks, were able to "cxplain"-ice design interventions that climinate- the representational biases we observe. Unfortunately, these interventions required fundamentally altering the training process (to train on only the hard feature at first) and also charact the representations in terms of all input patterns relevant to the hard feature. Each of these would be challenging or impossible to scale to analyzing a larger- scale system like a language model or a brain. Furthermore, they rely on knowing a priori exactly what features a system is computing, which is true by design in our work, but not more generally. Moreover, the complex interactions we observe between different features that are trained would make it exceedingly difficult to know what kind of biases expect in any system that computes complex distribution of tasks over rich imputs. Thus, while our experiments highlight the existence of these biases, it is not immediately clear how these observations can be directly used to improve existing representation analysos.
Biases do not mean representation analyses are hopeless: While some of this discussion may seem pessimistic, the fact that biases might impact our analyses do not mean that those analyses are useless. The features clearly are playing an important role in the models computations, they are just not the only features to do so. Thus, biases in the analyses do not invalidate their conclusions; instcad, the biases may limil those conclusions, and increase the challenge of gaining complete understanding of system.
8.1 Limitations & future directions
There are a mumber of limitations to our work that would be useful to address in the future. First, we focused on simple, synthetic datasets. While this choice allowed complete control over the tasks, the features, and their correlations, it is possible that biases would be different on more realistic datasets with larger sets of entangled ures. However, concurrent work from Fel et al. (2024) finds compatible results for larger models trained on ImageNet.
Moreover, we focused on classification because, again, made the computational solution more straightfor- ward to analyze. It. would be interesting to consider how these biases differ under other objectives it is
21

---

Published in Transactions on Machine Learning Research (9/2024)
possible that continuous functions yield qualitatively different feature learning than discrete classifications though an increasing amount of work has converged on some kind of (sequential, soft) classification, e.g. language modeling. However, a limitation of our tasks relative to language modeling is that the represen- tations we consider are not richly contextual. Transformers can learn to restructure feature representations according to the task context (Li McClelland] 2023); in future work it would therefore be important to study how representation biases interact with more richly contextual computation.
Morcover, the models we used were of the toy scale used in range of mechanistic interpretability work (e.g. Nanda et al., 2023 Yao et al. 2021). While we did not see dramatic effects of parameter count in the regimes we considered, we did observe somewhat larger biases in deeper models, and it is possible that qualitatively different biases would emerge in larger models; thus it would be useful to study these effects at scale in future work. Furthermore, we found that the biases we observed can be shifted in nontrivial ways by hyperparameter choices ($3.6). However, we did not observe these interactions in Transformers trained on more complex datasets. We therefore expoct that in standard training scttings, where models are trained on complex lasks without full convergence, the representational biases would be more similar to those observed in earlier stages of training. Nevertheless, future work should investigate these interactions.
Finally, our main analyses relied on linear regression. While these analyses can account for many forms of representation paatn-pppallllllyyy in $3.3 where the augmented all-input-pat terns space can account for any function of the hard imputs alone they are limited to information that is not linearly available from that augmented space. For example, if the hard features were represented in superposition (cf. Elhage et all 2022) with other noise inputs, a linear analysis from solely the hard feature space could not account for that. In general, it is difficult to completely test the possibility that some representation non-linearly encodes some information (if it were easy, then encryption would not work). However, the fact that we observe that the majority of the total variance in the system is explained by these linear analyses, and that they are sufficient for causally changing the behavior of the system in a targeted way ($6.1), supports the idea that linear analyses provide useful insight." | Finally, lincar analyses are empirically justifiable in the context of their frequent use in interpretability and model steering research (e.g. Liu et al., 2022: Turner et al. 2023 Oikarinen & Weng 2024), and more general arguments that information tends to be represented linearly [Park et al.] 2024). Nevertheless, more deeply exploring the space of nonlinear reprosentations within models remains important direction for future work.
9 Conclusions
We have trained deep learning models to compute multiple features of their inputs. We found that, even once all features have been learned, the models' representations are substantially biased towards or away from features by propertics like their complexity, the order in which they were Iearned, their prevalence in the dataset, or their position in the output sequence. These representational biases may relate to some of the implicit inductive biases of deep learning. More pragmatically, representational biases could pose challenges for interpreting learned representations, or comparing them between different systems in machine learning, cognitive science, and neuroscience.
Acknowledgements
We thank Roma Patel, Aaditya Singh, Spencer Frei, Dan Friedman, Lukas Muttenthaler, Asma Ghandehar- ioun, Ed Grefenstette, and the anonymous reviewers for comments and suggestions on carlier versions of the paper.
Note, however, that discrete features input and output can nevertheless be best served by more continuous internal representations. For example, the nonlinear hard feature
in the first experiments is naturally computed by first summing
binary values, giving sum between 0 and an intermediate step.
Thhough note that just observing the causal sufficiency of linear features does not rule the possibility that some nonlinear feature is playing a role that gets preempted by the intervention on the linear feature (ef. Mueller 2024
22

---

Published in Transactions on Machine Learning Research (9/2024)
References
Julius Adebayo, Justin Gilmer, Michael Muelly, Tan Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
Julius Adebayo, Michael Muelly, Harold Abelson, and Been Kim. Post hoc explanations may be ineffective for detecting unknown spurious correlation. In International conference on learning representations, 2021.
Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Newral Networks, 132:428 446, 2020.
Matteo Alleman, Jack W Lindsey, and Stefano Fusi. Task structure and nonlinearity jointly determine learned representational geometry. Proceedings of the International Conference on Learning Representations, 2024.
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Newral Information Processing Systems, 32, 2019.
Ben Baker, Benjamin Lansdell, and Konrad P Kording. Three aspects of representation in neuroscience.
Trends in cognilive sciences, 26(11):942 958, 2022.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspec- tives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1822 2013.
Ari S Benjamin, Ling-Qt Qi Zhang, Cheng Qiu, Alan A Stocker, and Konrad P Kording. Efficient neural codes naturally emerge through gradient descent learning. Nature Communicatiins,, 13(1))7972, 2022.
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, llya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. URL https://opp /openaiphl.....iccc... blob. core. windows. 2023). 2023.
http//ppppr///teeee html. (Date accessed: 14.05.
Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Viégas, and Martin Wat- tenberg. An interpretability illusion for bert. arXiv preprint arXiv:2144.07 07.4443 2021.
James Bradbury, Roy Frostig, Peter Hawkins, Matthow James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adamn Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: compos- able transformations of Python+NumPy programs, 2018. URL http://githu/ wtwcchuu..ommeeeeeeeeeeeeeeeeeee goo//ooge//jax
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfiel-Doddss Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monose- manticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023a. https://rraastoreerrirnniispppp
/tt////oommiiiieeaaaaiaio wet.tttieeeetttteeeeeee..e. index.html.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, pp. 2, 2023b.
Rosa Cao. Putting representations to use. Synthese, 200(2):151, 2022.
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in trans- formers. Advances in Neural Information Processing Systems, 35:18878-1889) 2022.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hicu Pham, Xuanyi Dong. Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in Newral Information Processing Systems, 36, 2024.
Patricia Smith Churchland and Terrence Sejnowski. Neural representation and neural computation. Philo- sophical Perspectáves, 4:343 382, 1990.
23

---

Published in Transactions on Machine Learning Research (9/2024)
Arthur Conmy, Augustine N Mavor-Parte Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. arXie preprinl orXim:2304.144: 149977 2023.
Colin Conwell, Jacob Prince, Kendrick Kay, George A Alvarez, and Talia Konkle. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? BioRrio, pp. 2022-03, 03, 2023.
Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Are neural nets modular? inspecting their functionality through differentiable weight masks. In International Conference on Learning Representa- tions, 2021.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.0860 2023.
Ishita Dasgupta, Erin Grant, and Tom Griffiths. Distinguishing rule and exemplar- baased based generalization in learning systems. In International Conference Machine Learning, pp. 4816-4830. PMLR, 2022.
Karl Deisscroth. Optogenctics: 10 years of microbial opsins in neuroscience. Nature neuroscience, 18(9): 1213 1225, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 TEFE conference on computer vision and pottern recognition, pp. 248-255, Icce, 2009.
Marin Dujmovié, Jeffrey Bowers, Federico Adolfi, and Gaurav Malhotra. Obstacles to inferring mechanistic similarity using representational similarity analysis. RioRaie, 2023. doi: 10.1101 22222.04.05.871335 URL https://www biorxiv wwwoorii..rg///oooo.. org/ otg///otteet/errr//////////////0
04. 487133
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hotfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint ar Xix::22009. 10652, 2022.
Thomas Fel, Louis Bethune, Andrew Kyle Lampinen, Thomas Serre, and Katherine Hermann. Understanding visual feature reliance through the lens of complexity. arXiv preprint
2024.
Dan Friedman, Andrew Lampinen, Lucas Dixon, Danqi Chen, and Asma Ghandeharioum. Interpretability illusions in the gencralization of simplified models. arXiv preprint arXi::2312.0300 2023a.
Dan Friedman, Andrew Kyle Lampinen, Lucas Dixon, Danqi Chen, and Asma Ghandeharioun. Comparing iepresentational and functional similarity in small transformer language models. In UniReps: the First. Workshop on Unifying Representations in Neural Models, 2023b.
Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, volume 34, pp.
Curran Associates, Inc., 2021.
Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Teard, and Noah Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning, pp. 160-187. PMLR, 2024.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-traine enns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint Xrv: 122.1221,, 2018.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaclis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelli- gence, 2(11):665 673, 2020.
24

---

Published in Transactions on Machine Learning Research (9/2024)
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In Pro-
ceedings of the AAAI conference artificial intelligence, volume 33, pp. 3681-3688 2019.
Mark Harrower and Cynthia A Brewer. Colorbrewer. org: an online tool for selecting colour schemes for maps. The Cartographic Journal, 40(1):27-33) 37, 2003.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Jonathan Heck, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2023. URL http://github. com/google/flax
Katherine Hermann and Andrew Lampinen. What shapes feature representalions? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems, 33:9995 10006, 2020.
Katherine Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convo- lutional neural networks. Advances in Neural Information Processing Systems, 33:19000-19015, 2020.
Katherine L Hermann, Hossein Mobahi, Thomas Fel, and Michael C Mozer. On the foundations of shortcut learning, Xiv preprint arXiv:2310. 16228, 2023.
John Hewitt, Michael Hahn, Surya Ganguli, Percy Liang, and Christopher D Manning. Runs can generate bounded hierarchical languages with optimal memory. arXie prepriul urXiv:2010.077 0.05155, 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR (Poster), 3, 2017a
Trina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher P Burgess, Matko Bosnjak, Murray Shanahan, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner. Scan: Learning hierarchical compositional visual concepts. arXiv preprint arXiv: 177. 03339, 2017b.
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do compressed deep neural networks forget? arXie preprint arXiv: 191.00248, 2019.
Eghbal A Hosseini, Martin Schrimpf, Yian Zhang, Samuel Bowman, Noga Zaslavsky, and Evelina Fedorenko. Artificial neural network language models predict human brain responses to language even after a devel- opmentally realistic amount of training. Neurobiology of Language, pp. 1 21, 2024.
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. arXiv preprint arXiv:2103.1044 2021.
Sarthak Jain and Byron C Wallace. Attention is not explanation. arXiv preprint arXiv:1902.10188 2019.
Eric Jonas and Konrad Paul Kording, Could neuroscientist understand microprocessor? PLoS compu- tational biology, 13(1):e1005268, 2017.
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. SGD on neural networks learns functions of increasing complexity. Advances in newral information processing systems, 32, 2019.
Meenakshi Khosla and Alex H Williams. Soft matching distance: metric on neural representations that captures singlo-neuron tuning. arXiv preprint XXi::2311.00466 2023.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arxiv preprint arXiv:1412.6900 2014.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. Tn International conference on machine learning, pp. 3519-3529. PMLR, 2019.
25

---

Published in Transactions on Machine Learning Research (9/2024)
Nikolaus Kriegeskorte and Jörn Diedrichsen. Pecling the onion of brain representations. Annual review of neuroscience, 42:407-432, 2019.
Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis the branches of systems neuroscience. Frontiers in systems neuroscience, pp- 4, 2008.
Pooya Laamerad, Asmara Awada, Christopher C Pack, and Shahab Bakhtiari. Asymmetric stimulus repre- sentations bias visual perceptual learning. Journal of Vision 24(1):10 10, 2024.
Yair Lakretz, Dieuwke Hupkes, Alessandra Vergallito, Marco Marelli, Marco Baroni, and Stanislas Dehaene. Mechanisms for handling nested dependencies in neural-network language models and humans. Cognition. 213:104699, 2021.
Andrew K Lampinen and Surya Ganguli, An analytic theory of generalization dynamics and transfer learning in deep linear networks. arXiv preprinl Xrxi 1809. 10374, 2018.
Simon B Laughlin. Energy as a constraint on the coding and processing of sensory information. Current opinion in neurobiology, 11(4):475 480, 2001.
WR. Levick and LN Thibos. Analysis of orientation bias in cat retina. The Journal of physiology, 329:243, 1982.
Yuxuan Li and James McClelland. Representations and computations in transformers that support gener- alization on structured tasks. Transactions on Machine Learning Research, 2023.
Xu Liu, Steve Ramirez, Potti T Pang, Corey Purycar, Arvind Govindarajan, Karl Deisscroth, and Susumu Tonegawa. Optogenetic stimulation of hippocampal engram activates fear memory recall. Nature, 484 (7394):381-38) 385, 2012.
Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems, 35:34651 34663, 2022.
Tlya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXie preprint orxi 1ei::117111000000 711.005101 2017.
Miao Lu, Beining Wu, Xiaodong Yang, and Difan Zou. Benign oscillation of stochastic gradient descent with large learning rate. In The Twelfth International Conference on Learning Representations 2023.
Ekdeep Singh Lubana, Eric J Bigelow, Robert P Dick, David Krueger, and Hidenori Tanaka. Mechanistic mode connectivity. In International Conference on Machine Learning, pp. 22965 23004. PMLR, 2023.
Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Universal- ity and individuality in neural dynamics across large populations recurrent networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and Garnett (eds.), Advances in Neural Infor- malion Processing Systems, volume 32, 2019. URL httpps:/proceedinigg neurips. wwttrii..ccppppeeee files/ fawp://oog//illllf./////66666610 www.:eaaeeee.eecccccaaaaaaeeeeee
pdf
David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 1982.
R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. Xiv preprint arXiv:1902.01000 2019.
R Thomas McCoy, Robert Frank, and Tal Linzen. Does syntax need to grow trees? sources of hierarchi- cal inductive bias in sequence-to-ssqqeece networks. Transactions of the Association for Computational Linguistics, 8:125-140, 2020.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 1177-1597777 2022.
26

---

Published in Transactions on Machine Learning Research (9/2024)
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Circuit component reuse across tasks in transformer language models. In The Twelith International Conference on Learning Representations, 2023.
Milton Montero, Jeffrey Bowers, Rui Ponte Costa, Casimir Ludwig, and Gaurav Malhotra. Lost in latent space: Examining failures of disentanglod models at combinatorial generalisation. Advances in Neural Information Processing Systems, 15:10136-10149 2022.
Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, and JefIrey Bowers. The role of disentanglement in generalisation. In International Conference on Learning Representations, 2020.
Depen Morwani, Benjamin L Edelman, Costin-Andrei Oncescu, Rosie Zhao, and Sham Kakade. Feature emergence via margin maximization: case studies in algebraic tasks. arXiv preprint arXiv:2811.07568, 2023.
Aaron Mueller. Missed causes and ambiguous effects: Counterfactuals pose challenges for interpreting neural networks. arviv preprint arXiv:2407.04000 2024.
Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D Manning. Grokking of hicrarchical structure in vanilla transformers. aixie preprinl urXiv:2305.1877 2023.
Simon Musall, Matthew Kaufman, Ashley L Juavinett, Steven Gluf, and Anne K Churchland. Single- trial neural dynamics are dominated by richly varied movements. Nature neuroscience, 22(10):1677-1666 2019.
Lukas Muttenthaler, Lorenz Linhardt, Jonas Dippel, Robert A Vandermeulen, Katherine Hermann, An- drew Lampinen, and Simon Korublith. Improving neural network representations using human similarity judgments. Advances in Neural Information Processing Systems, 36, 2024.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interprotability. arXiv preprint arXiv:2301.05211 2023.
Tuomas Oikarinen and Tsui-Wei Weng. Linear explanations for individual neurons. arXiv preprint arXiv:2405.06855, 2024.
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 3(3):c10, 2018.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.1 11895, 2022.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Forty-first International Conference on Machine Learning, 2024.
Fabian Pedregosa, Gaäl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2882 2011.
Steven T Piantadosi. Zipf's word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21:1112-1130, 2014.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization
beyond overfitting on small algorithmic datasets. arXio preprint or Xri:::2001.00 1001177, 2022.
GuanWen Qiu, Da Kuang, and Surbhi Goel. Complexity matters: Dynamics of feature learning in the presence of spurious correlations. arXiv preprint +=Xii:2403.03375 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. preprint, 188):9, 2019.
27

---

Published in Transactions on Machine Learning Research (9/2024)
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International conference machine learning, pp. 5301 5310. PMLR, 2019.
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):124020, 2019.
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRaiu, pp. 407007, 2018.
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. The neural architecture of languago: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45):e2155644115 2021.
Nicholas J Sexton and Bradley C Love. Reassessing hierarchical correspondences between brain and deep networks through direct interface. Science advances, 8(28)xaabm2219 2022.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitsalls of simplicity bias in neural networks. Advances in Newral Information Processing Systems, 33:9573 9585, 2020.
Harshay Shah, Andrew lyyss, and Aleksander Madry. Decomposing and editing predictions by modeling model computation. arXie preprint NKV:24444.1133, 2024.
Nicholas Shea. Representation in cognitive science. Oxford University Press, 2018.
Tha Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C Love, Erin Grant, Jascha Achterberg, Joshua B Tenenbaum, et al. Getting aligned representational aligument. Xiv preprinl arXiv:2310. 13018, 2023.
Remi Tachet, Mohammad Pezeshki, Samira Shabanian, Aaron Courville, and Yoshua Bengio. On the learning dynamics of deep neural networks. arXiv preprint arXiv:1809. 06848, 2018.
Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bern- hard Schölkopf, and Stefan Bauer. On disentangled representations learned from correlated data. In International conference on machine learning, pp. 10401 10412. PMLR, 2021.
Greta Tuckute, Jenelle Feather, Dana Boebinger, and Josh H McDermott. Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions. Plos Biology, (1(1)))e30)2366, 2023.
Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308. 2023.
Guillermo Valle-Perez, Chico Q Camargo, and Ard A Louis. Deep learning generalizes because the parameter- function map is biased towards simple functions. arXiv preprint arXio:1805.0888 08522, 2018.
Sjoerd Van Steenkiste, Francesco Locatello, Jürgen Schmidluber, and Olivier Bachem. Are disentangled representations helpful for abstract visual reasoning? Advances in neural information processing systems, 32, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Kaiyue Wen, Yuchen Li, Bingbin Liu, and Andrej Risteski. (un) interpretability of transformers: a case study with dyck grammars. ICML Workshop DeplooahheGenerrliieAA 2023.
28

---

Published in Transactions on Machine Learning Research (9/2024)
Hadley Wickham, Mara Averick, Jennifer Bryan, Winston Chang, Lucy D D'Agostino McGowan, Romain Frangois, Garrett Grolemund, Alex Hayes, Lionel Henry, Jim Hester, et al. Welcome to the tidyverse. Journal of open source software, 4(43):1686, 2019.
Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. arXiv preprint 1rx::19905 0tte:1908.4400 04,66, 2019.
Zhengxuan Wu, Atticus Geiger, Thomas Teard, Christopher Potts, and Noah Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca. Advances in Neural Information Processing Systems, 36, 2024.
Zhiwei Xu, Yutong Wang, Spencer Frei, Gal Vardi, and Wei Hu. Benign overfitting and grokking in relu networks for xor cluster data. In The Twelfth International Conference on Learning Represenlations, 2023.
Yihao Xue, Siddharth Joshi, Eric Gan, Pin-Yu Yu Chen, and Baharan Mirzasoleiman. Which features are learnt by contrastive learning? On the role of simplicity bias in class collapse and feature suppression. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 38938 38938-38970. PMLR, 23-29 Jul 2023. URL https://
proceedings .mr.iigs...r.....mm....m.....m.. mlr. wttpppeeeeeeuueeeeeeeee http//xee2dddd..
Daniel LK Yamins and James J DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3):356 365, 2016.
Daniel LK Yamins, Ha Hong, Charles F Cadien, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optinizee hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111((3):8619-8624 2014.
Shunyu Yao, Binghui Peng, Christos Papadimitrion, and Karthik Narasimhan. Self-attention networks can process bounded hierarchical languages. arXiv preprint arXiv:2105, 11115, 2021.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in Neural Information Processing Systems, 36, 2024.
Kaitlyn Zhou, Kawin Ethayarajh, Dallas Card, and Dan Jurafsky. Problems with cosine as mcasure of embedding similarity for high frequency words. arXiv preprial ar Xri:22205.0000 2022.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXi:22310.01 2023.
29

---

Published in Transactions on Machine Learning Research (9/2024)
A Supplemental methods
All models were trained in JAX (Bradbury et al. 2018 using the flax (Heck et al. 2023 library; networks were generally constructed from the corresponding flax examples. All representation regressions during Training were fit. using scikit-learn (Pedregosa el al. 2011). Plots were created with the Tidyverse (Wickham) et al. 2019 Plot color palettes are loosely based on Harrower Brewer (003)).
Demonstration colab: We provide a simple colab demonstrating the basic feature complexity result http:///f.....ttooooooo///aasaee
A.1 Hyperparameters & architectural details
In Table we report hyperparameter values for all main experiments. Note that different choices for some hyperparameters for the MLP experiments are systenatically tested in Scction [.7] for example a larger set of optimizers. Hyperparameter values generally chosen from prior work, from the flax (Heek et. al. 2023 examples; they were changed if the models exhibited training instability failed to learn.
Model
Experiment
Optimizer SGD
LR.
Train set size
Batch size
Max epochs
MLP
Basic
1-10-3
4096
Full
5 10
Prevalence
32768
512
Transformer
Letter sequence
AdamW
le-5
16384
512
10a
Language
32768
Dyck
3e-5
131072
10a
CNN
Basic
Adam
3e-4
8192
256
10a
Relational
3e-5
32768
ResNet
Basic
3e-4
8192
Relational
le-4
32768
Table 1: Hyperparameters for main experiments. above.)
(Where value is omitted, it is the same the value
Additional architectural details are provided below for each model.
The network contained four hidden layers, with sizes [256, 128, 64, 64), followed by a final output layer. Initialization varianco-scaling with scale 1. The nonlincarity the hidden layers lcaky rectifier with negative slope 0.01. The model was trained using sigmoid cross-entropy loss.
MLP:
The network was an encoder-decoder transformer (Vaswani et al. 2017) as implemented in the flax examples, with 4 layers in each component, 4 heads per layer, an embedding size of 128 (and thus Q/K/V dimension 32), and an MLP dimension of 512. The implementation uses the layernorm on the branch input rather than the residual connection, following Radford al. (2019). The model uses standard sinusoidal position embeddingg. The model was trained using softmax cross-entropy loss, as in standard language modeling.
Transformer:
CNN:
The architecture was a standard CNN that alternates convolution with mean pooling. Convolution
filter sizes were (6, 4, 3, 2), channels were (128, 128, 128, 256, 256), strides were (3, 1, 1, 2, 1), and pooling sizes and strides were (3, ; * 2, *). All convolutions and the MLP hidden layer used rectifier nonlinearities. The network was trained using a softmax cross-entropy loss for multi-way classification.
ResNet: The architecture was ResNet-18 implemented in the flax examples, followed by a two-layer MT.P classifier (to match the CNN more closely), with 512 units at the hidden layer.
30

---

Published in Transactions on Machine Learning Research (9/2024)
A.2 Datasets
In this section we provide further details on the datasets. Except where otherwise noted, all datasets were constructed such that different features were statistically independent, and feature values were balanced within each dataset.
A.2.1 Boolean
For the Boolean experiments, the inputs were vectors with 64 boolean values, and the outputs were vectors with 2 2 boolcan values. Some number of the inputs were devoted to each of the features, while the rest were set randomly. The inputs that were relevant for one feature did not overlap with those relevant for another feature. For our main experiments we redundantly encoded the linear features across four input units, to match the number used for the hardest features; however, Fig. 29 we show this redundant encoding does not substantially affect the results.
A.2.2 Language
All language problems were posed as sequenco-fo-ssqquence tasks, where an input sequence was provided and the model was asked to output a sequence of classifications of features of that input. As the position of Teatures the outpul sequence affects their variance, the output sequence order was randomized for each training run.
The datasets and features similar to the boolean tasks, except for larger input cabulary, and the sequential structure of the inputs. The input sentences were constructed from a vocabulary of 9 letters (A-I). There were imput letters provided per feature, with the relevant letters appearing at the beginning of that chunk, thus there was at least one irrelevant (noise) letter at the end of cach fcature's relevant inputs. Since features were used, input sequences were length x 5 = 40. The input letters were initially sampled randomly, but then labels were assigned each input to balance each feature (indepen- dently) across the dataset, and the relevant input letters were then resampled until the desired label was reached (rejection sampling). The features used were applied to chunks of 1-4 letters, with the feature being either exact match to letter string, all-but-one letter matching (XOR-like), or the sum of the letter matches mod 2.
Letter strings:
Naturalistic language: The sentences used for this work were sampled from a simple grammar that generates grammatical English sentences, but within a relatively constrained range. The grammar focuses on two types of sentence structurcs- with cither nested or succossive noun-verb dependencies- which are loosely inspired by prior work on grammar processing neural models (Lakretz et al. (021). The primitives of the grammar are provided below; the items with question mark were introduced with some probability (0.5 for the adjectives, 0.33 for all others). The valence and tense (present or past) of the sentence were set to be the same across all of the adjectives/verbee verbs in the sentence.
BENTESCE BENTESCE_STRUCTURE STRUCTURES {
"sageantial" "s "sha, (MT) (NI) (17) (17. 'seated': (At (A7) (82) tastushe, (AT) (17)
('evedt, feoplf,
"vondesfll"",
('bee reaceol, steerilles, feor "aesteel') D'coopaable', "bladd, 'gmerie', Itolerablee1,
TARG TANG AGRTTS =
("xingalar": "pareon", 'plosal's 'paopla
("asagolae:: "chsse, 'ploral"; "chs1dran "),
Osinggler's redule,, "plurol fedolle'3,
O'ingeler's raogl, "plceally "eggs)
("singeinr's "borse", "piural"; ")orses",, ("singeler's "ovi', "piarel's "oxia"),
('singeler's "zobot", plurell:; "robois"),
LAN0_08JB0T (0..000//BTEEREE
"ceax "it 'onngggstooee' Opreeentt.a disgelaris "cookel, "prosent. fy teese!, "past!; "cocksd"), objecti''s [iiiiee' "pizxus"), 1), (Caudirck' "sendviches") ('potato', "potatcae (]],
catt of
conjagatiozee' Uprosons, wixgelar's Peste', "presons.plore" leat part: lsta
31

---

Published in Transactions on Machine Learning Research (9/2024)
lobjacta's [C'iizza', 'pizzas"), 1), ('assdrick', 'sandetchez '), (Ipotato', )potatons)), ('banara', "banane"")
'conjgaatioaal: ('peeontt visgeler's
"powwi.pluus"" "raa",
"pastly "rax"?, 'objacta's [O'race', 'racez"), ), ("marathcn', "sarations
turiteli {
'onnjigationa: ( ("pronoss.tnnelarr Atngelari: "uritan', "protent.piaratt" Torseal, "partt; "srate") (),
"objuciars bbokk'',, , "booku"), , Eipuee,, "pocas"), Dlssuy', "esoagu')1
for -r de LAAE_,,EECT_, VEREESTTVEREE.. VERES (): vf uusssjugattion pppaatt
scsjsspptio "pputtssarp ["oojuuaticcss (ppaat))
"see's Uppessunt "tpessouissnngulaa""
singularhy "sees", "presut..lluaa"
"past"s 'sae?3,
Speesons.stngelarr "harr', "presons.plure" hear "pasti; 'seard"),
"Asow': ('presens_aanne seieenn.iinnsaae... "kzove', "preons.plurall Qaaoe'
"pastt; "enae"),
"EbSEE': atracian..taggiarr. "chinke', "prasnt.giaraa"' reaser'
"past"; "ezosgee")
Peay'
purresess.singulie "pstt" rsald"). "),
"says", 'porsent, laay's
is LASG RELATIOL PERSS. :tvma)) _Upest,iinppler" pop... <V[ppsi
("gresst singular"' "prezent.pierss""
"pass_pluzai's
EOCNTION, GLANSES "byukkey' for p'pukk,, 'school', 'bouse, rullicer, "laket "Iorest Te foz LCENES foz y.
From these sentences, we trained the model to output a variety of features: whether the sentence was written in present or past tense, the valence, the structure type (nested or sequential), whether each of the nouns was singular or plural, and their identities. The latter features require parsing the sentence to determine its structure, and to identify which noun appears where while ignoring the other noun (and any that occur in distractor location clauses).
For the Dyck language experiments we used sentences sampled from Dyck-(20,10) that is, 20 bracket types and a max depth of 10. We also enforced a length limit of 64 tokens. We sampled each sequence one token at a time: first we selected whether it would be an open or close bracket. the type was unconstrained, we set a probability of 70% that it would be the same type as the previous bracket; e.g., that an open bracket would be followed by another open bracket. This biased sampling yields more interesting structures. Tf the token was at 0 or max depth, or close to the max length, the type constrained appropriately to fit the language constraints.
Dyck:
If the token was selected to be an open bracket the bracket class was randomly sampled from the set of 20 bracket classes. If it was a close bracket, it was chosen to match the corresponding opening bracket.
After the first token, when the depth reached 0, the sentence was terminated with 50% probability; otherwise, new opening bracket was sampled. Thus, this dataset contains sentences of varying lengths.
We trained the model to compute 5 features about the sentence: the class of the first token, the class of the first token that appeared at the max depth of the sentence, the maximum depth itself, the number of root nodes (i.e. opening brackets at depth 0), and the max branching factor (ie. the max number of opening brackets contained within another pair, or the number of root nodes if it was larger). Note that unlike the naturalistic language lasks above, the sentence length is correlated with various uratures such as the number of root nodes. Thus we control for length our analyses.
A.2.3 Vision
show examples of each of the 10 colors, textures, and shapes that used in the vision experiments. Note that, while our experiments were inspired by Hermann & Lampinen (2020) we do not use precisely the same feature sets that they did; our textures and shapes are somewhat simpler and the fact that we do not randomly rotate them likely makes the classification problem slightly easier. However, we do not expect this to substantially alter the results.
In Fig.
In Fig. 18 we show example stimuli for the basic and relational tasks nosks-note that in the latter case, the basic features are reported for the object in the left "column" of the image, and the relational features are
32

---

Published in Transactions on Machine Learning Research (9/2024)
reported for the objects the right. Note that for both basic and relational tasks, all objects randomly offset within their allotted space.
X
OTTHU
Figure 17: Examples of the 10 colors, textures, and shapes used in the vision experiments.
H
(a) Basic
(b) Basic 2
(c) Relational
(d) Relational 2
Figure 18: Example stimuli for the vision datasets with single objects (a-b) or for the tasks including relational features among multiple objects (c-d).
B
Supplemental analyses
B.1
MLP loss curves
In Fig. 19
we show the feature-wise loss curves corresponding to the base plot in Fig. 2 the losses for both features are trending towards zero.
33

---

Published in Transactions on Machine Learning Research (9/2024)
Representation variance explained final)
Tost
Test accuracy (%)
100
feature
75
easy hard
Value
50
25
10000
20000
30000
10000
20000
30000
10000
20000
30000
Training steps
Figure 19: Test losses by feature, well accuracics and reprosentation variance explained, for the models plotted in Fig. Both features' losses approach zero they are learned.
B.2 Representation variance explained curves with and without normalization
In the learning curves we generally plot variance explained normalized by total variance at the end of training. In Fig. 20 we compare this to the values normalized by current total variance. While the qualitative biases are similar, the alternative normalization makes the earlier learning dynamics more visible. Furthermore, also shows that initialization, the easy feature explains more variance than the hard feature but all hard input patterns explain even more.
R Representation varlance explained (% of current total variance)
R Representation variance explained (% of final total variance)
feature
Value
40
easy hard hard all input patterns
20
0
10000
20000
30000
10000
20000
30000
Training steps
Figure 20: Comparing variance explained as proportion of total variance at the end of training (right, plotted elsewhere in this paper) vs. normalized by current total variance; biases are qualitatively similar, but the alternative plot makes the initial learning dynamics clearer.
B.3
MLP learning curves by training order
In Fig. 21 we show the learning curves for easy and hard features, across different training orders. When the hard feature pretrained, it occupies more variance than the easy feature initially, but the easy feature quickly overtakes it once it starts to be learned. However, when considering all input patterns associated with the hard feature, the effect of pretraining is much clearer; and the curve more clearly shows the inflation in the later phase of learning that is observed with the easy feature in the other cases.
34

---

Published in Transactions on Machine Learning Research (9/2024)
Representation variance explained (% of final)
Test accuracy (%)
100
Simultaneous
feature
50
easy
hard
25
hard input
patterns
100
75
Pretrain easy
Value
50
25
100
75
Pretrain hard
50
25
2
3
1
2
3
Training steps (1e5)
Figure 21: Learning curves for MLPs learning casy and hard tasks, depending on training order simultaneous training, pretraining task. Left panels show test. accuracy, right show variance explained in the penultimate representations. (Lines means across 10 seeds, areas are 95%-Cls).
B.4 Per-unit representation variance analysis
In this section we present analyses of how each of the 64 units in the penultimate layer of the model represents each feature for a single run of the basic model trained on the linear and sum mod 2 tasks. In Fig. 22 we show snapshots of the variance explained by each feature for each unit, at various salient stages in training. In Fig. 23 we show individual learning curves for all the units. The qualitative patterns are similar to the aggregate case, but there are interesting individual effects observe. Most notably, the units that end up carrying the most variance about the difficult feature also those that initially carry the most variance about the easy feature, and the variance they carry about the easy feature is inflated as the hard feature is learned. Thus, most of the individual units also continue to carry more variance due to the casy feature. Only a subset of the initially-easy-seles contributes to the sparsity difference observed below). However, a few neurons that initially carried at least as much signal about the hard inputs as easy ones do tend become somewhat more selective for the hard input features as the harder task learned. Furthermore, many of the units carry non-trivial signal about other imputs (or nonlinear patterns of the feature-relevant inputs) that is also amplified as the other features
neurons acquire some hard-relevant, signal later (which likely
35

---

Published in Transactions on Machine Learning Research (9/2024)
are learned - either because those inputs are weakly correlated with task feature within the dataset by chance, or just due to the learning dynamics.
95% 1500)
100% (1.000)
Hard begins leamnin (72020)
0.0.00
2.06
0,002
Unit vertarce atrixted to festure
0.021
0,0225
0:07
4.
Hard best 55% (42250)
Hard 100% (43300)
Final 195000:
feature
Fard all rouut catlirrs routs
10
70
Unit Isorted final total variance)
Figure 22: The distribution of variance explained by each feature across the 64 units of an MLP at various stages of training. (The ordering of units consistent across the plots, and determined by the final plot.)
36

---

Published in Transactions on Machine Learning Research (9/2024)
2000
2303
2000
1300
47000
4750
79880
4000
1910
1505
3000
1900
7000
1305
2005
3000
450
2008
4000
2006
4000
6000
2000
0000
1000
3000
4300
Figure 23: Raw representation variance plotted across learning for each of the 64 units in the penultimate layer of an MLP. Note that this figure plots raw variance explained, without normalizing by the total variance of the unit, to facilitate comparing across units. Note also that the vertical axis log-scaled; where the variance explained the test set was is omitted from the plot.
37

---

Published in Transactions on Machine Learning Research (9/2024)
B.5
Feature complexity also biases sparsity of representations
Simultaneous
Pretra in easy
Pretra in hard
0.25
feature
casy
0.15
0.10
(OROORIWX) OR(22))
AND(KXXY22
XORIANDEN.S ANDD
ANDECOROWA), XOR(Y (1)
XOROWX!
SUMIWXX Y.2152
XORIORIWXD. OR(ZZ)
ANDWAX
XOR(ANDIWX X), (NDIIZI
ANDOOORIWK XI, XOROY 22)
XOR(WC))
SUMIWX Y.ZKZ
XOROROWX) OR(YZ)
ANDIKX.
XORIANDIW,N AND(Y.2)
ANDOXOR(W.X) XOR(Y
XOR(WX)
SUMIWXX. 22122
Hard feature
Figure 24: Harder features also yield sparser representations. (Lines are means across 10 seeds, errorbars are 95%-CIs)
We also find that more complex features tend to be represented more sparsely in the models representations (Fig. (4). To measure sparsity, we evaluate the hidden representations, and then standardize element-wise so that each unit has variance 1. We then fit logistic regression decoding the feature, and evaluate the entropy of the sofumax of the absolute value of the regression coefficients. If the regression integrates information more uniformly across all the units, this entropy value will be higher, while if fewer units are contributing, the entropy value will be lower; thus, we subtract this measure from the maximum entropy (a uniform distribution) and divide by the maximum entropy to calculate a 0-1 sparsity score for each feature. By this measure, the harder features are noticeably sparser the models representations, even the case that they are pretrained.
B.6 More features of varying complexity
While our main MLP experiments only trained models to compute two features, deep networks trained on real tasks will compute many more. To move slightly closer this regime, we trained networks predict four features (N.B. also more features in our language and vision experiments). which we randomly sampled from set of five features of varying complexity. We trained the models to predict all features simultancously, and as above evaluated test accuracy and variance explained by each feature.
We show the results in Fig. 25 In general, the patterns are roughly what would be expected from the two-feature experiments above: adding more simple features to the dataset tends to suppress more complex features, while adding more complex features inflates simpler ones. Note that we cannot disentangle these effects fully here, but see the learning curves above Appx. B.3 for evidence that both inflation and suppression happen in the two feature case. Features of intermediate difficulty tend inflate simpler features but interfere with more complex ones.
38

---

Published in Transactions on Machine Learning Research (9/2024)
Linear
AND(AND.AND)
XOR
AND(XOR.XOR)
SUM %
100
90
(960 Aocunooie
70
Feature
evaluated
Value
50
Linear
AND(AND,AND; XOR
.ND SUM %
199 poupeeio
40
20
0
0 2
3
2
1
0
2
# features of other type dataset
Figure 25: Training MLPs on four features, sampled from a set of varying complexity. This figure shows the effect on lest accuracy (lop) and representation variance explained (boltom) of a certain type of feature (colors) from the number of features of another type (panels). For example, the bottom left panel shows the representation variance of cach feature type, as function of of how many of the other features in the dataset are linear. As the number of linear features in a dataset increases, the variance explained by other features decreases noticeably. Conversely, the presence of more hard features in the dataset tends to inflate the variance of all simpler features. Features of intermediate difficulty, such as XOR, tend to inflate simpler features, but suppress equally or more complex ones. (Note that due to small dataset size, the hardest feature is only generalized reliably when all other features are linear. Results from training on 128 datasets with randomly sampled sets of four features. Lines are averages, errorbars are 95%-Cls.)
B.7 The effects of MLP architecture and other hyperparameters
How specific are our findings to the particular MLP architecture we evaluated? In Fig. 26 we show the effect of htppt//wdth//
ratios (i.e. ratios of layer sizes) of the MLP on our findings; in short, deeper MLPs show somewhat larger gaps, width and aspect ratio appear to have little effect. Residual MLPs also show qualitatively similar biases (Fig. 27), in contrast to our finding that ResNets have strikingly different biases to CNNs ($I.).
Prior work has found that tanh and rectifier networks have qualitatively different representation-llannie properties (Alleman et al. 2024), thus we performed a similar comparison here (Fig. 28 Overall patterns are similar, but we do note a slightly smaller gap in general with tanh networks. This observation may fit with the results of Alleman et al., who found that tanh network representations were generally more driven by the output structure, while rectifier network representations were driven more by imput structure; that finding might suggest. a somewhat smaller gap for tanh networks in our setting, since the output structure is identical for both features, but the input structure differs.
In the main experiments encoded the easy feature redundantly across input units (to match the number of inputs used for the hard feature); in Fig. 29 we show that this redundant encoding of the easy fcature slightly increases the gap, but does not change the qualitative pattern of results.
In our main experiments, we used variance scaling initializers with an initialization scale of 1.0 for the network weights (that is, we initialized the weights to approximately preserve the variance of signals through the network at the beginning of training). However, we show in Fig. 30 that smaller initialization scales yield somewhat larger biases towards the easy features.
39

---

Published in Transactions on Machine Learning Research (9/2024)
Representation variance explained (% of final)
Test accuracy (%)
75
100 vvalue
feature
easy
50
hard
hard all input patterms
25
(128, 128)
(128, 128, 128, 126)
(256. 64)
a20 128, 128, 128, 128, 128, 128, 128)
(128, 64, 32. 32)
(256, 128, 64, 64)
(256. 256 128, 128. 64, 64. 64, 64)
(512, 256, 128, 128)
(128, 128)
(328. 128, 128, 128)
(128, 128. 128, 128, 128, 128, 128, 128)
(128, 64, 32, 32)
(256, 128, 64, 64)
(256, 256, 128, 128, 64, 64, 64. 64)
(512. 256, 128. 128)
(296,
Model hidden sizes
Exploring the effect of architecture the gap between easy (linear) and hard (sum % 2) features. All architectures generalize well. Layer width does not seem to have substantial impact the represcntation gap in most cases. However, deeper models tend to show larger gaps. (Bars are mcans across 10 seeds, errorbars are 9Is-Cls).
Figure 26:
Representation varlance expla ined (%)
Test accuracy (%)
100
feature
Value
easy
hard
input
patterns
25
(64, 64, 64, 64)
(128, 128, 128, 128)
(64, 64, 64, 64, 64, 64)
(128, 128, 128, 128. 128. 128, 128, 128)
(64, 64, 64, 64)
(128, 128, 128, 128)
(64, 64, 64, 64, 64, 64, 64)
(128. 128, 128, 128, 128, 128, 128, 128)
Model hidden sizes
Figure 27: MLPs with residual connections also show qualitatively similar patterns of bias. (Bars are means across 10 seeds, errorbars are ols)..s).
40

---

Published in Transactions on Machine Learning Research (9/2024)
Simultaneous
Pretrain easy
Pretrain hard
40
max Aquas
variance exploined (%)
20
feature
casy
hard
60
hard all Input
patterns
40
(ORIOR(WAH) ORT/Z2
ANDIWXX..22
XOR(ANDIWXX) ANDIY.2)
ARDEZORIWAL XORO 201
KORIBEA)
SUMON 1,2
ORRWE) oR(1.2))
ANDOR XY.2
KOR JANDENLL ANDIT.21)
ANDIXORNEE), XOR(TZI)
XORIWAA)
SUMIW Y,22262
Xor(OR(Wa)) ORT(Z1)
ANDIW.XY X.Y
XORIANDBE ANDIY 23
ARDIFOR(NA)) XOROY
KOR xy]
SUMIN 2.222
Hard feature
Figure 28: Effect of nonlinearity (rows) the results. Patterns qualitatively similar with either Leaky ReLU (as used in our other experiments) or tanh nonlinearities; however, tanh shows somewhat, smaller gaps in general. (Bars are means across 10 seeds, errorbars are 9ls).
60
Representaton variance
explained (%)
40
feature
easy
hard
20
8
Number of redundant easy inputs
Figure 29: Redundantly encoding the easy feature across 4 or 8 input units results in slightly stronger gaps between easy and hard features, but qualitative results are similar with only a single easy input. (Bars are means across 5 seeds, errorbars are CIs).
B.7.1 Dropout and optimizer interact for MLP architectures
The results we observed in the main experiments were with SGD. In Fig. 31 we show that various other optimizers (e.g. Kingma & Ba 2014 Loshchilov & Hutter 2017 Chen et al. 2024 show similar effects without dropout. However, Fig. 32 we show that enabling dropout has different effects on different optimzers convergence. For SGD and adagrad, the bias direction is consistent across dropout rates, though the bias magnitude shows minimum at moderate dropout rates, but for other optimizers the biases reverse
41

---

Published in Transactions on Machine Learning Research (9/2024)
80
Representation variance
60
explained (%)
feature
easy
40
hard hard all input patterms
20
0.33
0.66
1.00
Init scale
Figure 30: Smaller weight initializations yield somewhat larger biases towards easy features. (Bars means across 5 seeds, errorbars are 91s)
Representtion variance explained (%)
80
feature
casy
hard
hard all input patters
20
5gd
adagrad
adam
adermner
lomage
iton
mapopp
Optmizer
Figure 31: Effect of optimizer the results without dropout. Patterns qualitatively similar with all optimizers. However, more dramatic interactions with dropout enabled, Fig. B.7.1 (Bars are means across 10 seeds, errorbars are 95%-CTs.)
or disappear at moderate or high dropout rates. However, this effect only occurs at convergence; earlier in learning the easy features dominate, even after the hard features are generalized fairly well. Furthermore, Fig. 33 we show that the feature biases Transformers trained on sequence datasets are more consistent, even with dropout enabled.
42

---

Published in Transactions on Machine Learning Research (9/2024)
adagrad
adam
adamT
fromage
lion
50
Representaion variance
sepained 196)
feature
easy hard hard all input patterns
25
50
50
75
100
75
100
50
100 50
75
50 100
Hard feature test accuracy
Figure 32: There is strong interaction between dropout rate and optimizer for MLP architectures, but only at convergence. To visualize the results despite very different learning speeds with the different optimizers, this plot visualizes representation content as a function of hard feature learning, by plotting hard feature test accuracy on the horizontal axis. While all optimizers show similar patters effects without dropout, and with dropout even fairly late stages of learning the hard feature (90-95% generalization), dropout causes different effects at convergence with different optimizers. SGD, and Adagrad show consistent gaps between easy and hard features where they converge, but most other optimizers show bias towards the harder features at moderate high dropout rates. (Bars are means across 10 seeds, errorbars are 95%--Is SGD and AdaGrad failed to learn with high dropout rates, so they are omitted from the last row.)
43

---

Published in Transactions on Machine Learning Research (9/2024)
Encoder 2 (%)
Penultimate R 2 (%)
2.0
Representation variance explained (%)
1.5
1.0
1-exactnatch 2 --excttmacch
3-exactmacch
e-xacttaatc
2 --alibutone
3 --alibuoone
A-allbutone
-ummmodz
exxactmatch
2 7-exactmatch
3-exactmatch
4-exactmatch
Z-alibtonn
3-alibutone
4- -allbutonee
4-summod2
Feature
Figure 33: Unlike the MLP results above, the feature biases of Transformers trained on the more complex
sequence tasks (cf. Fig. 7 remain fairly consistent even with dropout enabled. Note that this plot is produced convergence, all models achieve >99% test accuracy. Thus, the bias switch convergence scems less likely to occur in larger models trained randomly sampled datasets and seeds, errorbars are 95%-CIs.)
complex tasks. (Bars means across 64
44

---

Published in Transactions on Machine Learning Research (9/2024)
B.8 Transformer sequential learning curves
In Fig. 34 we show learning curves for transformers trained on the letter-sequence tasks, with one feature enabled at a time. We obscrve some representation biases due feature order, more so at the encoder layer. In fact, there seems to be relatively little change in the feature ranking at the encoder layer once the first few features learned. One possible explanation is that the encodings may be "good enough" to not require substantial further learning to identify other features at other positions, thus leaving lingering bias.
At the decoder layer, we see more complex patterns. We use 100 epochs to smoothly transition to adding the loss on the new feature, but even so we observe substantial interference, which increases with each subsequent feature. We also see some decay towards the end of learning each feature, likely due to the weight decay. The magnitude of bias appears vary as different features are trained. Note that because the transformer's processing shared across time points, variance explained for a given feature tends increase even before its loss weight is enabled. The weight sharing may also contribute to the overall inflation of the variance explained by each feature as learning progresses (the staircase pattern), even once that feature has been completely learned.
1.0
Encoder R-2 (%)
Representation variance explained (%)
Feature
0.5
training order
2
60
Penultimate R-2 (%)
40
20
2000
4000
6000
8000
Training epochs
Figure 34: Learning curves for sequential training in Transformers. Feature order biases are noticeable, but appear weaker than in the MLPs, at least at the penultimate layer. However, several other qualitatively interesting patterns appear, including very little change in the encoder after the first few features are learned, and decay, interference, and recovery in the decoder. (Lines are averages across 48 scods. Dashod lines indicate when the loss begins ramp up for each feature.)
B.9 Additional Dyck language results
In this section we present some further analyses the transformers trained on the Dyck languages. First, in Fig. 35 we present the representation variance explained by each feature without controlling for sequence length.
45

---

Published in Transactions on Machine Learning Research (9/2024)
Second, as mentioned in the main text, the models do not learn these features equally well (Fig. 36 Thus, in Fig. 37 we plot the variance explained by each feature a function of test accuracy. We do not see a clear correlation between how well feature is generalized and how much variance it explains, (although the variability in each feature's test accuracy is low, which might reduce our power to detect such an effect).
Encoder R R (%)
Penultimate R 2 (%)
Representation variance explained (%)
40
30
20
20
10
10
token class
Deepesttoken class
Max depth
root nodes
Max branching
Firsttoken class
Deepest token class
Max depth
Foot noots nodes
Max branching
Feature
Figure 35: Representation variance explained without controlling for length at the final encoder layer (left), and pre-logits decoder layer (right) for transformers trained to identify semantic and structural features of sentences from but the correlations with length change the numerical values. (Bars are averages across 32 seeds, intervals are 95%-CIs.)
Results are broadly qualitatively similar to the length-controlled results in Fig.
Feature
token token class
Deepest
0.4
Max depth
nodes
branching
Density
85
g0
100
Test accuracy (%)
Figure 36: Test accuracy distributions (across runs) on superficial and structural features of sentences from (yckk-(20100)
46

---

Published in Transactions on Machine Learning Research (9/2024)
Encoder (%, length controlled)
Penultimate (%, length controlled)
Representatinn variance explained (%)
40
4
30
Feature
Deepest token class Max depth
nodes
2
Max branching
First token class
0.85
0.90
0.95
1.00
0.85
0.90
0.95
1.00
Feature test accuracy (%)
Figure 37: Representation variance explained at the final encoder layer (left), and pre-logits decoder layer (right) vs. test accuracy for transformers trained to identify superficial and structural features of sentences from 12ck-120,10). (Lines are linear model fits across 32 seeds, intervals are SE.)
B.10
Additional vision experiment results
B.10.1
ResNet representational biases for shape are affected by the number of class labels and the presence of distractors
We noted a surprising shift in reproscntational biases of ResNets between our two vision scttings: favoring shape in one case, but favoring color and texture in another. Here, we disentangle the contributors to this effect, by removing some of the confounding changes.
First, in Fig. 39a replace the original single-object classification task with the binary-label version, without changing the input distribution or adding new objects, we still see shape biases (though they are somewhat less pronounced).
show that these changes not driven solcly by the shift to binary labels; if we simply
Second, we keep training only on binary classifications of the basic features, but switch to the input distribu- tion of the ional lasks; that is, we ellectively introduce two distractor objects, which the model simply has to ignore (because we train it only to report binary classifications of the first object). Surprisingly, the biases shift in this case to favor color and texture over shape (Fig. 39bp.
Thus, because the biases were reduced with binary labels in the original setting, and flipped in the multi- object sctting, it seems that both changes may contribute to the bias shift, but with larger contribution from the distractor objects.
47

---

Published in Transactions on Machine Learning Research (9/2024)
Test accuracy (%)
Spatial pool R (%)
Penultimate 2 (%)
100
30
75
20
10
Value (%)
feature
color texture
50
shape
5
25
0
0
10
15
20
5
10
15
20
0
10
15
20
poch
(a) CNNs
Testaccu (%)
patial pool R "2 (%)
Penultimate R 2 (%)
100
5.0
15
75
Value (%)
feature
color
50
texture
0.0
shape
5
25
-2.5
0.0
2.5
5.0
7.5
10.0
0.0
2.5
5.0
7.5
10.0
0.0
2.5
5,0
7.5
10.0
Epoch
(b) ResNets
Figure 38: Learning curves for test accuracy and variance explained for CNNs and Res.ets trained on the color-eexture-shape tasks. (a) For CNNs Color, which learned first, quickly comes to dominate at the spatial pooling layer; the penultimate layer produces relatively less bias. (b) For Resnets, color briefly dominates, but is quickly overtaken by shape. (Lines averages across 32 seeds, areas are 95%-Cls.)
48

---

Published in Transactions on Machine Learning Research (9/2024)
Spatial pool R 22 (%)
Penultimate R 22 (%)
S patiil pool 22 (%)
Penultimate 2 (%)
Representatinn variance explained (%)
Representation variance
15
30
explained (%)
10
20
10
color (binary)
texture (binary)
shape (binary)
color (binary)
texture (binary)
shape (binary)
color (binary)
texture (binary)
shape (binary)
color (binary)
texture (binary)
shape (binary)
Feature
Feature
(a) Single object alone
(b) With distractor objects
Figure 39: Representational biases of ResNets trained to report binary classifications of the color, shape, and texture of single object shift depending the other objects in the image. (a) with other objects in the image, the representations stiil show shape bias, but more weakly than with the original 10-way classifications (Fig, [ib]. (b) With distractor objects in the image, the biases shift to favor color and texture. (Lines are averages across 32 scods, areas are 95%-CIs.
B.11 Keeping all but the top-k principal components
In Fig. 40 we show that if we drop the top principal components from the model's representations (rather than preserving them, as is typical), the predictions of the hard feature are robust to dropping more components, because the top components are occupied with the easy feature.
100
Accuracy (%)
feature
easy hard
9 10 11 12 13 14 15 Top representation SVD components removed
2
3 4
Figure 40: If we drop the top-k principal components from a models representations, the easy feature predictions will be harmed at a smaller & than the hard feature. (Cr. Fig. 14
B.12 Low-dimensional visualizations
In this section we show several low-dimensional visualizations of the representation space of models trained complex feature (sum % 2) and varying number of easy features (1-4).
49

---

Published in Transactions on Machine Learning Research (9/2024)
In Fig. 41 we show visualizations of the first six principal components of the models. The easy features always occupy at least the first two principal components. In the case that there is only a single feature, the hard feature is separated along the third principal component. As more easy features are added, they occupy successively more principal components, and the harder feature is correspondingly shifted one PC higher for each easy feature that's 's added. Given this, somewhat surprising that two PCs are devoled the single easy feature the one casy-feature case, when just one would suffice.
In Fig. 42 we show t-SNE visualizations of the same networks. It is noticeably casier to see the hard feature separating the clusters in these plots than in the top few principal components, which demonstrates the benefits of a nonlincarity and locality bias of t-SNE. Nevertheless, more easy features are added, it becomes hard to tell that there is another feature separating the clusters, particularly in the case of 4 easy features.
Taking a step back, how would the results of this scction extrapolate to more realistic settings? If we imagine more realistie problem regime in which tens, hundreds, or perhaps even thousands of features contribute to model's predictions, it seems quite likely that some of the computationalll-imppe howplly-mmooottat-bbbee but-ut--bu-coople features would be entirely swamped by the noise of the easier features in any typical feature visualization.
50

---

Published in Transactions on Machine Learning Research (9/2024)
-9
(a) easy feature, PCs 1/2
(b) easy, PCs 3/4
(c) easy, PCs 5/6
44
(d) easy, PCs 1/2
(e) easy, PCs 3/4
(f) 2 easy, PCs 5/6
(g) easy, PCs 1/2
(h) easy PCs 3/4
(i) 3 easy, PCs 5/6
(i) 4 easy, PCs 1/2
(k) easy, PCs 3/4
(I) 4 easy, PCs 5/6
Figure 41: Principal components visualizations of penultimate representations from MLPs that are trained to output a fixed hard feature (sum % 2), together with different numbers of easy linear features (increasing from 1-4 across rows). As easy features are added, their labels (colors) spread across more principal components (columns), and thus the dimension along which the hard features (shapes) separate is pushed into higher and higher principal components. With easy feature (top row), the first two PCs are clustered according the easy feature (a), but the hard feature is clustered along the third principal component (b). When two easy features are learned, the third principal component is used to help represent them, and the hard feature is pushed into the fourth principal component ( When three four easy features are learned, the hard feature is pushed into the 5th or 6th PCs, respectively & Db. There is remarkably consistent progression that as each easy feature is added, another PC is captured, and the hard feature is pushed one PC higher.
51

---

Published in Transactions on Machine Learning Research (9/2024)
40-
Hard Posture
12,
12, 31
Hard Fuature
20-
-20
40
-40
-30
-0
-10
10
20
30
-30
-20
-10
10
20
20
(a) easy feature
(b) easy features
40-
Easy Peatares
Basy heatures es
20
45
12
0.1.33
12
10
Hard
D.5.0 13
1.
D. 0
11
10.1.0
-20
-40
(c) 3 easy features
(d) 4 easy features
Figure 42: 1-SNE visualization of the penultimate representations from MLPs that are trained to output a fixed hard feature (sum % 2), together with different numbers easy linear features (increasing from 1-4 across pancls). With a small number of easy fcatures, the clustering according to both the easy feature values (colors) and the hard features (shapes) emerges quite clearly. The existence of the hard feature is generally readily visible in these t-SNE plots than in examining just the top few principal components (ef. Fig. AlI. However, more easy features are introduced, they come to dominate the representation space, and with 4 easy features it would be difficult to discriminate the hard features separating most clusters if they were not already labelled with shapes.
52

---

Published in Transactions on Machine Learning Research (9/2024)
B.13
RSA result resilience
In Fig. 43 show that the RSA results in the main text are qualitatively similar if different metrics are used for the analysis, or if Spearman rank correlation is used to compare the dissimilarity matrices.
pearson
spearman
(SUM%2 SUMS (SUM%2, ANDIXOR XOR)
SUM XOR
SSM %2 AND
USM Linear
(ANDIXOR XOR S.J.SUM U
(AND(XOR, XOR ANDIXOR K.R.R. XORO
[AN XXRRXXOR XOR
HANDIE XORO AND
MINDIXOP (XO
OXOR M
(XOR, AND(XOR XOR)
cosine
XOR
AND]
(AND, ANDIXOR XOR)
AND XOR AND]
Network B tasks
[Linear, SUM%
(Linear, AND(XOR, R.COO.XO XOR)
Linear,
Linear, AND]
(Linear, Linear)
(SUM% % SUM%2)
(SUM%2, OOR XXRR
ISUM SUM % XOR
S SUMS %2 AND
(S UM% % Linear
JANDIXOR R XOR), R) SUM%2)
(OND(XORXXOR) XXRR XOR XOR
XXR XOR
(ANDIX OR XORT AND
(ANDIXOR XOR Linear
OXO 3UM%
euclidean
(XOR, ANDIXOR XOR [XOR, XOR
XOR AND
(XOR, Linear)
(AND. 2
(AND, AND(XOR XOR
(AN XOR
AND AND
(AND. Linean
[Linear. UM
(Linear, ANDIXOR XOR R
(Linear, XOR
[Lnear, AND
(Linear, Linear)
(Linear, Linesar
POPOO (R.OOR)
(Lineor PeBorSSUMME SUM%22
(AND) Lineor
(AND, (AND, ANDIXON XOR)
SUM%)2
(XOR Linesr
(XOR SUMP)2
ANDIXOS XOR]
VANDOX A
(OR) (R) Linear
ANDDXXR X
XOR)
(SX SUM%
Linear
AUM9 SUMMM (S USM2 (SATSSSUMM
Anese Linear)
Sumaa 1, AND
PANS XOR)
(Linear. ANDOXOS XOR)
(Linear, SUM9
ZANA Linear)
(AND) AND
AND. XOR)
(AND, ANDIX OR XOR)
(AND SUM%)2
Linear)
KOR AND
XOR)
CORRXOO XOR)
(XOR, SUM9)2 X. Linear) LANDOX
DDDOO (OR) AND
AANDA XOR XOR)
FRR NOM9
KORL
MOMMO Livean
Representational similarity
[Lines)
(Linear, ANDDAN
(XOR, ANDIXO
AND
R.
ANDIX XOR
BOX0O
1.00
0.75
(ANDIXOR
(XOR.
MaaDoCco XOR
IS CUMMMA
0.50
0.25
0.00
Network A tasks
Figure 43: RSA results with different dissimilarity metrics (Cosine distance on the top, Euclidean distance on the bottom), and comparison correlations (Pearson on the left vs. Spearman on the right). Results are qualitatively similar in all cases. (Results computed across 5 seeds per feature pair.)
53


Reviews:
[{"review_id": "MyygB80el4", "reviewer": "Reviewer_GLvA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the primary strengths of the paper are as follows:\n1. I appreciate that the authors have provided their definitions for terms at the start of the paper. Given that there is no consensus answer to the question \"what is a feature?\", I think this is important.\n2. The questions being asked are of interest to the community, in the sense that finding generalisable biases in feature learning order or dominance would be valuable.\n3. I like the number of setups (although I think the number has limited the rigour).\n\nSome more minor strengths:\n1. I found the insight provided in section 6.2 noteworthy: \"In particular, if the features required to solve the harder OOD splits are more complex, they may occupy relatively less variance, and therefore be more readily lost when keeping only the top principal components.\"\n\nI think the main weaknesses are:\n1. It seems that the way \"variance explained\" is measured is by fitting a linear predictor. This seemingly assumes a linear representation of the feature, which might be problematic if more complex features are represented non-linearly in the network. As such, the linear probe might explain less variance for complex features. This weakness may be somewhat ameliorated if the probes are trained only on the penultimate layer (especially if the last layer is an unembed rather than an MLP). However, it is still a major concern in my mind.\n2. In section 3.2, how did the authors come up with \"all patterns of inputs relevant to the hard feature\"? What if the network is using a really unconventional representation that isn't included in this list of patterns? See [Progress measures for grokking via mechanistic interpretability](https://arxiv.org/abs/2301.05217).\n3. Given that the tasks in section 3 and many of the tasks in section 4 are algorithmic, I'm unsure how well the results in those sections generalise. We know from grokking that the training dynamics and representations associated with algorithmic tasks may be very different from natural datasets. Given this, I think sentences like the following are claiming too much: \"Thus, while these results show that the biases can be shifted in nontrivial ways by hyperparameter choices, we expect that easy feature biases will persist in standard training settings, where models are generally not trained to complete convergence on their tasks.\"\n\nSome more minor weaknesses:\n1. Could the authors please fix the formatting of the paper? For example, can page 8 and 11 be fixed?\n2. I think that grokking and algorithmic tasks should be discussed more in the related work. I can provide some suggested literature if the authors would like.\n\nI also have some questions:\n1. Can you describe in more detail what was done in section 3.2? The weakness I have listed in this section may be a result of my misunderstanding.\n2. Can the authors also provide further justification for the use of the linear predictor?\n\nUpdate: after author changes, I have decided to change to \"Yes\" for \"Claims and Evidence\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors train neural networks to predict features of the input. They then examine properties of how these features are learnt by varying properties of the setup, such as feature complexity, learning order, and the distribution of features.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As it currently stands, I do not think the work is sufficiently rigorous for publication in TMLR. As stated above, I am concerned by the fact that the authors seem to assume that features are embedded linearly and rely on this assumption throughout the rest of the article. In addition, many of the claims seem too far-reaching for the level of evidence provided.\n\nI would like to see:\n1. Focus on building experiments with much greater rigour. If the authors are going to analyse properties of feature learning assuming linear features, I would like them to study a model where we are much more confident that these features do end up being linear. In my eyes, the work would have been much stronger if the authors had simply focused on a very small number of training setups such as [othello-gpt](https://www.neelnanda.io/mechanistic-interpretability/othello) rather than many cases where they have not demonstrated that information is actually encoded linearly.\n2. An improvement in the formatting and writing. Although TMLR is more accepting of papers without headline claims and more verbose discussion of results, I believe that the current writing needs to be much more succinct and structured. As it currently stands, when I read the paper I feel like I am gaining some interesting observations but no clear take-home messages.\n3. A narrowing of claims made based on algorithmic datasets and a more rigorous handling of the differences between algorithmic and more natural datasets.", "broader_impact_concerns": "I do not think there are any real ethical concerns with this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3Uzuxh4vFA", "reviewer": "Reviewer_qC5t", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper flows well using the simple MLP setting and then generalizing to transformers to CNNs. They accompanying notebook is simple and makes things clear, too. They do a good job of contextualizing the work with transformers and dictionary learning with sparse autoencoders.\n\nSome of these methods may behave differently with much larger models that are much deeper. I feel like the different behavior or residual nets is under-explored.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper explores the biases of learned feature representations in neural networks. By feature they mean a predicted property of the raw input, \"Does the sequence contain 'abc'?\" or \"What is the color?\". Representation is some encoded form of the input which is typically the activations of the last layer of a neural network.\n\nThey come up with a systematic way to explore how much variance is explained in the representations due to a specific feature. The feature may be \"easy\" or \"hard\" which is loosely defined as how many nonlinear transformations of the input is needed to compute the feature. They carry out experiments with synthetic data and several different model architectures.\n\nThey consistently find that most of the variance is explained by these simpler features. Things like learning order or residual connects seem to affect the affect the bias. They discuss the implications this has for mechanistic interpretability work that uses Principal Component Analysis (PCA) or really anything that uses a metric similar to L2 reconstruction loss since the work would suffer from the same bias.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper is good enough as is, but I would be curious to see more work on larger models and a better understanding on why residual nets seem to have different biases.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nAAFzDDHfI", "reviewer": "Reviewer_VVSP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nThis paper investigates representation bias in deep learning models by systematically manipulating feature attributes and the training regimes, which can potentially offer insights into how deep models work.\n\n**Weaknesses:**\n\n1.\tThe experiments focus on toy/synthetic datasets, which may limit the generalizability of conclusions about representation bias to more complex tasks. For example, in vision tasks, the authors suggest that \"features like color are easier for models to learn.\" Does this conclusion hold across different model architectures, such as classic ConvNets (AlexNet, ResNet, VGG, DenseNet, etc.) and Transformer-based architectures (ViTs, Swin Transformers)? Furthermore, how do these findings extend to more complex tasks? Therefore, the authors need to consider the applicability of their conclusions about representation bias, e.g., whether they are task-specific, architecture-specific, or both. Additionally, validating these findings on more complex real-world datasets and across a wider range of model architectures would provide a more comprehensive understanding.\n\n2.\tThe paper needs to provide clear mathematical definitions for important concepts, instead of relying on vague textual descriptions. For instance, in Section 2, the authors mention \"...even when the features play similar computational roles\". What are \"similar computational roles\"? Does this refer to the similar output vectors or probabilities? To address this, the authors should clarify the important concepts in Section 2, such as \"input stimulus\", \"feature\", \"feature representation\", and \"representation variance explained by a feature\", providing precise mathematical definitions or formulas. If it is challenging to abstract these concepts into unified mathematical definitions across different tasks, the authors could provide specific definitions for classic tasks.\n\n3.\tFor visual tasks, the authors should provide examples of the synthetic datasets used. Furthermore, the authors should specify the details of the non-residual CNNs, including the number of layers, the network structure, and whether all non-residual CNNs, such as VGG and AlexNet, are considered.\n\n4.\tMinor: There are problems with the layout of the figures. For example, the caption of Figure 9 is truncated and cannot be fully displayed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper empirically finds that learned feature representations are biased by various factors, such as feature complexity, the learning order of different features, and the distribution of features. Experiments show that these biases are affected by architectures (i.e., MLPs, Transformers and ConvNets) and training regimes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper finds that feature representations of DNNs are usually biased by the feature complexity, the order of features in learning, and the distribution of features. These finding are of interests to the community of artificial intelligence. Experiments have well supported these findings. Reviewers agree the acceptance of this paper.
Decision: Accept (poster)