Early learning of the optimal constant solution in neural networks and humans

## Abstract

Abstract Deep neural networks learn increasingly complex functions over the course of training. Here, we show both empirically and theoretically that learning of the target function is preceded by an early phase in which networks learn the optimal constant solution (OCS) – that is, initial model responses mirror the distribution of target labels, while entirely ignoring information provided in the input. Using a hierarchical category learning task, we derive exact solutions for learning dynamics in deep linear networks trained with bias terms. Even when initialized to zero, this simple architectural feature induces substantial changes in early dynamics. We identify hallmarks of this early OCS phase and illustrate how these signatures are observed in deep linear networks and larger, more complex (and nonlinear) convolutional neural networks solving a hierarchical learning task based on MNIST and CIFAR10. We explain these observations by proving that deep linear networks necessarily learn the OCS during early learning. To further probe the generality of our results, we train human learners over the course of three days on the category learning task. We then identify qualitative signatures of this early OCS phase in terms of the dynamics of true negative (correct-rejection) rates. Surprisingly, we find the same early reliance on the OCS in the behaviour of human learners. Finally, we show that learning of the OCS can emerge even in the absence of bias terms and is equivalently driven by generic correlations in the input data. Overall, our work suggests the OCS as a universal learning principle in supervised, error-corrective learning, and the mechanistic reasons for its prevalence.

## 1 Introduction

Neural networks trained with stochastic gradient descent (SGD) exhibit various simplicity biases, where models tend to learn simple functions before more complex ones [, ]. Simplicity biases hold significant theoretical interest as they provide an explanation for why deep networks generalize in practice, despite their expressivity and the presence of minima with arbitrarily high test error [, , ].

The characterisation of simplicity biases is still incomplete. Some explanations appeal to distributional properties of input data, pointing out that SGD progressively learns increasingly higher-order moments [, ]. Other approaches focus directly on the evolution of the network function, proposing that networks initially learn a classifier highly correlated with a linear model. Importantly, networks continue to perform well on examples correctly classified by this simple function, even when overfitting in later training []. This implies that dynamical simplicity biases help models generalise, by locking in initial knowledge that is not erased or forgotten during later training [, ].

Deep linear networks have proven to be a valuable tool for studying simplicity biases. A key finding is that directions in the network function are learned in order of importance [, ]. This phenomenon, known as progressive differentiation, connects modern deep learning theory to both to human child development and to the earliest connectionist models of semantic cognition [, ].

Our contribution proposes a connection between these works by characterizing networks in the earliest stages of learning in terms of input, output, and architecture. In the hierarchical setting by Saxe et al. [], we demonstrate both theoretically and empirically that neural networks initially learn via the output statistics of the data. This function has been termed the optimal constant solution (OCS) by Kang et al. [], who demonstrated that networks revert to the OCS when probed on out-of-sample inputs. Here, we demonstrate and prove how linear networks, when equipped with these bias terms, necessarily learn the OCS early in training. We furthermore highlight the practical relevance of these results by examining early learning dynamics in complex, non-linear architectures.

Biological learners also display behaviours that imply the input-independent learning of output statistics. In probability matching, responses mirror the probabilities of rewarded actions [, , ]. Learners often display non-stationary biases that are driven by the distribution of recent responses [, , ]. In paired-associates learning accuracy can depend not only on a learned input-output mapping but also on knowledge of the task structure [, ]. Humans also display simplicity biases and preferentially use simple over complex functions [, , , , ]. However, relatively little attention has been devoted to the dynamics of these biases. We conduct experiments to determine whether humans replicate early reliance on the OCS.

Figure: Figure 1: Universal early learning of the optimal constant solution (OCS). A Graphical illustration of our hypothesis. B The hierarchical learning task used across learners and the OCS solution $\hat{\mathbf{y}}_{ocs}$. C Learning task in linear networks with bias terms. D The task in CNNs. E The task for humans.
Refer to caption: x1.png

### 1.1 Contributions

- •
We devise exact solutions for learning dynamics to analyse linear networks with bias in the input layer. Even when initialized at zero, this component substantially alters early learning dynamics.
- •
We empirically characterise early learning in these linear networks as being dominated by average output statistics. We explain this result with a theoretical analysis which reveals that average output statistics are always learned first when the network contains bias terms.
- •
We further highlight the practical relevance of these theoretical results in a hierarchical learning task for humans as well as complex, non-linear architectures by empirically demonstrating that all learners develop stereotypical response biases during early stages of training.
- •
We conjecture on the basis of the developed theory that an early response bias should emerge even in absence of bias terms. We empirically demonstrate that learning of the OCS can indeed be driven purely by generic correlations in the input data.

### 1.2 Related work

Deep linear networks.
In deep linear networks analytical solutions have been obtained for certain initial conditions and datasets [, , , ]. Progress has also been made in understanding linear network loss landscapes [] and generalisation ability []. Despite their linearity these models display complex non-linear learning dynamics which reflect behaviours seen in non-linear models []. Moreover, learning dynamics in such simple models have been argued to qualitatively resemble phenomena observed in the cognitive development of humans [, ].

Biological response biases. Humans and animals routinely display response biases during perceptual learning and decision making tasks [, , , , ].
In these tasks decisions are frequently made in sequences where responses and feedback steer decisions beyond the provided perceptual evidence [, , , , ].
Non-stationary response biases can be driven by feedback on previous trials [, ] or might reflect global beliefs about the statistics of a task [, ]. Importantly, response biases are particularly pronounced in early learning [, , ] and their influence appears to be strongest when uncertainty about the correct response is highest [, ].

Simplicity biases in machine learning.
Simplicity biases in neural networks have been studied extensively both theoretically [, ] and empirically [, ]. Work on the distributional simplicity bias emphasises the importance of input data and proposes that models learn via progressive exploitation of dataset moments [, ]. On the other hand, neural networks have been found to express simpler functions during early training [, , , ]. Our work draws a connection between these findings and highlights how input statistics bias early learning towards output statistics.

### 1.3 Paper organisation

We initially review the linear network formalism in Section 2 on which we base our theoretical analysis. In Section 3 we derive learning dynamics for linear networks with bias terms trained on a classic hierarchical task and we document substantial changes in early dynamics. Section 4 characterizes this period of early learning empirically, and provides a theoretical explanation. We then in Section 4 validate the relevance of our findings for learning in complex models. Section 5 demonstrates the prevalence of early OCS learning in humans. Finally, Section 6 further probes generality by considering natural datasets and models that do not strictly fulfil the previous theoretical assumptions.

## 2 Linear network preliminaries

Here, we briefly review the analytical approach to learning dynamics in linear networks developed by [, ].
Consider a learning task in which a network is presented with input vectors $\mathbf{x}_{i}\in\mathbb{R}^{N_{in}}$ that are associated to output vectors $\mathbf{y}_{i}\in\mathbb{R}^{N_{out}}$. The total dataset consists of $\{\mathbf{x}_{i},\mathbf{y}_{i}\}^{N}_{i=1}$ with $N$ samples. For our setting we consider two layer linear networks where the forward pass computes $\hat{\mathbf{y}}_{i}=\mathbf{W}^{2}\mathbf{W}^{1}\mathbf{x}_{i}$ and shallow networks with forward pass $\hat{\mathbf{y}}_{i}=\mathbf{W}^{s}\mathbf{x}_{i}$. Here weight matrices are of dimension $\mathbf{W}^{1}\in\mathbb{R}^{N_{hid}\times N_{in}}$, $\mathbf{W}^{2}\in\mathbb{R}^{N_{out}\times N_{hid}}$, and $\mathbf{W}^{s}\in\mathbb{R}^{N_{out}\times N_{in}}$. We train our networks to minimise a squared error loss of the form $\mathcal{L}(\hat{\mathbf{y}})=\frac{1}{2}\sum_{i=1}^{N}\|\mathbf{y}_{i}-\hat{
\mathbf{y}}_{i}\|^{2}.$

We optimise networks using full batch-gradient descent in the gradient flow regime. When learning from small initial conditions, dynamics in these simple networks are solely dependent on the dataset input-output and input-input correlation matrices []. Using singular value decomposition (SVD), these matrices can be expressed as

$$ $\mathbf{\Sigma}^{yx}=\frac{1}{N}\mathbf{YX}^{T}=\mathbf{USV}^{T},\quad\mathbf{ \Sigma}^{x}=\frac{1}{N}\mathbf{XX}^{T}=\mathbf{VDV}^{T}.$ (1) $$

Here $\mathbf{X}\in\mathbb{R}^{N_{in}\times N}$ and $\mathbf{Y}\in\mathbb{R}^{N_{out}\times N}$contain the full set of input vectors and output vectors. Crucially, if the right singular vectors $\mathbf{V}^{T}$of $\mathbf{\Sigma}^{yx}$ diagonalise $\mathbf{\Sigma}^{x}$ (see Proposition 1) then the full evolution of network weights for deep and shallow networks through time can be described as

$$ $\mathbf{W}^{2}(t)\mathbf{W}^{1}(t)=\mathbf{UA}(t)\mathbf{V}^{T}.$ (2) $$

Here $\mathbf{A}(t)$ is a diagonal matrix. The evolution of these diagonal values $\mathbf{A}(t)_{\alpha\alpha}=a_{\alpha}(t)$ at each time-step $t$ then follows a sigmoidal trajectory as expressed in Eq. 3. For shallow networks we can similarly describe the evolution of the weight matrix $\mathbf{W}^{s}(t)$ as $\mathbf{UB}(t)\mathbf{V}^{T}$. Here the diagonal values $\mathbf{B}(t)_{\alpha\alpha}=b_{\alpha}(t)$ evolve as seen in Eq. 4

$$ $a_{\alpha}(t)=\frac{\nicefrac{{s_{\alpha}}}{{d_{\alpha}}}}{1-(1-\frac{s_{ \alpha}}{d_{\alpha}a_{0}})e^{-\frac{2s_{\alpha}}{\tau}t}}$ (3) $$

$$ $b_{\alpha}(t)=\frac{s_{\alpha}}{d_{\alpha}}(1-e^{-\frac{d_{\alpha}}{\tau}t})+b _{0}e^{-\frac{d_{\alpha}}{\tau}t}$ (4) $$

In Eq. 3 $s_{\alpha}=\mathbf{S}_{\alpha\alpha}$ and $d_{\alpha}=\mathbf{D}_{\alpha\alpha}$ denote the relevant singular values of $\mathbf{\Sigma}^{yx}$ and the eigenvalues of $\mathbf{\Sigma}^{x}$ respectively, $a_{0}$ are the singular values at initialisation, and $\tau=\frac{1}{N\epsilon}$ is the time constant where $\epsilon$ is the learning rate. In Eq. 4 $b_{0}$ is the initial condition given by the initialisation. Importantly, these relations reveal that singular values control learning speed. These solutions hinge on the diagonalisation of $\mathbf{\Sigma}^{x}$ through $\mathbf{V}$. Prior work has focused on the case of white inputs, i.e. $\mathbf{\Sigma}^{x}=\mathbf{I}_{N}$ where $\mathbf{I}_{N}$ denotes the $N\times N$ identity matrix. The solution holds trivially as any $\mathbf{V}$ will orthogonalise $\mathbf{\Sigma}^{x}$ []. We discuss a relevant relaxation of this condition in Proposition 1. While solutions can be derived for some non-white inputs, little attention has been devoted to learning dynamics in these scenarios. We will show how these solutions apply when networks contain bias terms in the input layer.

## 3 Exact learning dynamics with bias terms

Figure: Figure 2: Exact learning dynamics. A Deep linear networks with (left) and without (right) bias. B Shallow linear networks with (left) and without (right) bias. Top row: Exact and simulated $\mathbf{A}(t)\text{ and }\mathbf{B}(t)$ for deep and shallow networks respectively. Bottom row: Exact and simulated loss.
Refer to caption: x2.png

In this section, we derive exact learning dynamics in linear networks with bias terms and analyse the resulting changes in the dynamics. This extension to the theory by Saxe et al. [] forms the basis for our later discussion. For simplicity, we focus on input bias terms and uncorrelated data, but explore bias terms in other layers and correlated inputs in Section A.5.2 and Section 6, respectively.

##### Closed-form learning dynamics.

We consider uncorrelated inputs $\mathbf{X}=\mathbf{I}_{N}$ where $\mathbf{I}_{N}$ denotes the $N\times N$ identity matrix. Our linear network with a bias term in the input layer will compute $\mathbf{W}^{2}(\mathbf{\tilde{W}}^{1}\mathbf{x}_{i}+\mathbf{\tilde{b}})$ where $\mathbf{\tilde{b}}$ are learnable bias terms.

The diagonalisation of $\mathbf{\Sigma}^{x}$
through $\mathbf{V}$ in Eq. 1 is not generally possible if the input layer includes bias terms. Here, we state the condition under which learning dynamics can be described in closed-form.

###### Proposition 1 (Feasibility of closed-form learning dynamics) .

For any input data $\mathbf{X}\in\mathbb{R}^{N_{in}\times N}$ and
output data $\mathbf{Y}\in\mathbb{R}^{N_{out}\times N}$ it is possible to diagonalize
$\mathbf{\Sigma}^{x}$ by the right singular vectors $\mathbf{V}$
of $\mathbf{\Sigma}^{yx}$ if $\mathbf{Y}^{T}\mathbf{Y}$ and $\mathbf{X}^{T}\mathbf{X}$
commute.
The converse holds true only if $\mathbf{X}$ has a left inverse.

A proof is given in Section A.5.4. We put this statement to use to assess the effect of a bias term on learning, building on the formalism from Section 2. To this end, we re-express the network weights as $\mathbf{W}^{1}=\begin{bmatrix}\mathbf{\tilde{b}}&\mathbf{\tilde{W}^{1}}\end{bmatrix}$ with inputs defined as $\mathbf{x}_{i}=\begin{bmatrix}1&\mathbf{I}_{i}^{T}\end{bmatrix}^{T}$ where $\mathbf{I}_{i}$ denotes the $i$th column of the $N\times N$ identity matrix (see Section A.5.1). Exact learning trajectories for the hierarchically structured output data are depicted in Fig. 1B.

Importantly, the introduction of a bias term $\mathbf{X}\rightarrow\begin{bmatrix}\mathbf{1}_{N}\>\mathbf{X}\end{bmatrix}^{T}$ does not affect the commutativity of $\mathbf{X}^{T}\mathbf{X}$ and $\mathbf{Y}^{T}\mathbf{Y}$ for the hierarchical dataset, as the constant mode $\mathbf{1}_{N}$ (i.e., a vector of 1s) is already an eigenvector to both these similarity matrices (see Section A.5.6). In consequence, the analytical solutions in Section 2 remain applicable. We generalise these considerations in Section 6 and Section A.5.6.

Fig. 2 shows that linear networks with bias terms have a distinctly different
early learning phase when compared to vanilla linear networks. While both models converge to a zero loss solution, we observe that the final network function with bias terms contains an additional non-zero singular singular value with their associated singular vectors. We devote the next section to analyzing this change in the early dynamics.

###### Proposition 1 (Feasibility of closed-form learning dynamics) .

## 4 Bias terms drive early learning towards the optimal constant solution

Figure: Figure 3: Early learning is driven to the OCS. Top row: Network outputs for a single output unit in response to all inputs $\mathbf{x}_{i}$. Bottom row: Continuous correct-rejection scores $f^{tn}$ for the three hierarchical levels as indicated by colours (Section A.4) Performance approaches levels expected under the OCS (dotted lines). Left: Linear networks with bias terms. Center: CNNs. Right: Linear networks without bias terms.
Refer to caption: x3.png

In this section, we qualitatively characterize what causes observed changes in early learning dynamics. We find that early learning dynamics are driven by average output statistics and provide a theoretical explanation. We then demonstrate the generality of this result by highlighting how early learning of average output statistics can be similarly observed in complex, non-linear architectures.

A naive strategy to learning is to minimise error over a set of samples while disregarding information conveyed by the input. Previous work has recently termed this network function the optimal constant solution (OCS) []. The OCS can be formalised as $\hat{\mathbf{y}}_{ocs}=\operatorname*{argmin}_{\hat{\mathbf{y}}\in\mathbb{R}^{
N_{out}}}\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}(\hat{\mathbf{y}},\mathbf{y}_{i})$ and represents the optimal function $\hat{\mathbf{y}}$ that is independent of input $\mathbf{x}_{i}$. For mean-squared error, it is straightforward to show that the minimiser is the average output $\hat{\mathbf{y}}_{ocs}=\frac{1}{N}\sum_{i}^{N}\mathbf{y}_{i}\eqqcolon\bar{
\mathbf{y}}$.

Setup. We train linear networks and Convolutional neural networks (CNN) on the hierarchical learning task in Fig. 1. For CNNs we design a variation of the task whereby one-hot inputs are replaced with eight randomly sampled classes of MNIST []. We use standard uniform Xavier initialization [] and trained CNNs on an squared error loss. A full description of the CNN experiment and hyperparameter settings is deferred to Section A.8. We there also replicate our results with CIFAR-10 [], non-hierarchical tasks, alternative loss functions, and CelebA [] in Section A.9. We also show results for shallow networks in Section A.6.

### 4.1 Empirical evidence

We identify three separate empirical observations that support early learning of the OCS:

##### Indifference.

Linear networks and CNNs initially change outputs while not differentiating between input examples. In Fig. 3 (top) we show the empirical and analytical activation of an output unit associated with the highest level of the hierarchy for all $\mathbf{x}_{i}$. Networks with and without bias terms learn to differentiate inputs correctly. However, networks with bias terms produce input-independent, non-zero outputs in early training as would be expected under the OCS.

##### Performance.

We provide continuous correct-rejection scores for our task which approach levels expected under the OCS in Fig. 3 (bottom). Effectively, this metric measures wrong beliefs about the presence of target labels across the different levels of the hierarchy – it describes how strongly outputs $\hat{\mathbf{y}}$ align with the desired outputs $\mathbf{y}$ while focusing on zero entries only. We calculate the metric separately for the three hierarchical levels (details in Section A.4). Linear networks and CNNs with bias terms approach performance levels that would be provided by the OCS (dotted lines) for each of the three hierarchical levels. Linear network without bias terms do not produce this behaviour.

##### OCS alignment.

The distance between outputs $\hat{\mathbf{y}}_{i}$ of linear and non-linear networks and $\mathbf{y}_{ocs}$ approaches zero in early training. Fig. 6 (top) shows how the $L_{1}$ distance of sample-averaged network outputs and the OCS approaches zero before later converging to the desired network function.

### 4.2 Theoretical explanation

In this section, we extend the linear network formalism to understand the mechanism behind early learning of the OCS. We first show how bias terms in the input layer are directly related to the OCS. Afterwards, we prove that the OCS is necessarily learned first in these settings.

##### The OCS is linked to shared properties.

Having established the applicability of the linear network theory in Section 3 we now seek to understand how the early bias towards the OCS emerges. To this end, notice how bias terms can be written in terms of the constant eigenmode $\mathbf{1}_{N}$:

###### Proposition 2 (The OCS is linked to shared properties) .

If $\mathbf{1}_{N}$ is an eigenvector to the similarity matrix $\mathbf{X}^{T}\mathbf{X}\in\mathbb{R}^{N\times N}$,
then the sample-average $\bar{\mathbf{x}}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{x}_{i}$
will be an eigenvector to the correlation matrix $\mathbf{X}\mathbf{X}^{T}\in\mathbb{R}^{N_{\text{in}}\times N_{\text{in}}}$
with identical eigenvalue $\lambda$. An analogous statement applies
for $\mathbf{Y}^{T}\mathbf{Y}$ and $\mathbf{Y}\mathbf{Y}^{T}$. The converse does not hold true in general.

We prove this statement in Section A.5.5. Importantly, it establishes a connection between the feature and sample dimensions of $\mathbf{X}$ and $\mathbf{Y}$. If $\mathbf{1}_{N}$ is an eigenvector to $\mathbf{X}^{T}\mathbf{X}$ and $\mathbf{Y}^{T}\mathbf{Y}$ already, it implies that the addition of a bias term will directly add to its eigenvalue, $s_{ocs}^{2}\rightarrow s_{ocs}^{2}+1$, even if it is initialized at zero. We show in Section A.5.6 that these assumptions on $\mathbf{X}$ and $\mathbf{Y}$ hold strictly for our hierarchical task design, and more generally relate to symmetry in the data (Section A.5.6). We discuss in Section 6 how this property extends to natural datasets where strict symmetry is absent.

Crucially, it now follows from Proposition 2 that the time-dependent network correlation $\hat{\mathbf{\Sigma}}^{yx}(t)=\mathbf{UA}(t)\mathbf{V}^{T}$ in Eq. 2 will contain a strongly amplified OCS mode $a_{ocs}(t)\mathbf{u}_{ocs}\mathbf{v}_{ocs}^{T}=a_{ocs}(t)\bar{\mathbf{y}}\bar{
\mathbf{x}}^{T}$ by virtue of the modified singular value $\sqrt{s_{ocs}^{2}+1}$ entering Eq. 2 and thereby the network function.
Consequently, learning dynamics will be driven by the outer product of average input and output data. Moreover, this implies that given some input $\mathbf{x}_{i}$ to Eq. 2,
the network’s OCS mode contributes

$$ $\mathbf{\hat{y}}_{ocs}(\mathbf{x}_{i})=a_{ocs}(t)\mathbf{u}_{ocs}\mathbf{v}_{ ocs}^{T}\mathbf{x}_{i}=a_{ocs}(t)\bar{\mathbf{y}}\bar{\mathbf{x}}^{T}\mathbf{x }_{i}\propto\bar{\mathbf{y}}.$ (5) $$

The OCS mode in the time-dependent network function will hence necessarily drive responses towards average
output statistics. Note that Eq. 5 also
highlights that the more an input example is aligned to average
inputs, the more the network’s responses will reflect average outputs. In particular, this makes the expected output $\mathbb{E}_{\mathbf{x}}[\hat{\mathbf{y}}(\mathbf{x})]\propto\bar{\mathbf{y}}$. Throughout learning, the evolution of $a_{ocs}(t)$
and scale-dependent alignment of $\mathbf{x}_{i}$ and $\bar{\mathbf{x}}$
will determine the network’s reliance on the OCS mode.

###### Proposition 2 (The OCS is linked to shared properties) .

##### Early learning is biased by the OCS mode.

We established that network responses are driven by average
output statistics $\bar{\mathbf{x}}$ and $\bar{\mathbf{y}}$, but why are *early*
dynamics in particular influenced by the OCS? The learning speed of
the SVD modes in the time-dependent network function are controlled
by the magnitude of singular values $s_{\alpha}$ as seen in Eq. 3.

###### Theorem 1 (Early learning is biased by the OCS mode) .

If $\mathbf{1}_{N}$ is a joint non-degenerate eigenvector to positive input and output similarity matrices $\mathbf{X}^{T}\mathbf{X}$ and $\mathbf{Y}^{T}\mathbf{Y}$, the OCS mode $s_{ocs}\bar{\mathbf{y}}\bar{\mathbf{x}}^{T}$ will have leading spectral weight $s_{0}\equiv s_{ocs}$ in the SVD of the input-output correlation matrix $\mathbf{\Sigma}^{yx}$.

We prove this statement with help of the Perron-Frobenius theorem [] in Section A.5.7.
Consequently,
the optimal constant mode is learned at a faster rate than remaining SVD components and transiently dominates the early network function.
Notably, this applies to our task data $\mathbf{Y}^{T}\mathbf{Y}$ (see Section A.5.6) and leads to characteristic learning signatures observed in
Fig. 3.

###### Theorem 1 (Early learning is biased by the OCS mode) .

##### Integrated formulation of architectural biases.

Figure: Figure 4: Loss curves for different bias variations.
Refer to caption: x4.png

So far, we analysed how bias terms on the input layer affect the singular value spectrum.
Our empirical results in Section 4.1 suggest a more general dynamical bias towards the OCS stemming purely from architectural properties. We here use the neural tangent kernel
$\mathsf{NTK}(\mathbf{x}_{i},\mathbf{x}_{i^{\prime}})=\sum_{k}\frac{d\hat{
\mathbf{y}}_{i^{\phantom{\prime}}}^{\phantom{T}}}{d\theta_{k}}\frac{d\hat{
\mathbf{y}}_{i^{\prime}}^{T}}{d\theta_{k}}$
[] to directly and comprehensively describe the affected time evolution of the network response $\frac{d}{dt}\hat{\mathbf{y}}_{i}=\mathsf{NTK}(\mathbf{y}_{i}-\hat{\mathbf{y}}_
{i})$
at the cost of a closed-form solution. Because changes in network outputs are proportional to the NTK it can been viewed as an architecture-induced learning rate [].
For completeness, we now consider a network that contains input $\mathbf{b}^{1}$ and output $\mathbf{b}^{2}$ bias terms.

###### Proposition 3 (NTK of linear networks with bias terms) .

The neural tangent kernel of a two-layer linear network with input and output-layer bias $\hat{\mathbf{Y}}=\mathbf{W}^{2}(\mathbf{W}^{1}\mathbf{X}+\mathbf{b}^{1})+
\mathbf{b}^{2}$ in the high-dimensional regime in early training reads

$$ $\displaystyle\mathsf{NTK}(\mathbf{X},\mathbf{X})=\sigma_{\mathbf{W}^{2}}^{2} \mathbf{I}_{N_{out}}\otimes\Bigl{(}2\mathbf{X}^{T}\mathbf{X}+\underbrace{ \mathbf{1}_{N}\mathbf{1}_{N}^{T}}_{\leftrightarrow\mathbf{b}^{1}}\Bigr{)}+ \mathbf{1}_{N_{out}}\mathbf{1}_{N_{out}}^{T}\otimes\underbrace{\mathbf{1}_{N} \mathbf{1}_{N}^{T}}_{\leftrightarrow\mathbf{b}^{2}}.$ $$

The tensor product $\otimes$ separates the components that operate on output and sample space.
We briefly review the $\mathsf{NTK}$ and derive this expression in Section A.5.3. The highlighted terms originate from the bias term
$\frac{d\hat{\mathbf{y}}_{i^{\phantom{\prime}}}^{\phantom{T}}}{d\mathbf{b}}
\frac{d\hat{\mathbf{y}}_{i^{\prime}}^{T}}{d\mathbf{b}}$
entering the $\mathsf{NTK}$, manifesting in the appearance of the constant mode $\mathbf{1}_{N}$. Importantly, these terms do not scale with the size of the learned bias – they are present even if the bias is initialized at zero. Intuitively, their contribution stems from the architecture’s potential to learn a bias, enabling rapid changes in output $\hat{\mathbf{y}}$.
The $\mathsf{NTK}$ also reveals a qualitative difference between input and output bias: Whereas the term that is induced by $\mathbf{b}^{1}$ shows attenuated growth due to the multiplication by the weights of initial scale $\sigma_{\mathbf{W}^{2}}\ll 1$, the output bias $\mathbf{b}^{2}$ immediately changes the output significantly. Loss curves which demonstrate the effect of different bias terms are displayed in Fig. 4.

To recapitulate this section: We first rephrased a learnable bias term in the architecture as a shared feature in the input data. We then found that the associated singular value in Eq. 2 drives the learned network function towards the OCS (Eq. 5). Finally, we proved that the bias affects early learning. We furthermore show that this tendency is even more pronounced for bias terms in later layers of the network. Overall, these results demonstrate how architectural bias terms induce early OCS learning.

###### Proposition 3 (NTK of linear networks with bias terms) .

## 5 Correspondence of linear networks, complex models, and humans

Figure: Figure 5: Early response bias towards the OCS across learners in the hierarchical learning task. Dashed red line indicates chance performance. Dashed vertical grey lines indicates breaks between days for human learners. Colour code as in Fig. 3.
Refer to caption: x5.png

In this section, we demonstrate how human learners, linear networks, and non-linear architectures show strong correspondences in their early learning on the hierarchical task displayed in Fig. 1.

Setup. The hierarchical learning task has previously been used extensively in the study of semantic cognition [] and requires learners to develop a hierarchical one-to-many mapping as seen in Fig. 1B. We adapted the task for human learners while maintaining the underlying structure: Input stimuli were represented as different classes of planets and output labels were represented as a set of plant images (see Fig. 1E and Fig. 8). We also trained CNNs as in Section 4. Importantly, the hierarchical structure results in a non-uniform distribution of labels with average labels equal to $\mathbf{y}_{ocs}$.
Human learners received supervised training over three days. A full description of the experimental paradigm is given Section A.2. We then compute correct-rejection scores (TNR) as in Section 4 while splitting performance across the hierarchical levels (details of metric in Section A.4).

Neural networks produced continuous outputs in $\mathbb{R}^{N_{out}}$ while humans responded via discrete button clicks in $\{0,1\}^{N_{out}}$ . To compare networks to humans we discretize network outputs by stochastically mapping continuous responses into $\{0,1\}^{N_{out}}$. Procedure details are given in Section A.3.

Results. The key results of our experiments are presented in Fig. 5. Intriguingly, we find that human learners, linear networks, and CNNs all display characteristic early response biases. Note that chance TNR is equal between all three levels of the hierarchy. Biological as well as artificial learners display an initial "drop" in TNR at the top level of the output hierarchy. The result indicates a general lack of specificity and an overly liberal response criterion for output labels on the top level of the hierarchy.
To appreciate the significance of this result it is important to understand that the task can be learned without the development of these early response biases: In particular, linear networks without bias terms do not show this behaviour (see Section A.7). Surprisingly, the human response signature demonstrates that these learners, just as artifical networks, display an early bias towards the OCS. We conjecture that early learning of the OCS might be a universal phenomenon that emerges during error-corrective training. We replicate the human result with a second cohort of learners in Section A.2.
Notable is also the difference between shallow and deep linear networks. Response biases seem more transient in shallow networks and appear to more closely mirror human learners. However, quantitative comparisons are challenging due to inherently differing learning timescales.

## 6 Generic input correlations can equivalently drive OCS learning

We have established how the earliest phase of learning in linear networks is driven by the OCS. Crucially, in linear networks OCS learning hinges on bias terms in the network architecture. However, in non-linear architectures, such as CNNs, the network is driven towards the OCS even in the absence of bias terms (Fig. 6B, Top). The appearance of the data term $\mathbf{X}^{T}\mathbf{X}$ in Proposition 3 suggests an equivalent effect that is induced by the data itself.

###### Corollary 1 (Input correlations induce early OCS) .

If $\mathbf{1}_{N}$ is an eigenvector of the data similarity matrix $\mathbf{X}^{T}\mathbf{X}$ with non-degenerate eigenvalue $s_{0}$, then the OCS response during early learning will be driven according to its magnitude.

This statement follows directly from the joint diagonalisation of Eq. 1 and subsequent projection onto the OCS $\mathbf{y}_{ocs}$. We furthermore hypothesize that neural networks will be driven towards the OCS if training data contains more generic input correlations where $\mathbf{1}_{N}$ is not an exact eigenvector.

Setup. We trained CNNs on the hierarchical task in Section 4. Inputs were given by eight randomly sampled classes of MNIST (Fig. 6B, Top). To isolate the effect of input correlations we created a second dataset where randomly sampled classes of MNIST were copied on orthogonal subspaces of a larger image (Fig. 6B, bottom). Importantly, this procedure removes all between-class correlations.

Figure: Figure 6: Distance from the OCS. A Linear networks with (top) and without (bottom) bias terms. B CNNs without bias terms. Top normal inputs. Bottom "orthogonal" inputs.
Refer to caption: x6.png

Results. The main result of our experiment is displayed in Fig. 6. CNNs which learn from standard MNIST images are strongly driven towards the OCS. In contrast, early dynamics for the "orthogonal" MNIST do not display this tendency. Strikingly, the early dynamics with standard MNIST classes are highly similar to those observed in linear network with bias terms, while the dynamics for the latter task resemble those seen in the linear network without this feature.
To verify that generic input correlations are indeed causing these differences we explore the eigenspectrum of the data correlation matrices. We sample 100 images from all 10 classes and compute a correlation matrix $\mathbf{X}^{T}\mathbf{X}$ from flattened images. First, we find that the eigenspectrum for standard MNIST images is dominated by a single eigenvector (Fig. 7, top-left). In contrast, the eigenspectrum of the orthogonal MNIST task does not display this property (Fig. 7, top-center). Further, recall that input bias terms lead to a non-degenerate constant eigenvector $\mathbf{1}_{N}$ in the input correlation matrix (Section 4). Similarly, we find that the first eigenvector $\mathbf{v}_{1}$ of $\mathbf{X}^{T}\mathbf{X}$ is indeed highly aligned to $\mathbf{1}_{N}$ (Fig. 7, right), whereas this is not the case in the orthogonal MNIST. We additionally show similar results for CIFAR-10 and CelebA. Theoretical considerations suggest that these correlations originate from an approximate symmetry in the data (see Section A.5.6) and might hence be ubiquitous beyond these datasets.

Overall, we here demonstrated that early learning of the OCS can be driven by properties of the architecture (bias terms) or data (input correlations). Our results also highlight that input correlations are a common feature of standard datasets: Early learning of the OCS might be inevitable in many settings. To see a practical implication of these results we briefly discuss fairness implications of OCS learning in Section A.10.

Figure: Figure 7: Dataset eigenspectra and constancy of first eigenvector. Left: Eigenspectra of $\mathbf{X}^{T}\mathbf{X}$ for different datasets. Right: Alignment of first eigenvector in $\mathbf{X}^{T}\mathbf{X}$ with the constant vector $\mathbf{1}_{N}$.
Refer to caption: x7.png

###### Corollary 1 (Input correlations induce early OCS) .

## 7 Discussion

We found that the inclusion of bias terms in linear networks shifts early learning towards the OCS, even when initialized at zero. We demonstrated that early, input-independent simplicity biases occur in practice, affecting both non-linear networks and human learners. Our contribution complements prior work on simplicity biases by highlighting factors that drive networks in the earliest stages of learning; connecting input, output, and architecture. Our findings highlight how simple linear networks can serve as useful tools to investigate simplicity biases in significantly more complex systems.
A limitation of our work is the focus on qualitative comparisons between linear networks and non-linear systems. We chose linear networks to allow for a rigorous description of the dynamics of learning. Methods from mean-field theory may provide a more precise tool to analyze a wider range of systems directly. Secondly, the ambiguity between architecture and data in driving the OCS does not allow us to determine the underlying mechanism in human learners. Future studies might address this limitation by manipulating correlations in stimuli or by recording of neural data. Our proposed framework furthermore lends itself to theoretically studying the effect of noise on the network, for example by random matrix theory. We leave this investigation for future work.

## Acknowledgments and Disclosure of Funding

We thank Satwik Bhattamishra, Aaditya Singh, Clémentine Dominé, Lukas Braun, Devon Jarvis, and Kevin Huang for useful feedback, discussions, and comments on earlier versions of this work.
This work was funded by a European Research Council (ERC) Consolidator Award (725937) to C.S., a Wellcome Trust Discovery Award (227928/Z/23/Z) to C.S., and a UKRI ESRC Grand Union Doctoral training partnership stipend awarded to J.R. This work was also supported by a Schmidt Science Polymath Award to A.S., and the Sainsbury Wellcome Centre Core Grant from Wellcome (219627/Z/19/Z) and the Gatsby Charitable Foundation (GAT3850). A.S. is a CIFAR Azrieli Global Scholar in the Learning in Machines & Brains program.

## Author Contributions

The conceptualisation of the project was developed by J.R. with C.S. and A.S. providing supervision. J.R. is responsible for all empirical results in networks, the human experiment, and initial theoretical ideas. J.B. and J.R. collaboratively devised the main theoretical results in the paper.
J.B. primarily developed the theoretical presentation in the appendix.
The initial draft was written by J.R. with further iterations and appendices written collaboratively by J.B. and J.R. All authors contributed to polishing of the draft and C.S. and A.S. provided supervision on all aspects of the project.

## Appendix A Appendix / supplemental material

### A.1 Overview

Our appendix has the following sections:

- •
In Section A.2, we describe the human experiment in more detail and show the results of a replication in a second cohort. We furthermore report statistical tests and describe ethical considerations.
- •
In Section A.3, we outline how we bring neural network and human responses displayed in Section 5 into a common space for direct comparison.
- •
In Section A.4, we outline how we compute the TNR rates $f^{tn}$ used in Section 4 and Section 5.
- •
In Section A.5, we provide additional theoretical derivations and remaining proofs to the statements in the main text.
- •
In Section A.6, we show OCS signatures in shallow networks with bias terms.
- •
In the short Section A.7, we show how linear networks without bias terms behave on the task in Section 5.
- •
In Section A.8, we describe hyperparameters, datasets, and further training details used for our CNN experiments.
- •
In Section A.9, we describe the results of additional experiments investigating early emergence of the OCS in non-linear models.
- •
In Section A.10 we show an additional solvable case of linear networks with bias terms under class imbalance.

### A.2 Human learning experiment

We directly translated the hierarchical task setup used by Saxe et al. [] into an experimental paradigm. Our design attempts to stay as close to the original task structure used for neural networks as possible. We designed the task as a mapping from 8 distinct input stimuli represented as planets to a set of 3 associated output stimuli represented as plants (see Fig. 8, left).

Figure: Figure 8: Human task design. Left: Hierarchical learning task, adapted for human participants. Centre: Trial structure as experienced by human participants. Right: Example screen during response period (top), Example screen during feedback period (bottom).
Refer to caption: x8.png

In the task, participants had to learn to associate which outputs properties are associated with each input. Unbeknownst to the participants we imposed a hierarchical structure on output targets (Fig. 8, left). In the structure some output labels are associated with more than one input. As a control for analyses we also included an additional control input-output pair (similarly represented by a planet and a plant; not shown here and excluded from current analysis). We recruited a cohort of 10 subjects that were trained over the course of three days with one daily session. The cohort was recruited as part of a larger neuroimaging experiment but our analysis presented here is exclusive to behavioural results. We further replicated our results in a second cohort of 46 human subjects recruited via the online platform Prolific (prolific.com). Results of the replication of the study can be seen in Fig. 9.

Figure: Figure 9: The human replication cohort. While learning is slower, the qualitative pattern indicating reliance on the OCS is replicated. Left: TNR rate. Right: TPR rate.
Refer to caption: x9.png

Each day of at home training consisted of 8 blocks of training with 22 trials each (160 standard trials and 16 control trials) which lasted about one hour. The trial structure during training is shown in Fig. 8, centre. During training trials, subjects were shown the stimulus on screen and were required to press three buttons, presented below the planet image (Fig. 8, right). The subjects received fully informative feedback on each trial and were forced to repeat the trial in the case of incorrect selection until the correct properties were selected. The location of buttons was shuffled on screen for each trial and for each forced repetition. For each button clicked correctly on their first attempt participants received a bonus point. We displayed a block-wise bonus in the corner of the screen throughout the task. Participants were payed slightly above local minimum wage as a baseline and received a substantial performance dependent bonus (on average about one-third of the baseline pay). We include a screenshot of the initial instructions in Fig. 10. Beyond this initial instruction screen participants received more nuanced instructions about clicking of buttons and feedback in the beginning of the task.

Statistical tests. While our focus is on qualitative patterns in human behaviour, we compute statistical tests on the TNR rates for human results seen in the main text (Fig. 5). We averaged all blocks in a given day and performed a two-way repeated measures ANOVA to assess the effect of day and hierarchy level on TNR. The two-way repeated measures ANOVA revealed significant main effects of day $F(2,18)=57.22,\>p<.0001,\>\eta^{2}=.25\text{ and level }F(2,18)=6.25,\>p=.033,
\>\eta^{2}=.18$. Beyond this we also found a significant interaction of day and level $F(4,36)=9.795,\>p=.0056,\>\eta^{2}=.042$. A Mauchly test indicated that the assumption of sphericity had been violated for level $\chi^{2}=.03,\>p<.5\text{ and the interaction term }\chi^{2}=.006,\>p<.5$. Significance values are reported with Greenhouse-Geisser correction. The results confirm that performance between levels are significantly different depending on day and hierarchical level.

Ethical considerations. Human participants performed a simple, computerised learning task without the collection of personal identifiable information or substantial deception. Human data collection was handled strictly in line with institutional guidelines and under institutional review board approval. We obtained informed consent for each participant before commencing the study. We highlighted that participants could withdraw at any time without penalty or loss of compensation by simply exiting full-screen or informing the experimenter. We provided contact emails in the case of concern or questions. Data was handled in a strictly anonymised format and stored on password secured devices. Participants were payed above minimum wage for their country of origin.

Figure: Figure 10: Initial instructions received by participants after the collection of informed consent.
Refer to caption: extracted/5690185/Supplemet_figs/instructions.png

### A.3 Discretizing network responses for comparison to humans

Discretization. In our task, neural networks produced continuous outputs. This is distinct from human learners who were required to give discrete responses. We now describe the discretization that allows us to compare human and network responses. Fundamentally, we conceptualise inference as a noisy process by which responses are sampled from a distribution over output labels. That is, we treat outputs from our linear network as logits. We first feed network outputs $\mathbf{\hat{y}}_{i}$ through a softmax function with temperature 0.2 and subsequently sample three responses without replacement. The procedure maps continuous outputs $\mathbf{\hat{y}}_{i}$ to binary responses vectors in $\{0,1\}^{N_{out}}$.

Expected solutions. Here we describe the derivation of expected solutions used in Fig. 5, dashed lines. The derivation of these "expected responses" under the sampling procedure allows to make the reliance of network responses on the exact solutions in Section 3 clear.

Consider network outputs $\hat{\mathbf{y}}_{i}(t)=\mathbf{W}^{2}(t)\mathbf{W}^{1}(t)\mathbf{x}_{i}$. We transform these outputs through a softmax function $\sigma_{\beta}:\mathbb{R}^{N_{out}}\rightarrow(0,1)^{N_{out}}$. Let $S=\{s_{1},s_{2},s_{3}\}$ denote the set of three unique response indices sampled from $\sigma_{\beta}(\hat{\mathbf{y}}_{i}(t))$ without replacement, where $s_{n}\in\{1,2,\ldots,N_{out}\}$ for $n=1,2,3$, and all $s_{n}$ are hence distinct. The probability distribution $\sigma_{\beta}(\hat{\mathbf{y}}_{i}(t))$ is dependent on time $t$, therefore denote the produced probability of $S$ as $P_{t}(S)$. For each of these sets $S$ we can compute an associated TNR for each of the $k\in\{1,2,3\}$ levels in the hierarchy. We denote this random variable as $X^{k}_{S}$. We can then compute expected solutions to inference behaviour as

$$ $\mathbb{E}_{t}[X^{k}_{S}]=\sum_{S\subseteq\{1,2,...,m\},\,|S|=3}P_{t}(S)X^{k}_ {S}$ (6) $$

### A.4 TNR rate/Correct-rejection score

Here we describe the metric used in the bottom panel of Fig. 3 and in Fig. 5. The metrics effectively describes TNR (correct-rejection scores). We use the metric on continuous network responses in $\mathbb{R}^{N_{out}}$ in Fig. 3. We also use the metric on discretised networks responses in $\{0,1\}^{N_{out}}$ and for human responses in $\{0,1\}^{N_{out}}$ in Fig. 5.

Given responses in $\mathbf{\hat{y}}$ or in and target vectors $\mathbf{y}\in\mathbb{R}^{N_{out}}$ the metric computes the alignment between target and response vectors while only focusing on zero entries in $\mathbf{y}$. Furthermore we compute the metric separately for the $k\in\{1,2,3\}$ separate levels of the hierarchy where the entries $s$ and $e$ denote relevant start and end indices of level $k$ in the vectors $\mathbf{\hat{y}}$ and $\mathbf{y}$. The metric is then computed as

$$ $f^{tn}_{k}(\mathbf{\hat{y}},\mathbf{y})=\frac{(\mathbf{1}_{N_{out}}-\mathbf{ \hat{y}})_{s:e}^{T}\,(\mathbf{1}_{N_{out}}-\mathbf{y})_{s:e}}{\sum^{e}_{i=s}( \mathbf{1}_{N_{out}}-\mathbf{y})_{i}},$ (7) $$

where $s:e$ is a "slicing" notation that takes the subvector between indices $s$ and $e$.

If for all desired entries of 0 0 in $\mathbf{y}$ the vector $\mathbf{\hat{y}}$ is equal to 0 0 the metric will be at 1. Correspondingly if entries in $\mathbf{\hat{y}}$ are larger than zero the metric $f^{tn}_{k}(\mathbf{\hat{y}},\mathbf{y})$ will decrease. Thus, the metric measures wrong beliefs about the presence of target labels across the different levels of the hierarchy.

### A.5 Additional theoretical results and proofs

#### A.5.1 Equivalence of bias terms

In this section, we give more detail on the method used in in Section 3 of how to reformulate a bias term in terms of the network weights and a constant feature in the input.

Consider a network with an explicit input bias term $\mathbf{b}^{1}$,

$$ $\hat{\mathbf{y}}=\tilde{\mathbf{W}}^{1}\tilde{\mathbf{x}}+\tilde{\mathbf{b}}^{1}$ $$

This is equivalent to introducing a constant component to the vector
$\mathbf{x}$,

$$ $\tilde{\mathbf{x}}\rightarrow\mathbf{x}\coloneqq\left[\begin{array}[]{c}1\\ \tilde{\mathbf{x}}\end{array}\right],$ $$

and using the network

$$ $\hat{\mathbf{y}}=\mathbf{W}^{1}\mathbf{x},$ $$

as we can write

$$ $\displaystyle\left(\mathbf{W}^{\mathbf{{1}}}\mathbf{x}\right)_{m}$ $\displaystyle=\sum_{j=0}^{N_{in}}W_{mj}^{1}x_{j}$ $\displaystyle=W_{m0}^{1}1+\sum_{j=1}^{N_{in}}W_{mj}^{1}x_{j}$ $\displaystyle=W_{m0}^{1}1+\sum_{j=0}^{N_{in}-1}\tilde{W}_{mj}^{1}\tilde{x}_{j}$ $\displaystyle\equiv b_{m}^{1}+\sum_{j=0}^{N_{in}-1}\tilde{W}_{mj}^{1}\tilde{x} _{j}.$ $$

In order to match a given i.i.d. initialization $b_{m}^{1}\sim\mathcal{N}\left(0,\,\sigma_{b}^{2}\right)$
where $\sigma_{b}\neq\sigma_{w}$, the component that needs to be
added to $\tilde{\mathbf{x}}$ to get equivalence needs to be $\sigma_{b}/\sigma_{w}$.

#### A.5.2 Learning dynamics for bias terms

We here derive analytical expressions for the learning speeds of input
and output bias terms for a two-layer deep linear network discussed in the
main text,

$$ $\hat{\mathbf{y}}=\mathbf{W}^{2}\left(\mathbf{W}^{1}\mathbf{x}+\mathbf{b}^{1} \right)+\mathbf{b}^{2}.$ $$

We decompose $\mathbf{W}^{2}=\mathbf{UA}^{(2)}\mathbf{R}^{(2)}$ and $\mathbf{W}^{1}=\mathbf{R}^{(1)}\mathbf{A}^{(1)}\mathbf{V}$
by means of a singular value decomposition (SVD). We here make the assumption of balancedness
$\mathbf{W}^{1}(0)\mathbf{W}^{1T}(0)=\mathbf{W}^{2T}(0)\mathbf{W}^{2}(0)$ [] at the beginning
of training, which implies $\mathbf{R}^{(2)}\mathbf{S}^{(2)2}\mathbf{R}^{(2)T}=\mathbf{R}^{(1)}\mathbf{S}^
{(1)2}\mathbf{R}^{(1)T}$.
For clarity, we further assume the simplification

$$ $\mathbf{R}^{(2)T}=\mathbf{R}^{(1)}\eqqcolon\mathbf{R},\>\mathbf{A}^{(2)}= \mathbf{A}^{(1)}\eqqcolon\sqrt{\mathbf{A}}.$ $$

We here just state these relations without further comment to complement the respective derivation for the weights in []. This decomposition then allows to rewrite the gradients.

##### Input bias term

$$ $\displaystyle\tau\frac{d}{dt}\mathbf{b}^{1}$ $\displaystyle=\nabla_{\mathbf{b}^{1}}\mathcal{L}$ $\displaystyle=\left(\mathbf{y}-\hat{\mathbf{y}}\right)^{T}\mathbf{W}^{2}$ $\displaystyle=\left(\mathbf{y}-\left(\mathbf{W}^{2}\left(\mathbf{W}^{1}\mathbf {x}+\mathbf{b}^{1}\right)+\mathbf{b}^{2}\right)\right)^{T}\mathbf{W}^{2}$ $\displaystyle\mathbb{E}_{\mathbf{x}}$ $\displaystyle\rightarrow\left(\bar{\mathbf{y}}-\mathbf{W}^{2}\left(\mathbf{W}^ {1}\bar{\mathbf{x}}+\mathbf{b}^{1}\right)-\mathbf{b}^{2}\right)^{T}\mathbf{W}^ {2}$ $\displaystyle=\left(\bar{\mathbf{y}}-\mathbf{UAV}\bar{\mathbf{x}}-\mathbf{U \sqrt{\mathbf{A}}\mathbf{R}b}^{1}-\mathbf{b}^{2}\right)^{T}\mathbf{U}\sqrt{ \mathbf{A}}\mathbf{R}^{T}$ $\displaystyle=\bar{\mathbf{y}}^{T}\mathbf{U}\sqrt{\mathbf{A}}\mathbf{R}^{T}- \bar{\mathbf{x}}^{T}\mathbf{V}^{T}\mathbf{A}\mathbf{R}^{T}-\mathbf{b}^{1T} \mathbf{R}\mathbf{A}\mathbf{R}-\mathbf{b}^{2T}\mathbf{U}\sqrt{\mathbf{A}} \mathbf{R}^{T}$ $\displaystyle=\left(\mathbf{Y}\mathbf{1}_{N}\right)^{T}\mathbf{U}\sqrt{\mathbf {A}}\mathbf{R}^{T}-\left(\mathbf{X}\mathbf{1}_{N}\right)^{T}\mathbf{V}^{T} \mathbf{A}\mathbf{R}-\mathbf{b}^{1T}\mathbf{R}\mathbf{A}\mathbf{R}^{T}-\mathbf {b}^{2T}\mathbf{U}\sqrt{\mathbf{A}}\mathbf{R}^{T}.$ $$

Here, we denoted the expectation over the data samples as $\mathbb{E}_{\mathbf{x}}$. Projecting from the right with $\mathbf{R}_{\alpha}\in\mathbb{R}^{N_{\text{hidden}}}$
gives

$$ $\tau\frac{d}{dt}\left(\mathbf{b}^{1T}\mathbf{R}_{\alpha}\right)=\bar{\mathbf{y }}^{T}\mathbf{U}_{\alpha}\sqrt{a_{\alpha}}-\bar{\mathbf{x}}^{T}\mathbf{V}_{ \alpha}^{T}a_{\alpha}-\mathbf{b}^{1T}\mathbf{R}_{\alpha}a_{\alpha}-\mathbf{b}^ {2T}\mathbf{U}_{\alpha}\sqrt{a_{\alpha}}.$ (8) $$

##### Output bias term

$$ $\displaystyle\tau\frac{d}{dt}\mathbf{b}^{2}$ $\displaystyle=\left(\mathbf{y}-\left(\mathbf{W}^{2}\left(\mathbf{W}^{1}\mathbf {x}+\mathbf{b}^{1}\right)+\mathbf{b}^{2}\right)\right)$ $\displaystyle\mathbb{E}_{\mathbf{x}}$ $\displaystyle\rightarrow\bar{\mathbf{y}}-\mathbf{W}^{2}\left(\mathbf{W}^{1} \bar{\mathbf{x}}+\mathbf{b}^{1}\right)-\mathbf{b}^{2}$ $\displaystyle=\bar{\mathbf{y}}-\mathbf{UAV}\bar{\mathbf{x}}-\mathbf{U}\sqrt{ \mathbf{A}}\mathbf{R^{T}b}^{1}-\mathbf{b}^{2}$ $\displaystyle=\mathbf{Y}\mathbf{1}_{N}-\mathbf{UAVX}\mathbf{1}_{N}-\mathbf{UR^ {T}b}^{1}-\mathbf{b}^{2}.$ (9) $$

Notably, the derivative in Eq. 8 is proportional to
the singular vectors of the weights $a_{\alpha}$, so that its growth
is attenuated, analogous to the sigmoidal growth in deep linear networks []. In contrast, the learning signal $\frac{d}{dt}\mathbf{b}^{2}$ in Eq. 9
is not affected by the initialization of the weights and is hence
$\mathcal{O}(1)$ already at the beginning of learning, reminiscent of shallow networks.

#### A.5.3 Neural tangent kernel

In this section, we review the neural tangent kernel (NTK). This
object is useful because it directly describes the learning dynamics
in output space $\hat{y}$ [, ] as we briefly
demonstrate here. We then calculate the NTK for our specific architecture
to yield Proposition 3 in the main text. The following makes use of Einstein summation
convention.

For a vector-valued model $\hat{\mathbf{y}}(\mathbf{x})\in\mathbb{R}^{N_{out}}$ parametrized
by a parameter vector $\theta$, the evaluation on sample $\mathbf{x}_{i}$
from training data at $\mathbf{x}_{i^{\prime}}$ evolves as

$$ $\displaystyle\tau\frac{d}{dt}y_{m}(\mathbf{x}_{i})$ $\displaystyle=\sum_{k}\frac{dy_{m}(\mathbf{x}_{i})}{d\theta^{k}}\frac{d\theta^ {k}}{dt}$ (10) $\displaystyle=-\eta\sum_{k}\frac{dy_{m}(\mathbf{x}_{i})}{d\theta^{k}}\frac{d \mathcal{L}}{d\theta^{k}}$ (11) $\displaystyle=-\eta\left[\sum_{k}\frac{dy_{m}(\mathbf{x}_{i})}{d\theta^{k}} \frac{dy_{m^{\prime}}(\mathbf{x}_{i^{\prime}})}{d\theta^{k}}\right]\frac{d \mathcal{L}}{dy_{m^{\prime}}}(\mathbf{x}_{i^{\prime}})$ (12) $\displaystyle\eqqcolon-\eta\,\mathsf{NTK}_{mm^{\prime}}(\mathbf{x}_{i},\, \mathbf{x}_{i^{\prime}})\left(y_{m^{\prime}}(\mathbf{x}_{i^{\prime}})-\hat{y}_ {m^{\prime}}(\mathbf{x}_{i^{\prime}})\right),$ (13) $$

where we used the chain rule and that the parameters update according
to gradient descent with learning rate $\eta$, $\frac{d\theta^{k}}{dt}=-\eta\frac{d\mathcal{L}}{d\theta^{k}}$. The last line has defined the NTK.
We set $\eta=1$ in the main text for simplicity, as it does not change
trajectory and thereby convergence in the case of gradient flow. In
addition, we evaluated $\frac{d\mathcal{L}}{dy_{m^{\prime}}}(\mathbf{x}_{i^{\prime}})$ for
the case of MSE loss $\mathcal{L}(\mathbf{x}_{i^{\prime}})=\nicefrac{{1}}{{2}}\sum_{m^{\prime}}\left
(y_{m^{\prime}}(\mathbf{x}_{i^{\prime}})-\hat{{y}}_{m^{\prime}}(\mathbf{x}_{i^
{\prime}})\right)^{2}$.
The last line of Eq. 10 reveals that the NTK acts as an
effective learning rate, as noted by Roberts et al. [].

We here consider a two-layer linear architecture $\hat{Y}_{m}^{i}(\mathbf{X})=W_{mk}^{2}\left(W_{kj}^{1}X_{j}^{i}+b_{k}^{1}
\right)+b_{m}^{2}$
where we adopt Einstein summation convention over repeated indices.
The parameters are $\theta^{k}\in\left\{\mathbf{W}^{2},\mathbf{W}^{2},\mathbf{b}^{1},\mathbf{b}^{2
}\right\}$.
Herein, $m$ indexes output features and $i$ indexes data samples.
The non-zero gradients are

$$ $\displaystyle\frac{d\hat{Y}_{m}^{i}}{dW_{mk}^{2}}$ $\displaystyle=W_{kj}^{1}X_{j}^{i}+b_{k}^{1}$ $\displaystyle\frac{d\hat{Y}_{m}^{i}}{db_{m}^{2}}$ $\displaystyle=1_{m}$ $\displaystyle\frac{d\hat{Y}_{m}^{i}}{dW_{kj}^{1}}$ $\displaystyle=W_{mk}^{2}X_{j}^{i}$ $\displaystyle\frac{d\hat{Y}_{m}^{i}}{db_{k}^{1}}$ $\displaystyle=W_{mk}^{2}1_{k}.$ $$

Inserting this into Eq. 10, we get

$$ $\displaystyle\mathsf{NTK}_{m_{1}m_{2}}(X_{\;j}^{i_{1}},X_{j}^{\,i_{2}})$ $\displaystyle=I_{m_{1}m_{2}}\>\left(X_{\;j^{\prime}}^{i_{1}}W_{j^{\prime}k}^{1 }W_{kj^{\prime\prime}}^{1}X_{j^{\prime\prime}}^{\,i_{2}}+b_{k}^{1}b_{k}^{1}\right)$ $\displaystyle+1_{m_{1}}1_{m_{2}}$ $\displaystyle+W_{m_{1}k}^{2}W_{km_{2}}^{2T}\>X_{\;j}^{i_{1}}X_{j}^{\,i_{2}}$ $\displaystyle+W_{m_{1}k}^{2}1_{k}1_{k}W_{km_{2}}^{2}.$ $$

or in matrix notation, collecting similar terms

$$ $\displaystyle\mathsf{NTK}(\mathbf{X},\mathbf{X})$ $\displaystyle=\mathbf{I}_{N_{out}}\otimes\mathbf{X}^{T}\mathbf{W}^{1T}\mathbf{ W}^{1}\mathbf{X}+\mathbf{b}^{1T}\mathbf{b}^{1}$ $\displaystyle+\mathbf{1}\mathbf{1}^{T}\,\otimes\,\underbrace{\mathbf{1}\mathbf {1}^{T}}_{\leftrightarrow\mathbf{b}^{2}}$ $\displaystyle+\mathbf{W}^{2}\mathbf{W}^{2T}\,\otimes\,\Bigl{(}\mathbf{X}^{T} \mathbf{X}+\underbrace{\mathbf{11}^{T}}_{\leftrightarrow b^{1}}\Bigr{)}$ $\displaystyle\in\mathbb{R}^{N_{out}\times N_{out}}\,\otimes\,\mathbb{R}^{N \times N},$ $$

where the left hand side operator in the tensor product $\otimes$
is acting in output space $m_{1}m_{2}$, whereas the right hand side
operator acts in pattern space $i_{1}i_{2}$. The notation $\leftrightarrow\mathbf{b}$
indicates that a term is due to the bias term. To illustrate this, the NTK acts on the set of labels $\mathbf{Y}\in\mathbb{R}^{N_{in}\times N}$ as follows:

$$ $\left(\mathsf{NTK}(\mathbf{X},\mathbf{X})\,\mathbf{Y}\right)_{i}^{m}=\sum_{m^{ \prime}}^{N_{out}}\sum_{i^{\prime}}^{N}\mathsf{NTK}(X_{i},X_{i^{\prime}})^{mm^ {\prime}}Y_{i^{\prime}}^{m^{\prime}}.$ (14) $$

For simplicity, we approximate $\mathbf{W}^{2}(0)\mathbf{W}^{2T}(0)=\sigma_{\mathbf{W}}^{2}I_{N_{out}}$
and $\mathbf{W}^{1T}(0)\mathbf{W}^{1}(0)=\sigma_{\mathbf{W}}^{2}I_{N_{in}}$, which approximately holds for
initialization

$$ $\mathbf{W}^{1}(0)\sim\mathcal{N}\left(0,\,\sigma_{\mathbf{W}}^{2}/N_{hid} \right),\,\mathbf{W}^{2}(0)\sim\mathcal{N}\left(0,\,\sigma_{\mathbf{W}}^{2}/N_ {hid}\right),\,\mathbf{b}^{1}=0,\,\mathbf{b}^{2}=0$ $$

where $N_{hid}$ is the size of the hidden layer and both $N_{in}$ and $N_{hid}$ are large. This leaves

$$ $\displaystyle\mathsf{NTK}(\mathbf{X},\mathbf{X})$ $\displaystyle=\sigma_{\mathbf{W}}^{2}\mathbf{I}_{N_{out}}\,\otimes\,\Bigl{(}2 \mathbf{X}^{T}\mathbf{X}+\underbrace{\mathbf{11}^{T}}_{\leftrightarrow b^{1}} \Bigr{)}\;+\;\mathbf{11}^{T}\,\otimes\,\underbrace{\mathbf{11}^{T}}_{ \leftrightarrow\mathbf{b}^{2}}.$ $$

### Proofs

#### A.5.4 Feasibility of closed-form solution

See 1

###### Proof.

We would like to know when the right singular vectors $\mathbf{V}$ (denote as $\mathbf{V}^{yx}$ here for clarity)
of $\mathbf{\Sigma}^{yx}=\mathbf{U}^{yx}\mathbf{S}^{yx}\mathbf{V}^{yx}$
match these of $\mathbf{\Sigma}^{x}=\mathbf{U}^{x}\mathbf{S}^{x}\mathbf{V}^{x}$.
First, to reduce the problem to $\mathbf{V}^{yx}$, note
that $\mathbf{\Sigma}^{yxT}\mathbf{\Sigma}^{yx}=\mathbf{V}^{yx}\mathbf{S}^{yx2}
\mathbf{V}^{yx}$,
so that what remains to show is $\left[\mathbf{\Sigma}^{yxT}\mathbf{\Sigma}^{yx},\,\mathbf{\Sigma}^{xx}\right]=0$,
where $\left[\mathbf{A},\,\mathbf{B}\right]\coloneqq\mathbf{AB}-\mathbf{BA}$
denotes the commutator between two matrices $\mathbf{A}$ and $\mathbf{B}$.
We compute the two terms as

$$ $\displaystyle\mathbf{\Sigma}^{yxT}\mathbf{\Sigma}^{yx}\mathbf{\Sigma}^{xx}$ $\displaystyle=\mathbf{X}\mathbf{Y}^{T}\mathbf{Y}\mathbf{X}^{T}\mathbf{X} \mathbf{X}^{T}$ $\displaystyle\mathbf{\Sigma}^{xxT}\mathbf{\Sigma}^{yxT}\mathbf{\Sigma}^{yx}$ $\displaystyle=\mathbf{XX}^{T}\mathbf{XY}^{T}\mathbf{YX}^{T}$ $$

The commutator vanishes if these terms match, which happens for the
simpler equality

$$ $\mathbf{Y}^{T}\mathbf{YX}^{T}\mathbf{X}=\mathbf{X}^{T}\mathbf{XY}^{T}\mathbf{Y},$ $$

or $\left[\mathbf{Y}^{T}\mathbf{Y},\,\mathbf{X}^{T}\mathbf{X}\right]=0$. The converse follows only if
the tranformation $\mathbf{X}\ldots\mathbf{X}^{T}$ in the former equation is invertible,
which is the case if a left inverse $\mathbf{X}^{-1}\mathbf{X}=\mathbf{I}_{N_{in}}$ exists.

∎

###### Proof.

#### A.5.5 OCS and shared properties correspond to each other

We here link the OCS and shared properties stand in close relation, as the eigenvector $\mathbf{1}_{N}$ represents properties that are shared across all data samples.
See 2

###### Proof.

$$ $\mathbf{XX}^{T}\bar{\mathbf{x}}=\left(\mathbf{XX}^{T}\right)\frac{1}{N}\mathbf {X}\mathbf{1}_{N}=\frac{1}{N}\mathbf{X}\left(\mathbf{X}^{T}\mathbf{X}\right) \mathbf{1}_{N}=\frac{1}{N}\mathbf{X}\lambda\,\mathbf{1}_{N}=\lambda\frac{1}{N} \mathbf{X}\mathbf{1}_{N}=\lambda\bar{\mathbf{x}}.$ $$

∎

###### Proof.

#### A.5.6 Constant data mode 𝟏 N subscript 1 𝑁 \mathbf{1}_{N} bold_1 start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT is related to symmetry in the data

This section gives proof sketches based on symmetry in the dataset that
are sufficient to make $\mathbf{1}_{N}$ an eigenvector to $\mathbf{X}^{T}\mathbf{X}$ and $\mathbf{Y}^{T}\mathbf{Y}$,
and in particular hold for the dataset that we are considering. We anticipate that it is possible to formulate these statements in a more universal way by fully leveraging the cited literature.

The assumptions on symmetry should intuitively at least hold in an approximate manner for
many datasets, we expect that they indeed are the reason why we observe
a prevalence of $\mathbf{1}_{N}$, although they are not a necessary condition.

##### Continuously supported data 𝐱 ∈ ℝ N i ⁢ n 𝐱 superscript ℝ subscript 𝑁 𝑖 𝑛 \mathbf{x}\in\mathbb{R}^{N_{in}} bold_x ∈ blackboard_R start_POSTSUPERSCRIPT italic_N start_POSTSUBSCRIPT italic_i italic_n end_POSTSUBSCRIPT end_POSTSUPERSCRIPT

###### Proposition 4 (Continuous symmetry induces $\mathbf{1}_{N}$ ) .

If the pairwise correlations $\mathbf{y}_{i}^{T}\mathbf{y}_{i^{\prime}}$
in a dataset are rotationally symmetric, its similarity matrix $\mathbf{Y}^{T}\mathbf{Y}$
has eigenvector $\mathbf{1}_{N}$. Note that this is a weaker assumption than
the data itself being symmetric.

###### Proof.

We assume that $\mathbf{Y},\mathbf{X}$ have been sampled from a ground truth data distribution
$p(\mathbf{y},\mathbf{x})$. If $p(\mathbf{y})$ is rotationally symmetric and
$\mathbf{X}$ is comprised of samples $\mathbf{x}$ that are uniformly distributed
on the hypersphere, we can introduce the kernel function $\mathbf{y}_{\mathbf{x}_{i}}^{T}\mathbf{y}_{\mathbf{x}_{i^{\prime}}}=k(\mathbf{
x}_{i},\,\mathbf{x}{}_{i^{\prime}})=k(\mathbf{R}_{l}\mathbf{x}_{i},\,\mathbf{R
}_{l}\mathbf{x}_{i^{\prime}})=k(\mathbf{x}_{i}^{T}\mathbf{x}_{i^{\prime}})$
for any $R_{l}$ that is a representation of the group of rotations
$G=\text{SO}(N_{in})$ that faithfully acts on the “subsampled”
hypersphere $\mathbf{X}$ comprised of vectors $\mathbf{x}\in\mathbb{R}^{N_{in}}$.
It therefore only depends on the pairwise input similarity (hence
sometimes called dot-product kernel). If follows that for all vectors
$\mathbf{v}(\mathbf{X})\in\mathbb{R}^{N}$ that are evaluations of the functions
of the sample points $\mathbf{X}$

$$ $\mathbf{Y}^{T}\mathbf{Y}\,\mathbf{v}=k(\mathbf{X}^{T}\mathbf{X})\,\mathbf{v}=k (\left(\mathbf{R}_{l}\mathbf{X}\right)^{T}\left(\mathbf{R}_{l}\mathbf{X}\right ))\,\mathbf{v}=\mathbf{R}_{k}^{T}k(\mathbf{X}^{T}\mathbf{X})\mathbf{R}_{l}\, \mathbf{v}\>\Leftrightarrow\>\left[\mathbf{Y}^{T}\mathbf{Y},\,\mathbf{R}_{l} \right]=0,$ $$

where $\left[\mathbf{A},\mathbf{B}\right]\eqqcolon\mathbf{AB}-\mathbf{BA}$ is the commutator between two matrices.

It follows that we must have for all rotations $\mathbf{R}_{l}$

$$ $\mathbf{R}_{l}\left(\mathbf{Y}^{T}\mathbf{Y}\,\mathbf{1}_{N}\right)=\mathbf{Y} ^{T}\mathbf{YR}_{l}\,\mathbf{1}_{N}=\mathbf{Y}^{T}\mathbf{Y}\,\lambda_{\mathbf {R}_{l}}\mathbf{1}_{N}=\lambda_{\mathbf{R}_{l}}\mathbf{Y}^{T}\mathbf{Y}\, \mathbf{1}_{N}$ $$

with eigenvalue $\lambda_{\mathbf{R}_{l}}=1$.

meaning that $\mathbf{Y}^{T}\mathbf{Y}\,\mathbf{1}_{N}$ is an eigenvector to *all* $\mathbf{R}_{l}$.
This can only be the case if $\mathbf{Y}^{T}\mathbf{Y}\,\mathbf{1}_{N}\propto\mathbf{1}_{N}$, as this is the
only vector of values on the sphere that is invariant under any rotations.

∎

We point out that it can be shown more generally with tools from functional
analysis that the full spectrum of this kernel operator $k$ are the
spherical harmonics if the data measure $p(\mathbf{x})$ is spherically
symmetric [], see [] for a modern
presentation with tools from calculus. As the first harmonic $\mathcal{Y}_{l=0,m=0}(\mathbf{x})$
is constant, it follows that also the constant function $\mathbf{1}(\mathbf{x})\equiv 1$
is an eigenfunction when drawing a finite set of samples from this
kernel.

###### Proposition 4 (Continuous symmetry induces 𝟏 N subscript 1 𝑁 \mathbf{1}_{N} bold_1 start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT ) .

###### Proof.

##### Data on a graph 𝐱 ∈ ℝ N i ⁢ n 𝐱 superscript ℝ subscript 𝑁 𝑖 𝑛 \mathbf{x}\in\mathbb{R}^{N_{in}} bold_x ∈ blackboard_R start_POSTSUPERSCRIPT italic_N start_POSTSUBSCRIPT italic_i italic_n end_POSTSUBSCRIPT end_POSTSUPERSCRIPT

We here prove that the former statement holds for the hierarchical
dataset that is discussed in the main text, i.e. that $\mathbf{1}_{N}$ is an
eigenvector to $\mathbf{Y}^{T}\mathbf{Y}$.

First, note that it is easy to convince oneself of this by writing down the
matrices explicitly: Then, as the rows are just permutations of one
another, $\mathbf{1}_{N}$ is immediately identified as an eigenvector, because
$\sum_{i^{\prime}}^{N}\mathbf{Y}_{i}^{T}\mathbf{Y}_{i^{\prime}}{1}_{i^{\prime}}
=\mathbf{Y}_{i}^{T}\bigl{(}\sum_{i^{\prime}}\mathbf{Y}_{i^{\prime}}\bigr{)}$
will then not depend on $i$ and hence be proportional to $\mathbf{1}_{N}$.

To connect with the former symmetry-based argument Section A.5.6,
we here however give a proof that is based on the symmetry in the data:

###### Proposition 5 (Discrete symmetry induces $\mathbf{1}_{N}$ ) .

Consider a connected Cayley tree graph with adjacency matrix $\mathbf{A}$
and nodes $\mathbf{x}_{i}$. Furthermore, let $\mathbf{R}_{l}\in G$ be an element
of a faithful representation of the symmetry group $G$ that acts
on the graph nodes $\mathbf{v}$, i.e. that leaves its adjacency matrix invariant, $\left[\mathbf{R}_{l},\mathbf{A}\right]=0\>\forall\,\mathbf{R}_{l}$.

If $\mathbf{Y}$ are labels associated with the leaf nodes $\mathbf{X}$ (the outermost
generation of the graph, see []) and there exists a
similarity function $k$ such that $\mathbf{y}_{\mathbf{x}_{i}}^{T}\mathbf{y}_{\mathbf{x}_{i^{\prime}}}=\mathbf{y}
_{\mathbf{R}_{l}\mathbf{x}_{i}}^{T}\mathbf{y}_{\mathbf{R}_{l}\mathbf{x}_{i^{
\prime}}}=k(d(\mathbf{x}_{i},\mathbf{x}_{i^{\prime}}))$
$\forall\,\mathbf{R}_{l}$ where $d$ is the geodesic distance on the graph,
$\mathbf{1}_{N}$ will be an eigenvector of $\mathbf{Y}^{T}\mathbf{Y}$.

###### Proof.

From the symmetry assumption on the labels, we again have for any
vector $\mathbf{v}$ of node loadings $\left[\mathbf{R}_{l},\,\mathbf{Y}^{T}\mathbf{Y}\right]\mathbf{v}=0\>\forall\,
\mathbf{R}_{l}\in G$.
From this, we find that

$$ $\mathbf{R}_{l}\left(\mathbf{Y}^{T}\mathbf{Y}\,\mathbf{1}_{N}\right)=\mathbf{Y} ^{T}\mathbf{Y}\,\mathbf{R}_{l}\mathbf{1}_{N}=\mathbf{Y}^{T}\mathbf{Y}\,\lambda _{\mathbf{R}_{l}}\mathbf{1}_{N}=\lambda_{\mathbf{R}_{l}}\mathbf{Y}^{T}\mathbf{ Y}\,\mathbf{1}_{N}\>\forall\,\mathbf{R}_{l}$ $$

This shows that $\mathbf{Y}^{T}\mathbf{Y}\,\mathbf{1}_{N}$ is an eigenvector of $\mathbf{R}_{l}$ with eigenvalue
$\lambda_{\mathbf{R}_{l}}=1$ for any element of the symmetry group. The only
vector $\mathbf{v}$ that is invariant under *all* symmetry operations
of the graph is the constant vector $\mathbf{1}_{N}$.

∎

We briefly point out the rich literature on spectral graph theory
(for example [, ]) that might allow making
statements about the nature of the eigenvalues and other eigenvectors
as a function of the graph topology. We expect that this is possible
because the literature in the continuous case discussed in the next paragraph bases their arguments
on the Laplacian on the sphere, an operator that can be extended to graphs as well.
We leave these exploration for future work.

###### Corollary 2 .

Because $k(\mathbf{X}^{T}\mathbf{X})\coloneqq\mathbf{X}^{T}\mathbf{X}$ defines a particular case of
input-output similarity mapping, $\mathbf{1}_{N}$ is also an eigenvector to
$\mathbf{X}^{T}\mathbf{X}$ under the former assumptions of uniform data distribution.

###### Proposition 5 (Discrete symmetry induces 𝟏 N subscript 1 𝑁 \mathbf{1}_{N} bold_1 start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT ) .

###### Proof.

###### Corollary 2 .

#### A.5.7 Constant data mode 𝟏 N subscript 1 𝑁 \mathbf{1}_{N} bold_1 start_POSTSUBSCRIPT italic_N end_POSTSUBSCRIPT is the leading eigenvector

Here, we prove that the constant eigenvector $\mathbf{1}_{N}$ which is responsible
for the OCS solution is associated with the leading eigenvalue
of the input-output correlation matrix and hence drives early learning.

See 1

###### Proof.

Let $\mathbf{1}_{N}$ be an eigenvector to both similarity matrices $\mathbf{X}^{T}\mathbf{X}$
and $\mathbf{Y}^{T}\mathbf{Y}$ associated with eigenvalue $\tilde{\lambda}$. Moreover,
let $\mathbf{X}^{T}\mathbf{X}$ and $\mathbf{Y}^{T}\mathbf{Y}$ have positive entries. Then, the Perron-Frobenius
theorem [] guarantees that $\tilde{\lambda}$
is indeed the leading eigenvector to $\mathbf{Y}^{T}\mathbf{Y}$, $\tilde{\lambda}\equiv\lambda_{0}=s_{0}^{2}$.

By Proposition 2, $\bar{\mathbf{x}}$ and $\bar{\mathbf{y}}$ are now also the leading
eigenvectors for $\mathbf{XX}^{T}$ and $\mathbf{YY}^{T}$. Because the
eigenvectors of $\mathbf{YY}^{T}$ and $\mathbf{XX}^{T}$ are the left and right singular
vectors of $\mathbf{\Sigma}^{yx}$, respectively, with the eigenvalues being
the squares of the singular values, it follows that

$$ $s_{0}\mathbf{u}_{0}\mathbf{v}_{0}^{T}=\sqrt{\lambda_{0}}\bar{\mathbf{y}}\bar{ \mathbf{x}}^{T}.$ $$

∎

###### Proof.

### A.6 Shallow network OCS learning

In this brief section we show the OCS signatures of shallow networks with bias terms. The result is displayed in Fig. 11. We see similar behavioural signatures to deep linear networks. However, the tendency to the OCS is more transient.

Figure: Figure 11: Early learning in shallow networks with bias terms approaches the OCS.
Refer to caption: x10.png

### A.7 TNR in linear networks without bias terms

In this short section we provide a supplemental figure relevant for our results in Section 5: We train deep and shallow linear networks without bias terms. The learning setting and computation of metrics are equivalent to results in Fig. 5. We display the result in Fig. 12. While networks learn the task, early, response biases are fully absent in these models.

Figure: Figure 12: True negative rates for linear networks without bias terms. We do not see characteristic response patterns observed in Fig. 5.
Refer to caption: x11.png

### A.8 CNN datasets and hyperparameters

Datasets used. We used and adapted different image datasets for our experiments with CNNs. While the main text focused on results obtained with a variant of MNIST we report further experiments we conducted to highlight the universality of early OCS learning.

- 1.
Hierarchical MNIST. We randomly sampled eight digit classes. We subsequently replaced standard one-hot labels with label vectors containing the hierarchical structure as seen in Fig. 1.
- 2.
Hieararchical CIFAR-10. We applied the same procedure and randomly sampled eight classes from CIFAR-10 and replaced one-hot labels as for the hierarchical MNIST.
- 3.
Imbalanced-binary-MNIST. While not described in the main text we also report results in a setting with standard one-hot target vectors. We randomly sample two MNIST classes for training. To assess the impact of the OCS in early learning we introduced class imbalanced by oversampling one of the two classes by a factor of two.
- 4.
Standard CelebA. We perform experiments on CelebA’s face attribute detection task. The task offers a natural testbed for early learning of the OCS as face attribute target labels form a non-uniform distribution as seen in Fig. 15, bottom. We also normalised images in the dataset before training.

Model details.
We trained a custom CNN with 3 convolutional layers (layer 1: 32 filters of size 5$\times$5; layer 2: 64 filters of size 3$\times$3; layer 3: 96 filters of size 3$\times$3), followed by 2 fully connected layers of sizes 512 and 256. Activation functions for all layers were chosen as ReLUs. The final layer of the model did not contain an activation function when training with squared error loss. In experiments with the class imbalanced-binary-MNIST and cross-entropy loss the final layer contained a softmax function as non-linearity. For experiments on CelebA the final layer contained sigmoid activation functions and we trained with a binary cross-entropy loss over all 40 labels.

Training details.
For our results on hierarchical MNIST we train models with minibatch SGD with a batch size of 16 and with a relatively small step size of 1e-4 to examine the early learning phase. For all experiments we used Xavier uniform initialisation []. Whenever we use bias terms in the model we initialize these as 0 in line with common practice. For our main experiments we train models using a simple squared error loss function. However, to demonstrate generality we repeat experiments for the case of class imbalance using a cross-entropy loss and binary cross-entropy in the case of CelebA. All experiments are repeated 10 times with different random seeds with the exception of CelebA where we used 5 different random seeds, we provide standard errors in all figures (shaded regions). For experiments on the hierarchical CIFAR-10, the class imbalanced MNIST, and CelebA we kept all parameters as above but we increase step size to 1e-3. We trained CNN models on an internal cluster on a single RTX 5000 GPU. Runs took less than one hour to complete.

### A.9 Additional experimental results

To understand the generality of OCS learning we plot the results of experiments examining early learning of the OCS in these models. We mostly restrict ourselves to plots as seen in Fig. 6 as we deem these figures most instructive.

Hieararchical CIFAR-10. We train on a hierarchical version of CIFAR-10. Where we randomly sample 8 classes from MNIST and replace target labels by hierarchical vectors as in Section 4. We find the key signatures of early OCS learning: We find early indifference, the reversion of performance metrics to the OCS, and a small initial distance of average response the to the OCS solution. The results mirror behaviour on the hierarchical MNIST shown in Fig. 3 and Fig. 6.

Figure: Figure 13: Early OCS learning CNNs trained on hierarchical CIFAR-10. left: Network outputs for a single output unit in response to all inputs $\mathbf{x}_{i}$. Centre: Performance metrics $f^{tn}$ (Section A.4). Right: Mean distance of network responses from OCS. Averages taken over every 10 batches for plotting.
Refer to caption: x12.png

Class-imbalance MNIST. We train on an imbalanced MNIST task as described in Section A.8. We plot the results for training with squared error and cross-entropy loss functions in Fig. 14. Both settings show reversion to the OCS. Note that average model outputs in the case of the cross-entropy loss start relatively close to outputs expected under the OCS. Despite this proximity the model is still driven towards the OCS solution. The results on this imbalanced case highlight potential fairness implications. Given that network have been found to revert to the OCS when generalising [], early learning in the OCS setting can transiently, but significantly disadvantage minority classes. We further highlight this point in a second solvable case of linear networks with bias terms in Section A.10.

Figure: Figure 14: Mean distance of network responses from OCS in CNNs trained on the imbalanced MNIST task. Averages taken over each batch.
Refer to caption: x13.png

Standard CelebA. We show distance from the OCS for the CelebA face attribute detection task in Fig. 15, top. CelebA provides a useful test for our hypothesis as attribute labels display natural imbalances. We highlight the strong non-uniformity of the majority attribute labels in Fig. 15, bottom. We again train networks in two variants: one with squared-error loss and one with binary cross-entropy loss applied over all 40 face attributes. With both loss functions network responses are driven towards the OCS in early learning. This case further highlights the universality of early OCS learning. OCS learning might be especially undesirable in this setting for fairness reasons as the model will be overly liberal in the prediction of the most common face attributes.

Figure: Figure 15: Top: Mean distance of network responses from the OCS in CNNs trained on the CelebA face attribute prediction task. Bottom: Marginal probabilities of CelebA face attributes.
Refer to caption: x14.png

### A.10 Linear networks under class imbalance

In this section, we describe a second case of a solvable linear network with bias terms. Our dataset consists of two examples where one example appears twice as frequently. We show the data used on the right side of Fig. 16. The minority class has two identifying labels, while this construction appears artificial, it allows for the application of Proposition 1 and solutions to learning dynamics from Section 2 apply.

The case is of particular practical relevance as it illustrates the impact of early OCS learning under class imbalance, a common problem in machine learning where datasets are often naturally imbalanced [, ]. In practice, these settings are often addressed through oversampling of minority classes [, ]. Empirical work by Ye et al. [] documented that neural networks initially fail to learn information about the minority class while classifying most minority examples as belonging to the majority class. Subsequent theoretical work by Francazi et al. [] demonstrated that the phenomenon is caused by competition between the optimisation of different classes.

Our work adds to this literature by providing dynamics in a case of gradient-based learning under class imbalance learning that is exactly solvable.
Our exact solutions highlight the potential role of early OCS learning in the initial failure to learn about minority classes. The OCS solution substantially biases early predictions towards the majority class as seen in Fig. 16, centre. The results also can be understood as solvable analogous to early reversion to the OCS seen in the Imbalanced-binary-MNIST setting in Fig. 14.
The results highlight the potential fairness implications of early OCS learning as the learning phase systematically biases the model against the minority classes.

Figure: Figure 16: Early learning of the OCS in linear networks under class imbalance.
Refer to caption: extracted/5690185/Supplemet_figs/imbalanced_linear_net.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 8
presentation: 3
questions: See comments above.
soundness: 3
strengths: 1. The study contributes to the broader understanding of simplicity biases in machine learning, specifically in the early learning phases.

2. The paper’s exact characterization of the learning dynamics of deep linear networks with bias terms is a novel contribution. This analysis is theoretically sound and provides new insights on how simplicity emerges during the initial training phase.

3. The theoretical findings are supported by learning experiments involving both artificial neural networks and human learners, bridging machine learning and neuroscience.
summary: The paper shows that in the early phases of training neural networks learn the “optimal constant solution” (OCS), i.e., they mirror the statistics of the labels while being invariant to the input points. It considers deep linear networks with bias terms and derives the exact learning dynamics based on Saxe et al. (2014; 2019). It then proves that deep linear networks learn the OCS early in training. To support its findings, the paper performs experiments on both deep non-linear neural networks and human learners that display qualitative agreement with the theoretical analysis.
weaknesses: 1. The most significant weakness of this work is the lack of a definition and clear description of the studied “hierarchical task”. While the authors describe it as a "hierarchical category learning task" with hierarchical levels, there’s no clear explanation of the precise structure or production rules, such as the nature of the relationships between input categories and output labels. Without knowledge of the distribution of $X$, $Y$, and precise definitions of the hierarchical levels, I find it practically impossible to fully understand the study, particularly the empirical aspects. The link between the model of data and the OCS is also not discussed.

2. The *continuous correct-rejection scores*, which are crucially used to infer OCS learning, are not defined within the main text.

3. The figures and the corresponding captions are poor and challenging to interpret. Figure 1 contains 6 panels that lack sufficient standalone explanations. In Figure 2, the reader does not know what different colors represent.

4. Proposition 3 must assume some initialization statistics for the weights that are not specified.

5. In general, the paper presentation is dense, often lacking detailed explanations or explicit definitions. It also frequently refers the reader to the appendix, detracting from readability. 

In summary, the paper has potential, offering some interesting insights into early learning in neural networks and connections to human cognition. However, I think it requires significant rewriting and, in particular, improvements in clarity, structure, presentation, and detailed explanation and discussion of the empirical results.

*Typos:*

(L253) The word “singular” is repeated twice.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. Does the results in the paper indicate that bias towards OCS solution could eventually vanish after long enough training?
2. Fig. 3 and Fig. 5 seem to reflect different outcomes, where Fig. 3 shows a initial score drop for all hierarchical levels, only the top level score drop is prominent in Fig. 5, what would be the cause of this? Besides, do TNR and $f^{tn}_k$ refer to the same metric? If so, it's better to maintain consistent notations. 
3. Line 897, "Given responses in y or in and target vectors", a typo?
4. Due to the hierarchical label design used in the paper, it seems well related to the domain of multi-label learning, I would suggest to discuss some related work in this domain.
soundness: 3
strengths: The paper introduces an intriguing exploration of simplicity biases of optimal constant solution in training neural networks, with potential relevance to human societal patterns. This work could interest both the machine learning community and researchers in other fields, with possible extensions to topics like fairness and stereotyping.
summary: The paper theoretically and empirically examines early learning of the optimal constant solution (OCS) in neural network training. Under a hierarchical label setting, the authors analyze the learning dynamics of linear networks with bias terms, demonstrating that OCS learning happens in the early training stages. Empirical results and human studies confirm that this phenomenon extends beyond linear networks to more complex nonlinear networks and human learners.
weaknesses: My major concern about the paper is about its clarity: 
* Plots are not explained well, e.g., what does "output activation top" represent in Figure 3?
* Even after reading Appendix A.4, I'm still a bit confused about the definition of TNR rate, in equation 7, the level index $k$ appears on the left side of the equation but not on the right, should the $y$ term in right hand side be $y_k$ to denote the target label for a specific level? Besides, this definition seems important and should potentially be moved to the main paper. 
* The hierarchical MNIST dataset is only explained in the Appendix, and it's not quite clear how the hierarchical label structure is established for the dataset.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: Does the theoretical argument nicely extend to linear networks with more than two layers? In the multi-layer case, would a bias term present in an intermediate layer result in a bias towards OCS?

In what ways do the linear network solutions extend Saxe et al. (2014) precisely? Through the inclusion of bias terms, and a relaxed assumption (Proposition 1)?
soundness: 4
strengths: Though simplicity bias has been observed empirically and is thought to be important for generalization, its origin lacks a theoretical understanding. This paper is significant in providing a concrete theoretical explanation for a particular kind of simplicity bias (OCS learning)

Solves for parameter dynamics under a relaxed assumption (Proposition 1) compared to previous works (which typically assume uncorrelated inputs)

Demonstrates that linear networks can provide meaningful predictions of non-linear models, including CNNs

The relation to human learning is interesting and unique
summary: This work employs an analytically tractable analysis of learning dynamics via linear networks, focusing on the bias towards the optimal constant solution (OCS) early in training. A 2-layer (and 1-layer) linear network with no bias terms is first studied, whose learning dynamics can be solved exactly (under a commutativity assumption). This analysis is then extended to include a bias term in the first layer, in the case of uncorrelated inputs, and it is shown that the same approach works in this case (since the commutativity condition remains satisfied). Empirical evidence for early convergence to the OCS is then provided, which does not occur for linear networks without bias terms, and a theoretical explanation is provided, showing that the OCS mode dominates the input-output covariance matrix when biases are present. This phenomena is then related to human learning, showing that humans also display a bias towards the OCS early in learning. Finally, it is demonstrated that early bias towards the OCS is not solely driven by bias terms, and can instead be induced by properties of the data via the input-input correlation matrix.
weaknesses: It would be helpful to include a brief description of the hierarchical learning task in addition to Fig 1 for those unfamiliar with the setup

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 1
questions: - I do not understand the relationship between this work and a broad range of work [4, 5, 6] on spectral learning curves, where it is known that features are learned hierarchically in roughly the order of their power in the generative function. The original Saxe paper [1] demonstrates a similar result in a linear setting. How is the early-time learning of a constant solution related to this learning of eigenmodes in order of decreasing power? Is this a zeroth order term in some sort of a Harmonic expansion of the target function? I would like a clear and detailed mathematical exposition of this fact, either in the author response, or an update to the manuscript. I know, for instance, sometimes networks begin learning by attempting a kernel solution before transitioning to feature learning ("grokking", see [8]) -- viewed this way, are such settings examples of learning the "optimal linear solution" at early time?
- Are these results only true for hierarchical tasks? Again, the scope of where and when this is true is not clear. For example, the MNIST experiment (effect is weaker in Orth. MNIST) makes me think the results only hold in hierarchical settings (again, challenging "universality"). 
- I don't quite understand the claim around "generic input correlations" -- as I see it, you show that a range of settings where input data (independent of labels) has low-dimensional structure exhibit learning of OCS. But I thought the OCS was about mimicking the structure of labels independent of input, not the other way around? What does $X^TX$ have to do with learning the optimal distribution over labels ($X$ is the input matrix)?
soundness: 3
strengths: A major strength of the paper is that it is original, and makes claims about a broad range of phenomena. Generally, I think there isn't enough work studying how learning systems -- both artificial and natural -- exhibit qualitatively similar behaviors in various learning regimes. This paper studies linear networks (with and without bias), CNNs, and humans, and its contributions span from machine learning theory of linear networks to human psychological experiments, which is a breadth that is rare as well as nice to see. If it is true that the early time learning of the OCS is shared between a broad range of artificial networks and robustly replicated across humans, this would be an interesting and significant result. The inclusion of human data that qualitatively supports the hypothesis is a major strength.
summary: This paper proposes that the early-time learning of a constant solution that is input-independent and a pure function of the label distribution is a universal property of learning dynamics in neural networks and humans. It proves that linear networks with biases learn this early-time OCS and slightly more complicated and nonlinear networks exhibit qualitatively similar learning curves. Such learning dynamics are characterized by an initial dip in TNR for top levels of the output hierarchy in a hierarchical learning task, and such learning curves are also present when humans are trained and tested on such a task. Altogether, the paper seeks to provide theoretical and empirical evidence that early-time mimicry of the label distribution may be a general property of learning systems seeking to minimize error on a task.
weaknesses: The paper is quite confusing in terms of how it is laid out, and what the scope of which claims are. In my opinion, it tries do a bit too much. For instance, tensor trickery related to the NTK feels out of place/a low order concern when you didn't even take the time to define the hierarchical learning task clearly. I had to look through the Appendix as well as (Saxe, 2019) to get a sense of the task. Some specific weaknesses: 
- It's not clear the scope of the claims being made. Either you are 1) remarking that the learning curves between linear, nonlinear networks, and humans on a certain class of task look similar but the mechanistic reasons are not well understood, and this is interesting and warrants further study or 2) making a claim that ALL neural networks are marked by this early-time behavior, and they are driven by mechanisms you identify. I would be more sympathetic to the contributions of this paper if you were making claim (1), but since you use the words "universal" and "mechanistic" in your abstract, I take it you are making claim (2). If this is the main claim, then the evidence presented is inadequate as justification. This is for the reasons that follow.
- You spend much of the paper talking about linear networks with biases, but in Section 6 make the (extremely strong) claim that such reversion can *equivalently* arise from input correlations. If it is indeed true that very general covariate structure can give rise to this behavior, then why have you spent most of the paper reasoning about the contrived and simple setting of linear networks with bias? Phrases like "might hence be ubiquitous beyond these datasets" arouse suspicion. Also, how can I believe that early-time learning of OCS is "universal" when it vanishes in even linear networks if you ablate the biases?
- I put very little weight on experiments that work with MNIST and CIFAR. One can conjure up evidence for any hypothesis by studying a suitable version of such datasets with some contrived architecture. They are simple tasks that might have been useful 10 years ago but certainly do not convey any information in 2024, in my opinion. I think this paper is at its strongest when discussing linear networks where claims are made precise, or human studies where the empirical trends found in these (interesting and novel!) experiments can be unpacked. I think trying to convince a reader that this is a universal property by showing toy experiments "in the middle" of linear/humans (eg. CNNs) is not a good strategy, especially since by the end I am not even convinced the mechanism for this phenomenon is the same across even just linear networks and humans. Spending some time providing evidence for this latter claim and dropping any attempts at universality claims justified only by MNIST/CIFAR, would in my opinion be a good idea. 
- You remark that "characteristic learning signatures" arising from early time learning of the OCS involve a sort of step-wise learning structure. While I'm aware a variety of toy synthetic settings studied by theorists exhibit this behavior (see [7], for instance), no non-trivial data (ImageNet, C4) gives rise to dynamics that look like this (learning curves smoothly go to low loss rather than being step-wise). So certainly the word "universal" does not describe these types of learning curves that are "characteristic" of early-time learning of OCS. 

All in all, I find the presentation and scoping confusing and the claims (where I can understand them) unjustified. I laud the broadness and ambition of the study, but find it ultimately somewhat unconvincing in its present form. Originally, I gave this a rating of 3, but on reflection, I think the inclusion of human data that qualitatively exhibits early-time OCS is on its own interesting enough to bump that up to a 5. 

--------- 
[1] Saxe, Andrew M., James L. McClelland, and Surya Ganguli. "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks." arXiv preprint arXiv:1312.6120 (2013).

[2] Belrose, Nora, et al. "Neural Networks Learn Statistics of Increasing Complexity." arXiv preprint arXiv:2402.04362 (2024).

[3] Kang, Katie, et al. "Deep neural networks tend to extrapolate predictably." arXiv preprint arXiv:2310.00873 (2023).

[4] Bordelon, Blake, Abdulkadir Canatar, and Cengiz Pehlevan. "Spectrum dependent learning curves in kernel regression and wide neural networks." International Conference on Machine Learning. PMLR, 2020.

[5] Simon, James B., et al. "The eigenlearning framework: A conservation law perspective on kernel regression and wide neural networks." arXiv preprint arXiv:2110.03922 (2021).

[6] Bordelon, Blake, Alexander Atanasov, and Cengiz Pehlevan. "How Feature Learning Can Improve Neural Scaling Laws." arXiv preprint arXiv:2409.17858 (2024).

[7] Abbe, Emmanuel, et al. "The staircase property: How hierarchical structure can guide deep learning." Advances in Neural Information Processing Systems 34 (2021): 26989-27002.

[8] Kumar, Tanishq, et al. "Grokking as the transition from lazy to rich training dynamics." arXiv preprint arXiv:2310.06110 (2023).

Meta-review: This paper studies early learning of the optimal constant solution (OCS) in neural network training. Under a hierarchical label setting, the paper analyzes the learning dynamics of one- and two-layer linear networks with biases, demonstrating that OCS learning happens in the early training stages. Empirical results and human studies confirm that this phenomenon extends beyond linear networks to more complex nonlinear networks and human learners.

This paper studies a very important question about the simplicity biases of neural network training and identifies an interesting phenomenon. However, it's missing citations and discussion of some highly relevant work on the spectral bias of NNs [1-3]. In particular, [1] shows the early learning of the optimal linear function (1st-order); the optimal constant function studied in the current paper is the 0th-order version of that result. Although these results are not the same, the high-level arguments and intuitions are similar. The current paper's main argument is essentially that the constant all-one vector has a significant spectrum in the input covariance matrix. The fact that this implies early learning of OCS follows directly from arguments made in previous literature and is not novel. The authors are encouraged to discuss these connections in the next version and to make it clear what this paper's novel contributions are in the context of existing literature.

[1] Hu, W, Xiao, L, Adlam, B, Pennington, J. The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks. NeurIPS 2020.

[2] Cao, Y, Fang, Z, Wu, Y, Zhou, D, Gu, Q. Towards Understanding the Spectral Bias of Deep Learning. IJCAI 2021.

[3] https://en.wikipedia.org/wiki/Frequency_principle/spectral_bias

Decision: Reject