Do Mice Grok? Glimpses of Hidden Progress in Sensory Cortex
Authors: Tanishq Kumar
Blake Bordelon
Cengiz Pehlevan
Venkatesh N Murthy
Samuel J. Gershman
Venue: ICLR 2025 Poster (2025)
Keywords: neuroscience; representation learning; grokking; overtraining; cortex
Primary area: applications to neuroscience & cognitive science
TL;DR: We find evidence for rich feature learning in mouse piriform cortex during overtraining and propose it is driven by approximate margin-maximization, a known cause of grokking in deep learning.

Abstract: Does learning of task-relevant representations stop when behavior stops changing? Motivated by recent work in machine learning and the intuitive observation that human experts continue to learn after mastery, we hypothesize that task-specific representation learning in cortex can continue, even when behavior saturates. In a novel reanalysis of recently published neural data, we find evidence for such learning in posterior piriform cortex of mice following continued training on a task, long after behavior saturates at near-ceiling performance ("overtraining"). We demonstrate that class representations in cortex continue to separate during overtraining, so that examples that were incorrectly classified at the beginning of overtraining can abruptly be correctly classified later on, despite no changes in behavior during that time. We hypothesize this hidden learning takes the form of approximate margin maximization; we validate this and other predictions in the neural data, as well as build and interpret a simple synthetic model that recapitulates these phenomena. We conclude by demonstrating how this model of late-time feature learning implies an explanation for the empirical puzzle of overtraining reversal in animal learning, where task-specific representations are more robust to particular task changes because the learned features can be reused.

Published as a conference paper at ICLR 2025
Do MICE GROK? GLIMPSES OF HIDDEN PROGRESS IN SENSORY CORTEX
Tanishq Kumar" Blake Bordelon Cengiz Pehlevan Venkatesh N. Murthy Samuel J. Gershman
Harvard University
ABSTRACT
Does learning of task-relevant representations stop when behavior stops changing? Motivated by recent work in machine learning and the intuitive observation that human experts continue learn after mastery, we hypothesize that task-specific representation learning in cortex can continue, even when behavior saturates. In a novel reanalysis of recently published neural data, we find evidence for such learning in posterior piriform cortex of mice following continued training on a task, long after behavior saturates at near-ceiling performance ("overtraining"). We demonstrate that class representations in cortex continue to separate during overtraining, so that examples that were incorrectly classified at the beginning overtraining can abruptly be correctly classified later on, despite no changes in behavior during that time. We hypothesize this hidden learning takes the form of approximate margin maximization; the neural data, as well as build and interpret a simple synthetic model that re- capitulates these phenomena. We conclude by demonstrating how this model of late-time feature learning implies an explanation for the empirical puzzle of over- training reversal in animal learning, where task-specific representations are more robust to particular task changes because the learned features can be reused.
validate this and other predictions in
INTRODUCTION
Understanding how representations evolve during task learning is fundamental question in both neuroscience and machine learning. While the dynamics of learning are standard objects of study in both fields, the representational dynamies at play long after ceiling performance is reached on the training task are much less commonly studied.
In deep learning, this has changed with the discovery of the grokking phenomenon (Power al. 2022]. whereby neural networks first fit their training data to high accuracy, then many epochs of training later ("overtraining"). abruptly generalize. As phenomenon that epitomizes how neural networks can behave in sharp and unexpected ways, it has been under intense study (Nanda et al.) 2023 Liu et al. 2022b; Varma et al. 2023). Emerging theories posit that generalization occurs due to delayed feature learning taking place when the network has already achieved ceiling accuracy on the training task (Kumar et al. 2023 Lyu et al. 2023).
In psychology, there exists long line of work on learning tasks at all levels of expertise, includ- ing after task mastery (Ericsson al. 1993; Newell Rosenbloom 1988] Fitts & Posner [967] Mackintosh] 1969 Richman et al. 1972 Mead 1973). Many of these works attempt to address the intuitive observation that human experts in a variety of domains continue to learn on tasks at which they have achieved near-ceiling performance. There has also been much work using deep learn- ing to model perceptual learning tasks, to study the degree to which biological and artificial neural networks exhibit similar behavior this class of tasks (Wenliang & Seitz 2018 Yashar & Denison 2017).
Bakhtiari] 2019
The nature of representational changes play during overtraining learned tasks have been tack- led at fine grained (theorems proven about certain classes of deep networks) and coarse grained (qualitative explanations in psychology) levels. Such understanding, however, is broadly missing
*Correspondence tkuma r@col Tege. ha edu.u.

---

Published as a conference paper at ICLR 2025
at intermediate level: that of experimental neuroscience. This is because there several diffi- culties in naively testing the hypothesis that representations continue to evolve during overtraining in cortex. First, for many animals, recording neurons is invasive and risks damaging the animals, so recordings are typically only taken at the end of training to study the end-time learned representation Yuste 2013 Kim al., (016). Second, experiments in systems neuroscience do not typically de- sign stimuli as "training" and "test" in the way datasets are constructed in deep learning, so making claims about learning and generalization is not possible. We elaborate on further difficulties, as well as the connection of our phenomenon of interest to the related phenomenon of representational drift, in Appendix [A.3
Here, we revisit the neural data from a recent work with precisely the desirable qualities above: (Berners-Lee et al., 2023), who study neural activity during learned odor discrimination task in mouse piriform cortex. The mice are trained to discriminate one target odor from hundreds of nontarget odors, then continue to be trained for weeks on the same task after reaching behavioral mastery ("overtraining"), with neural firing rate data recorded during this time. After overtraining, they are given held-out test examples. Our starting point this work are two empirical observations in (Berners et al. 2023) that we seek to model and explain.
Decoding accuracy of training labels from population activity increases throughout over- training.
Mice overtrained for longer tend to do better on held-out test examples after the overtraining period is complete.
Our goal in this work is to understand the representational dynamics in cortex at play underlying these observations, and to what extent, if any, they are similar to those at play in deep neural net- works where learning continues after training accuracy reaches ceiling (Power et al. 2022). Our contributions are the following:
We find that target-nontarget odor class representations mouse piriform cortex continue actively separating during overtraining while behavior remains unchanged, loosely resem- bling the "hidden progress" (Barak et al., 2022 Nanda et al. 2023) that often taking place in synthetic deep learning settings.
We find that the margin of the maximum margin classifier at each day of overtraining is increasing in mouse cortex, implying that points (odor trials) sufficiently close to the optimal decision boundary may be classified incorrectly the beginning of overtraining, but correctly at the end, despite no outward change in behavior.
We construct a synthetic model of piriform cortex performing this task exhibits these prop- erties, well the grokking phenomenology. We interpret the resulting dynamics and use targeted ablations to trace the continued learning to margin maximization.
We use our insight to suggest a new, fine-grained and neural explanation for the overtrain- ing reversal effect, an empirical puzzle from experimental psychology in which animals overtrained on a task learn its reversal more quickly than those not.
We also note that [erners-Lee et al. 2023 are not the first to observe such effects experimen- odor representations was also observed by
tally: similar effects in sharpening of already-leande Shakhawat et al. (2014) Kadohisa & Wilson (006)), so that the phenomena seek to model in this work are likely to be robust at least in the setting of mouse olfaction.
2 RELATED WORK
Lazy/rich learning and margin maximization. We cast our ideas in the language of "lazy" and "rich" regimes of representation learning, introduced machine learning (Chizat et al. 2019 Jacot ct al. 2018). Such characterization of learning regimes has become increasingly popular com- putational neuroscience (Chizat et al.) 2019; Farrell et al., 2023 Flesch et al. 2021 2022 Ito & Murray 2023). In machine learning, lazy learning refers to trained neural network being well approximated by a kernel method in its initial neural tangent kemel (NTK) and the rich regime is when this approximation breaks down the network learns task-relevant features and its weights
2

---

Published as a conference paper at ICLR 2025
Oventraining
Decoding Accuracy vs Days of Training (Sliding Window]
2
0.85
0.75
larett cortargtt
0.70
Decoding
0.65
Mouse
Mouse
9.60
Mouse
Day Ovartraning
Day 26
0.50
100
12
14
1.6
10
20
24
26
Days
Figure 1: (Left) Behavior of mice on a binary discrimination task of odors, where mice indicate their selected choice by licking left or right. Y-axis is fraction of licks left vs right on given day. Day the first day of overtraining. Correct lick for non-target is left, and right for target. The mice can discriminate near-pos serr-peffeclly as overtraining begins. Reproduced with modification from (Berners-Lee et al., 2023). (right) Increase in decoding accuracy from 10-fold linear discriminant analysis for each session. Shaded colored lines in background show standard errors for each mouse.
move far from initialization. classification problems, margin maximization can be one type of rich learning (Mohamadi et al. 2024 Matyasko & Chau 2017). where the margin of classifier is defined as the distance from the decision boundary to the nearest data point. There are some sub- tleties around defining margin maximization in setting where decoder is being retrained evolving feature map: we state a precise working definition in Section 3.1.2
Grokking. Much theoretical work has been done understand the grokking phenomenon, discov- ered by PPwerr et al. (2022), ,, in which task-specific representation learning occurs long after training accuracy has saturated (Nanda et al. 2023 Barak al.) 2022 Liu al. 2022a). Current theories explaining grokking are varied: Liu et al. (2022b) claim it is due to weight norm at initialization, Nanda al. (2023) suggest it driven by weight decay, Varma al. (2023) suggest it is due to com- petition between memorizing and generalizing neural circuits. A flurry of recent theoretical papers unify these views this late-time feature learning. where networks begin lazy and then abruptly transition into the rich regime much later during training (Kumar et al. 2023 Lyu et al. 2023). Re- cent empirical evidence has accumulated in support of this perspective (Edelman al. 2024| Clauw ett al. 2024 Mohamadi al. 2024). where rich learning often takes the form of margin maximiza- tion in classification tasks. For instance, Morwani et al. (2023) and MMohamadi et al. (2024) show that on certain classes of tasks, margin maximization during overtraining provably causes grokking.
REVISITING NEURAL DATA FROM OVERTRAINING
3.1 SETUP
Berners-Lee al. (2023) collected tetrode recordings from posterior piriform cortex in adult mice during overtraining on a binary odor discrimination task (Figure The posterior piriform cortex (PPC) is a key brain region involved in olfactory processing (Blazing & Franks 2020 Srinivasan & Stevens, 2017). An odor is defined as an n-hot vector of length k. with = 3. 13. In other words, a unique combination of three chemicals out of thirteen possible chemicals. The "target" odor is uniquely defined as consisting of three specific chemicals, where non-targets never contain any of these three. The discrimination task mice are trained on is to distinguish the target from the nontarget odors. After initial training period of 8 days, the mice master the task, reaching ceiling behavioral performance (see Figure TT), yet training continues in the same way for further 18 days ("overtraining"). This period is when firing rate data is recorded. After this overtraining period is complete, the mice are exposed to held-out odors ("test/probe" examples), which share one odorant with the target odor and therefore more difficult correctly classify.
3

---

Published as a conference paper at ICLR 2025
Initial Represennaaional Similarity (Average, First 3 Days of Overtraining)
Mouse
Mouse R
Mouse S
Mouse
0.05
-0.04
0.11
-0.12
T
0.29
-0.18
0.31
-0.28
0.8
-0.04
0.09
NT
-0.12
0.13
NT
-0.18
0.23
NT
-0.28
0.27
0.6
Correlation
Ni
Ni
Final Representational Similarity (Average, Last 3 Days of Overtraining)
0.4
0.2
NT
0.51
NT
NT -0.11
0.50
NT
-0.69
0.67
Figure 2: Representational similarity matrices plotting average pairwise correlation between target and nontarget population responses from piriform cortex on the first (top) and last (bottom) days of overtraining. We drop the last few days of data for Mouse due to inconsistencies data; method- ological details A. We compare the separations (anticorrelation) above that of random/ablated baselines in Appendix A.5 finding them significantly larger.
3.1.1 REPRESENTATIONS OF TARGET/NONTARGER ODORS SEPARATE DURING OVERTRAINING
The key object of study throughout the neural data reanalysis will be the population firing rate ma- trix IR x which contains the neural response vector (activations) for each trial (presented stimulus) on given-mouse day. There are around N ss 200 trials per day each recording around D 15 neurons, and around 15-20 days for each of 4 mice overall. We compute the average dot product (over trials) between the (z-scored) firing rates between target and nontarget trials, finding that representations of target and non-target odors continue separate over the course of overtrain- ing. We plot the representational similarity matrices at the beginning and end of overtraining for each mouse (Figure 2}. with further methodological details deferred to Appendix A.I
The measurement of such representational similarity metries is standard in neuroscience (Kriegesko- ite et al.) 2008), and similar to common interpretability techniques deep learning that involve probing for directions in activation space (Bau al. 2019: Olah al., 2020 Elhage et al.) 2021 Zhang & Nanda 2023) and comparing them, for instance by computing their dot product. The con- tinued separation of class representations between the target and nontarget odors in F evidence that, despite no visible changes in behavior, there is continued representation learning in mouse PPC during overtraining this task. Such continued learning may aid in generalization on unseen odors, as indeed Berners-Lee et al. (2023) find empirically to be the case.
What do these task-relevant changes look like visually? In Figure 3 we plot the projection of odor representations given session onto the first 2 principal components of the firing rate matrix for that session, which trials by neurons for each mouse. We see generally that the bottom row has more separated representations than the top row, providing qualitative view of how target odors are more distinguishable PPC representation space after overtraining than before. These results provide glimpse of the representational dynamics at play decoder accuracy is increasing, broadly consistent with the increasing separation we see quantitatively in Figure 2 Note that this continued representation learning ('rich feature learning") despite changes in training accuracy ("behavior") precisely what characterizes the abrupt generalization found in the grokking phenomenology in
deep learning (Lyu & Li 2019 Kumar al. 2023 Mohamadi et al. 2024
4

---

Published as a conference paper at ICLR 2025
First Day of Overtraining
Mouse
Mouse
Mouse
Mouse T
lagge
Honvaaget
Last Day of Overtraining
Figure Neural representations (projected onto the top principal components) of target and nontarget odors on the first (top) and last (bottom) day of overtraining for each mouse, showing qualitative separation during overtraining measured quantitatively in the representational similarity matrices in Figure Cluster centers plotted for ease of visual comparison.
3.1.2 REPRESENTATIONAL CHANGES LEAD TO AN IMPROVED MAXIMUM MARGIN CLASSIFIER
We now take an alternative perspective on the class separation of odor representations and instead examine the robustness of the learned classifier, as measured by its margin, classical notion of confidence and robustness from statistical learning theory (Bishop 2006).
We begin by clarifying a few distinct notions of margin maximization that apply when a decoder is trained on evolving hidden representation.
1. A linear decoder is trained on a fixed feature representation ø for a fixed number of steps, which may or may not involve convergence of the decoder.
2. A linear decoder is trained
fixed feature representation & convergence.
3. The feature representation d; changes over time, and at each time step linear decoder is trained on the feature representation to convergence.
While the margin the classifier defined by the decoder can increase the decoder is trained for more steps on fixed feature representation as in (I), if the margin of the max-margin (converged) classifier increasing (3), it implies the features are evolving to allow for a max-margin classifier with larger margin. This latter notion of mangin-maximization that involves feature learning is the notion of margin maximization we will use.
We can define this mathematically follows. Let (x(x) N be the representation at time t for data v, and let (t) be the parameters of the max margin SVM solution to the label classification problem with o, (xx) as input point and y: label. Then the margin MMt) takes the form:
1
M(t)
where w(t) min w such that yow o(zx); 2
[w](t)
In these terms, our hypothesis is that the evolution on odor representations takes the form of (max) margin maxinization.
While the "margin" of a classifier usually refers to the distance to the single closest point, we take it refer the distance the closest 1% of points, neural activity underlying individual points
5

---

Published as a conference paper at ICLR 2025
Normalized Margins Across Days
00.020
0.0032
percentile (Support Vectors/Hardest Examples)
5th percentile (Hard Examples)
0.0030
0.018
1st Percentile
0.0028
5th Percentile
0.0026
0.016
0.0024
0.0022
0.0020
0.012
16
Day
Figure 4: The average distance from decision boundary within the smallest 1 1% and 5% distances extracted from linear support vector machine trained on population data from PPC. We normalize margins for each mice so data are comparable across mice, shaded bands show standard errors.
(trials) is highly variable. We track the closest points the learned decision boundary in linear SVM trained on the same population decoding data to convergence with a fixed number steps throughout. We measure the average distance from this classifier to the closest 1% (hardest amples) as well as 5% (hard examples) of points, finding in Figure that these quantities increase substantially during overtraining.
Therefore, PPC representations evolve to increase the margin of the maximum margin classifier trained those representations, similar how many works find that deep neural networks which "grok" during overtraining on classification tasks are implicitly driven to do so by maximizing mar- gin (Morwani al. 2023 Mohamadi et al., 2024 Lyu & Li 2019). This margin-maximization perspective is provocative not only because it provides a geometni and intuitive picture of contin- ued learning in this setting, but because it is suggestive of why certain generalizing solutions are learned. For instance, MMowwan et al. (2023); Mohamadi et al. (2024); Lyu & Li (2019) prove in various settings that the reason that neural networks often solve particular tasks (eg. modular arith- metic) using very specific types of features (eg. Fourier features in Nanda et al. (2023)), is because the regularized optimization trajectories are implicitly biased towards margin- seggin-maximizing solutions.
We emphasize that since test data (unseen test odors) is missing during the training period, we cannot make claims about whether the cortical representation begins lazy or rich (ie. evolve significantly during the initial 7-day learning period compared to initialization), and this is important avenue for future work and limitation of our reanalysis. While the optimization algorithms at play in brains form of "implicit-bias towards gener-
poorly understood, it is speculatively possible that alizing solutions may also be the reason that overtrained mice do not just memorize their training odors, but learn representations that are helpful difficult, held-out examples.
4 THEORY & MODELING
4.1 SYNTHETIC MODEL OF MOUSE PIRIFORM CORTEX
Setup. We construct synthetic version the odor discrimination task that captures key compu- tational considerations at abstract level. We consider more biologically plausible counterpart in Appendix A.7 finding the same phenomenology persists in the same way. We take our earlier definition of odors n-hot vectors of odorants (chemicals) in {0, 13*h that pass through dom projection, inspired by the functional role of the glomeruli (Blazing & Franks 2020), with 10, k 100. We train one-hidden layer multilayer perceptron (MIP) these embeddings

---

Published as a conference paper at ICLR 2025
to classify the one target odor from 200 non-target odors. We ReLU nonlinearity simulate non-negative firing rates. We also hold out 20 test examples that share an odorant (a single entry in the bit string) with the target. These examples are thus closer in both Hamming and projected space to the target, and therefore harder to discriminate. We train the MI.P with vanilla gradient descent on cross-entropy loss as is standard in classification.
Training and
Margin
Pedfect LEe AEL.SSSSSSS
Margin
Tauteee Twarraaaaaa In ArEura
0.4
Margh
Losk
0.3
2-
0.1
0.0
Test Loss
30
10
10
10
101
10"
10
10'
10'
10
Epoch
Tpod 1900
Figure 5: Recapitulating and interpreting mouse piriform cortex dynamics in a simple model. Top left: training and test loss over the course of training. Top right: margin over the course training. Bottom: projections of the top 2 principal components for several training epochs. Notice how the target and probes (test trials) separate during overtraining (Epochs 1000-9000) despite the target never being trained on probe example.
Mouse behavior and neural dynamics are captured by an overtrained MLP. This simple model captures the key computational phenomena at play in piriform data that was the subject of analysis in Section 3 In Figure 5
see that the test loss (on the probe trials) continues to decrease after training loss plateaus at a low value, in line with continued increase in the classifier margin. The readout layer of our MLP acts the "decoder" on the hidden layer representation, which evolving to allow for an increased max-margin solution to be found. We also see a separation between the target and the nontarget representations emerge during training (up to epoch 1000). However, at epoch 1000, when training loss has converged, the target and probes are not yet easily separable. Strikingly, overtraining on the target-nontarget separation task improves performance on the probe odors, which are network maximizes margin between the target and nontarget representations during the overtraining period after epoch 1000. This manifests as the red dot (target) gradually separating from the green cluster (test probes) during overtraining. There an abrupt grokking-like transition in test accuracy (the fraction of probe trials correctly classified) during the overtraining period, Epochs 1000-9000, exactly as we see in F
from the training set. This happens because the
We can test whether margin maximization is the causal factor in this simple model by ablating it and asking whether late-time test loss improvements vanish concomitantly. We do this by replacing the Cross-Ente Entropy objective, which is known to drive late-time margin maximization (Lyu & Li 2019; Lyu et al., 2023), with a Hinge loss which has a hard-margin objective instead. This means after the model achieves margin C each training point, loss that point vanishes, the late-time
7

---

Published as a conference paper at ICLR 2025
Fisher Discriminant During Training
Probe Accuracies
Duerlap
Test Accuracy (Overlap 11
Overisp
Accuracy (Overiap 21
Fisher Unnar Discriminnt
Dverlap
Overlap
0.8
Test Accuracy (Overlap
Accuracy (Overiap 4]
Perfect
Perfect trai Accuracy
2.5
0.6
1.0
0.4
0.5-
10'
10'
10'
10" 10' 10
10'
10"
10"
Epoch
Epoch
Figure 6: Probe trials are learned in order of increasing overlap with target. (Left) Tracking the Fisher Diseriminant (FD) between target and probe classes throughout training. This measures sig- nal noise in representation space. (Right) The accuracy ("behavioral performance") on increas- ingly difficult trials over time, exhibiting grokking.
loss incentive for margin maximization is removed. We find in this ablated setting margin saturates approximately when training loss does, and that test-loss saturates at this point in training as well, so that the grokking-like effect is ablated. Mathematical details and plots are in Appendix A.6 supporting the notion of a causal link.
Feature learning in the synthetic model. We increased the dimension of the task "odor" vector as well as the number of nonzero entries so that we can introduce many types of "probe" trials that share increasing amounts of odors (overlap) with the target, allowing our model is never trained on the probes, but monitoring probe accuracy allows us to measure how margin maximization on the original nontarget odors (which are guaranteed to have zero overlap with the target) can be helpful in solving more difficult trials.
vary difficulty. Note that
In Figure 6 track the Fisher Discriminant (FD) between target and probe classes, well as the accuracy on discriminating the two throughout training. The FD is statistical quantity that uses the linear combination of the features that maximize the separation between two classes, and represents the direction in the feature space along which the projected class means are maximally separated while minimizing the variance within each class. This is the quantity that allows the linear discriminant analysis decoding accuracy to increase; the portion of the lines after low train loss is reached mimics how the median LDA posterior probability on the correct class increases in Figure [ (right). Mathematical details about the Fisher Discriminant and its relation margin, LDA, and more are deferred to Appendix A.4
Possible biological drivers of late time learning. Certain types of loss functions have been proven in a deep lcarning setting to drive late-time margin maximization (Soudry et al.] 2017 Lyu Li 2019). Here, we speculate biologically plausible components of the objective cortex that may plausibly serve a similar role. A simple but plausible driver late-time learning is the set of en- ergetic/metabolic constraints on biological organisms. Such constraints been suggested give rise to a variety of neural phenomena (Sterling & Laughlin 2015), for example disentangled represen- tations (Pehlevan al. 2017: Olshausen & Field 1996: Plumbley 2003 Whittington et al., 2023). Pertinently, such constraints, in the form of weight decay in machine learning, are often the driving force for grokking in some settings (Nanda al. 2023) Liu et al. (022b). These constraints force neural networks to deviate from using a lazy (kernel) method, and force the weights to move in task-relevant directions (Lyu & Li] 2019 Kumar et al. 2023). Another possibility in [Berners-Lee al. (2023) is very small supervised error signal from occasional mistakes made by the mice for idiosyncratic reasons that could plausibly be driving late-time learning (See Figure [I).
Finally, key plausible driver of late-time learning in cortex is the existence of unsupervised terms in the implicit objective of organisms that penalize low confidence predictions. For instance, cross- entropy loss is nonzero even when example is correctly classified because it can be seen as penalizing the uncertainty of model predictions, rather than merely correctness. The fact that the loss is nonvanishing even when predictions correct serves signal to guide task-relevant
8

---

Published as a conference paper at ICLR 2025
learning during overtraining. If biological learning systems store uncertainty in predictions and/or use them for behavior, as indeed is known to be the case in rodent cortex (Kepecs et al., 2008) and brains more generally (Lak et al. 2017 Fetsch et al. 2014 Komura et al. 2013), then such implicit penalties on uncertainty will serve similar functional role to an unsupervised term that keeps loss nontrivial even as training accuracy saturates. This nonvanishing loss during behavioral plateau can plausibly drive continued feature learning during overtraining.
4.2 RICH LEARNING DURING OVERTRAINING CAN EXPLAIN OVERTRAINING REVERSAL
Does our model of the dynamics underlying learning during overtraining make any new predictions outside our immediate setup? One concrete prediction of late-time feature learning is faster adap- tation under task reversal for any learned task. In machine learning this means fast adaptation of a predictor to distribution shift of the form y(z) + -y(x). If a network learns features to com- pute y(z), it is possible many such features can be reused to compute -y(x). This is not true the network learns v(æ), for instance, in the lazy regime, in which case y was fit using the initial features, which were not task desk-deenn,,nt, so cannot be advantageously "repurposed" for a reversed variant of the task. This particular prediction about reversal learning may seem as oddly specific to machine learning audience, but in fact overtraining reversal has been a topic of study by experimen- tal pyschologists for decades (Mackintosh) 1969; Richman al., 1972. Mead 1973). Theoretical work in psychology offers qualitative cognitive explanations in terms of attention allocation (Love- joy 1966). where our model of overtraining naturally posits a stronger, more fine-grained, neural model for overtraining reversal that is the first of its kind to our knowledge. We begin by defining the phenomenon.
The Overtraining Reversal Effect. The reversal effect is the empirical finding that animals that are overtrained on a task more quickly adapt when the task is reversed. Concretely, when animals are given a perceptual discrimination problem and the rewarded mapping is reversed, those animals that were overtrained on the original mapping adapt to the task reversal in fewer trials than those that were not overtrained, see (Mackintosh) 1969) for further discussion!" ! We propose a simple explanation for this phenomenon in terms of feature learning in the modern machine learning sense.
A Mathematical Model. We illustrate this with the simplest possible network model that exhibits this effect, a two layer linear network (Saxe et al. 2013). Reversal learning proceeds in two steps. First, we train two layer linear network with N hidden neurons f(x) Noo w² -(xx,, where h (z)) = W/ x is optimized for T steps on task y(r). After this, a new readout vector v is introduced to give new output frev (z) = 1 v h(x) which is trained the reversed task -z(x). The matrix Noo
w can be updated in the second phase of learning as well. At large width N, we can invoke mean field theory ideas (Bordelon & Pehlevan 2022 see Appendix F.1.1), allowing to describe the hidden kernel K(æ,, x') N h1(2)) ) h(x') dynamics in both phases of training. For whitened data and squared error loss, it suffices to compute the projection of the kernel K, K2 y T Ky, and the projection of the predictions fy = yy ff along the direction of the labels y. which gives:
Ky(t) = / (t) (tf = 2
(t)² (v - (t)).
(2)
We show that the kernel evolves only in this yy? direction in Figure 7 (a), where points separate
by their target category. These dynamics are run from time t = 0 to time T, which gives a terminal value of the kernel alignment K,(T), In the second phase of learning (t T) where the new output frev is fit, the predictions on the reversal task in the y direction evolve as
Se (1) V (T) + 1)2 + 4n. (-y - frev
(3)
We note that larger kernel-target overlap K2 leads to faster training in this second phase t> T, as rate of learning on the second task is a direct function of alignment on the first task. We verify this with simulations in Figure 7 (b). Since (T) is monotonically increasing with T, longer training on the first task will accelerate learning of the reversed task, providing an explanation for overtraining reversal. The key intuition is that adaptation time depends on alignment K, = K_y
where y Ky (-y ((-k)) )
overtraining leads to faster adaptation.
We note that there has been debate on whether the phenomenon is general across species and tasks BBeck 1966 Warren 1978)
9

---

Published as a conference paper at ICLR 2025
100
1.0
Reversal Task (No Pretraining)
Reversal Task (25 steps)
y- -1
Reversal steps)
Loss
0.6
PC2
Time
0.4
0.2
-2
20
40
60
100
PC 1
Time
(a) Pretraining Feature Movement (t < 7
(b) More Pretraining = Faster Reversal (t 7)
Figure 7: A simple two layer model explains faster reversal learning after overtraining the original task. (a) We plot the original hidden representations (projected from N hidden dimensions to the top two principal components) of width N 250 network for 4 odor mixtures, with two odors positive and two odors negatively labeled. (b) When training a new readout on the reversal task (x() after pretraining for steps the original labels y(æ), find that additional pretraining steps T accelerate learning the reversal task, consistent with theory.
5 CONCLUSION
We find that recent work in mouse olfaction provides evidence for rich sensory learning taking place under veil of a behavioral plateau mice. This suggests that difficult tasks that require significant changes in learned representations may benefit from overtraining, performance remains outwardly unchanging. also offers a glimpse shared structure between artificial and biological networks, where phenomenologically similar dynamics appear in both cortex and MIPs.
if behavioral
Our work has number of limitations. A key limitation of our work is that the neural evidence draw upon is observational, and new experiments in systems neuroscience are required to validate this hypothesis. We describe in detail proposed experimental setup in Appendix A.2 Another is that the number of neurons recorded per mouse-day (around 20) is small, so that trends can be noisy (see Appendix A.S for an example). A third limitation that we consider only olfactory cortex, and only one type of perceptual discrimination task whether such late-time learning is universal property across animals and tasks remains to be seen. We hope our hypotheses drive further work on learning dynamics in cortex during overtraining in animals. Our study raises several new questions: What are the classes tasks in which such overtraining is beneficial, and what do they have in common? Can deep learning models of cortex hint at the answer? What is the nature of the implicit objective sensory cortex driving this late-time representation learning? Altogether, our work aims toward a more complete characterization of the behavior of sensory cortex perceptual discrimination tasks, and to unify the mechanics underlying such behavior with existing theory and observations in deep learning.
ETHICS STATEMENT
Our work models the dynamics of learning during overtraining in mice and MIPs, aiming towards a more complete characterization of learning in both artificial and biological neural networks. We do note any important ethical consequences particular.
REPRODUCIBILITY STATEMENT
We model the data of [Berners-Lee et 2023). which is publicly available for open reproduction, and will open source our code to train and interpret artificial networks.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Shahab Bakhtiari. Can deep learning model perceptual learning? The Journal of Neuroscience, 39 (2):194, 2019.
Boaz Barak, Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hid- den progress in deep learning: Sgd learns parities the computational limit. Advances in Neural Information Processing Systems, 35:21750-21764, 2022.
Carol A Barnes, Matthew Suster, Jiemin Shen, and Bruce L McNaughton. Multistability of cog- nitive maps the hippocampus of old rats. Nature, 388(6669):272-275, 1997.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Gan dissection: Vi- sualizing and understanding generative adversarial networks. Proceedings of the International Conference on Learning Representations (ICLR), 2019.
C. H. Beck, J. M. Warren, and R. Sterner. Overtraining and reversal learning by cats and rhesus monkeys. Journal of Comparative and Physiological Psychology, 62(2):332-335, 1966. doi: 10.1037/hh0033744.
Alice Berners-Lee, Elizabeth Shtrahman, Julien Grimaud, and Venkatesh N Murthy, Experience- dependent evolution of odor mixture representations in piriform cortex. PLoS Biology, 21))): e3002086, 2023.
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, New York, 2006. ISBN 978-0-387-31073-2.
Robin M Blazing and Kevin M Franks. Odor coding in piriform cortex: mechanistic insights into distributed coding. Current opinion neurobiology. 14:96-102, 2020.
Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evolution in wide neural networks. Advances in Neural Information Processing Systems, 35:32240-32256, 2022.
Lars Buesing, Johannes Bill, Bernhard Nessler, and Wolfgang Maass. Neural dynamics sampling: model for stochastic computation in recurrent networks of spiking neurons. PLoS computational biology, 7111)e1002211, 2011.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019.
Kenzo Clauw, Daniele Marinazzo, and Sebastiano Stramaglia. Information-theoortic progress mea- sures reveal grokking an emergent phase transition. ICML 2024 Workshop on Mechanistic Interpretabiliry, 2024.
Benjamin L Edelman, Ezra Edelman, Surbhi Goel, Eran Malach, and Nikolaos Tsilivis. The evolution statistical induction heads: In-context learning markov chains. arXiv preprint arXiv:2402.11004. 2024.
Neil Blhage, A Nanda, Chris Olah, Shan Carter, Nick Cammarata Gabriel Schubert Wang, and Curtis Hermandez. A mathematical framework for transformer circuits. In Transformer Circuits Thread, 2021. sirtanasetoeeeeerrrtsttssssss
Anders Ericsson, Ralf Th Krampe, and Clemens Tesch-Römer. The role of deliberate practice in the acquisition of expert performance. Psychological review, 100(3):363 406, 1993.
Matthew Farrell, Stefano Recanatesi, and Eric Shea-Brown. From lazy rich exclusive task representations in neural networks and neural codes. Current opinion in neurobiology, 83:102780 2023.
Christopher R Fetsch, Roozbeh Kiani, William Newsome, and Michael N Shadlen. Effects of cortical microstimulation confidence in perceptual decision. Neuron, 83(4):797-804, 2014.
Paul M Fitts and Michael Posner. Human Performance. Brooks/Cole Publishing Company, 1967.

---

Published as a conference paper at ICLR 2025
Timo Flesch, Keno Juechems, Tsvetomira Dumbalska, Andrew Saxe, and Christopher Summerfield. Rich and lazy learning of task representations in brains and neural networks. BioRxiv, pp. 2021- 04, 2021.
Timo Flesch, Keno Juechems, Tsvetomira Dumbalska, Andrew Saxe, and Christopher Summerfield. Orthogonal representations for robust context-dependent task performance in brains and neural networks. Neuron, 110(7):258--2770 2022.
David J. Freedman, Maximilian Riesenhuber, Tomaso Poggio, and Earl K. Miller. Comparison of categorization of stimuli in the monkey prefrontal cortex: Frontal eye field and inferior convexity. Journal of Neurophysiology. 90(2):670 2003.
Pieter M Goltstein, Sandra Reinert, Tobias Bonhoeffer, and Mark Hübener. Mouse visual cortex ar- represent perceptual and semantic features of learned visual categories. Nature neuroscience, (4(10))1441-1455 2021.
Takuya Ito and John D Murray. Multitask representations in the human cortex transform along a sensory-to-motor hierarchy. Nature neuroscience, 26(2):306-355, 2023.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernl: Convergence and gen-
eralization in neural networks. Advances in neural information processing systems, 31, 2018.
Mikiko Kadohisa and Donald A Wilson. Separate encoding of identity and similarity of complex familiar odors piriform cortex. Proceedings of the National Acudemy of Sciences, 103(41):
12206-1221,, 2006.
Clifford G Kentros, Naveen T Agnihotri, Samantha Streater, Robert D Hawkins, and Eric R Kan- del. Increased attention to spatial context increases both place field stability and spatial memory. Newron, 42(():283-299 2004.
Adam Kepecs, Naoshige Uchida, Hatim A Zariwala, and Zachary F Mainen. Neural correlates, computation and behavioural impact of decision confidence. Nature, 455(7210))227-233 2312 2008. URL URLhttps:: /uwmed. ncbi nlm. nih. /00//16902111//
doi: 10. 10.00///nuuur07000.
Tae Hoon Kim, Yajun Zhang, Julien Lecoq, Jason Jung, Jun Li, Hongkui Zeng, and Mark Schnitzer. Long-term optical access to an estimated one million neurons in the live mouse cortex. Cell Reports, 17(1))33385-339, 2016.
Yusuke Komura, Akifumi Nikkuni, Naofumi Hirashima, Takuya Uetake, and Atsushi Miyamoto. Responses of pulvinar neurons reflect subject's confidence in visual categorization. Nature Neuroscience, 16(6):749-755, 2013.
Nikolaus Kriegeskorte, Maricke Mur, and Peter A Bandettini. Representational similarity analysis- connecting the branches of systems neuroscience. Frontiers systems neuroscience, 2:249, 2008.
Kamesh Krishnamurthy, Ann Hermundstad, Thierry Mora, Aleksandra M Walezak, and Vijay Balasubramanian. Disorder and the neural representation of complex odors. Frontiers in Compu- tational Neuroscience, 1776, 2022.
Tanishq Kumar, Blake Bordelon, Samuel J Gershman, and Cengiz Pehlevan. Grokking as the tran- sition from lazy to rich training dynamics. arXiv preprint arXiv:2310.06110, 2023.
Armin Lak, Katsuhiro Nomoto, Mehdi Keramati, Masamichi Sakagami, and Adam Kepecs. Mid- brain dopamine neurons signal belief in choice accuracy during a perceptual decision. Current Biology, 27(6):821-82) 8321, 2017.
Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. To- wards understanding grokking: An effective theory of representation learning. Advances in Neu- ral Information Processing Systems, 35:34651-34663, 2022a.
Ziming Liu, Eric Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. arXiv preprint arXiv:2210.01111 2022b.
Elijah Lovejoy. Analysis of the overlearnig reversal effect. Psychological Review, 73(1):87, 1966.
12

---

Published as a conference paper at ICLR 2025
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint arXiv: 190.00990. 2019.
Kaifeng Lyu, Jikai Jin, Zhiyuan Li, Simon Shaolei Du, Jason D Lee, and Wei Hu. Dichotomy of early and late phase implicit biases provably induce grokking. In The Twelfth International Conference on Learning Representations, 2023.
Nicholas John Mackintosh. Further analysis the overtraining reversal effect. Journal of Compar- ative and Physiological Psychology, 67(2p2):1, 1969.
Emily A Mankin, Fraser T Sparks, Begum Slayyeh, Robert J Sutherland, Stefan Leutgeb, and Jill K Leutgeb. Neuronal code for extended time the hippocampus. Proceedings of the National Academy of Sciences, 109(4)):19462-19467, 2012.
Alexander Mathis, Dan Rokni, Vikrant Kapoor, Matthias Bethge, and Venkatesh N Murthy, Reading out olfactory receptors: feedforward circuits detect odors in mixtures without demixing. Neuron, 91(5): 110-111330 2016.
Alexander Matyasko and Lap-Pui Chau. Margin maximization for robust classification using deep learning. In 2017 International Joint Conference on Neural Networks (IJCNN), pp. 300-307. IEEE, 2017.
Philip G Mead. The effect of overtraining reversal shift behavior in rats reinforced with electrical brain stimulation. Physiological Psychology, 144)3330-32, 1973.
Mohamad Amin Mohamadi, Zhiyuan Li, Lei Wu, and Danica Sutherland. Why do you grok? a theoretical analysis of grokking modular addition. arXiv preprint arXiv: 12.322 2024.
Depen Morwani, Benjamin L Edelman, Costin-Andrei Oncescu, Rosie Zhao, and Sham Kakade. Feature emergence via margin maximization: case studies in algebraic tasks. arXiv preprint 1rX1v:2311.07568, 2023.
K Nakamura, Y Gu, D Kato, al. Representational drift in the mouse visual cortex. Nature Communications, 12:5186,20 2021.
Neel Nanda, Lawrence Chan, Tom Liberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretabilityy arXiv preprint MxXiv:2301.05217 2023.
Allen Newell and Paul Rosenbloom. Mechanisms of skill acquisition and the law of practice. In John R Anderson (ed.), Cognitive skills and their acquisition, pp. 1-55. Lawrence Erlbaum Associates, 1981.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. In Distill, 2020. https://distill.pu//20000
Bruno A. Olshausen and David J. Field. Emergence simple-cell receptive field properties by learning sparse code for natural images. Nature, 38 381(658))6007-609 1996.
Cengiz Pehlevan, Sreyas Mohan, and Dmitri B Chklovskii. Blind nonnegative source separation using biological neural networks. Neural computation, 29(11): (11111):2225225555 2954, 2017.
Mark D. Plumbley. Algorithms for nonnegative independent component analysis. IEEE Transac- tions Neural Networks, 14(3):534-543 2003.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- cralization beyond overfitting on small algorithmic datasets. arXiv preprint arX1v:2201.02.17, 2022.
Shanshan Qin. Shiva Farashahi, David Lipshutz, Anirvan M Sengupta, Dmitri B Chklovskii, and Cengiz Pehlevan. Coordinated drift of receptive fields in hebbian/ant-heebian network models during noisy representation learning. Nature Neuroscience, 26(2):339 2023.
Charles L Richman, Karol Knoblock, and Wayne Coussens 1. The overtraining reversal effect in rats: A function of task difficulty. The Quarterly journal of experimental psychology, 24(3): 291-298, 1972.
13

---

Published as a conference paper at ICLR 2025
Michael E Rule, Timothy O'Leary, and Christopher D Harvey. Causes and consequences of repre- sentational drift. Current opinion in neurobiology. 58:141-147, 2019.
Nicole C. Rust and James J. DiCarlo. Selectivity and tolerance ("invariance") both increase as visual information propagates from cortical area v4 Journal of Neuroscience, 30(39):12978-12999 2010.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6/20, 2013.
Carl E Schoonover, Sarah N Ohashi, Richard Axel, and Andrew JP Fink. Representational drift in primary olfactory cortex. Nature, 594(7864):54115444 2021.
Amin MD Shakhawat, Carolyn W Harley, and Qi Yuan. Arc visualization of odor objects reveals experience-dependeet ensemble sharpening, separation, and merging in anterior piriform cortex in adult rat. Journal of Neuroscience, 34(31):00206102210 2014.
Soudry, Hoffer, and Srebro. The implicit bias of gradient descent on separable data, arxiv. arXiv preprint arXiv: 10345, 2017.
Shyam Srinivasan and Charles F Stevens. A quantitative description of the mouse piriform cortex. bioRxiv, pp. 099002, 2017.
Peter Sterling and Simon Laughlin. Principles of Neural Design. MIT Press, 2015.
Merav Stern, Kevin A Bolding. LF Abbott, and Kevin M Franks. A transformation from temporal to ensemble coding in model of piriform cortex. Elife, 7:e34831, 2018.
LT Thompson and PJ Best. Long-term stability of the place-field activity of single units recorded from the dorsal hippocampus of freely behaving rats. Brain research, 509(2):299-30)) 1990.
Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. Explaining grokking through circuit efficiency. arXiv preprint arXiv:2309.02390 2023.
Peter Y Wang, Cristian Boboila, Matthew Chin, Alexandra Higashi-Howart Philip Shamash, Zheng Wu, Nicole P Stein, LF Abbott, and Richard Axel. Transient and persistent representatiion of odor value in prefrontal cortex. Neuron, 108(1):209-22) 224, 2020.
J. M. Warren. Overtraining and extradimensional shift learning by cats. Bulletin of the Psychonomic Society, 177377777-178, 1978. doi: 10.3758/BF0332966..
Li K Wenliang and Aaron R Seitz. Deep neural networks for modeling visual perceptual learning. Journal of Neuroscience, 38(27):6028-6)) 6044, 2018.
James CR Whittington, Will Dorrell, Surya Ganguli, and Timothy Behrens. Disentanglement with biological constraints: A theory of functional cell types. In The Eleventh International Conference on Learning Representations, 2023.
Amit Yashar and Rachel Denison. Feature reliability determines specificity and transfer of per- ceptual learning in orientation search. PLoS Computational Biology, 13(12):e1005882, 2017. 2017
Rafael Yuste. From the neuron doctrine to neural networks. Nature Reviews Neuroscience, 16(8): 487-497, 2015.
Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. arXiv preprint arXiv:2309.1600 16042, 2023.
Yaniv Ziv, Laurie D Burns, Erie Cocker, Elizabeth Hamel, Kunal Ghosh, Lacey Kitch, Abbas El Gamal, and Mark J Schnitzer. Long-term dynamics of cal hippocampal place codes. Nature neuroscience, 16(3):264-266, 2013.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 METHODOLOGICAL AND IMPLEMENTATION DETAILS
The raw data consists of around 70 sessions, each representing a recording of 20 random neurons on given mouse-day. The data is spikes over time for each neuron on each trial, of which there are 200-300 in session. After odor onset, mice are given 2.5s to lick in response, We count the number of spikes in the final 500ms part of this window and decode on this number. which gives us a Nx 7 neurons by trials spike count matrix to decode on for each mouse-day. We drop neurons with low (less than 4) number of spikes across all trials classifying them as unresponsive to the odors, z-score this matrix and use 10-fold cross validation to get decoding accuracy. over 20 iterations. Since session to session data is very noisy, we plot smoothed statistics using a sliding window and plot standard error of all statistics. It should be noted the last few days of mouse T overtrainining dropped because of notable inconsistencies in neural data such as the decoding accuracy plummeting to far below chance while behavior remained high, suggesting an unknown and idiosyncratic cause for unreliable decoding accuracy in this mouse [ trials, so every neuron contributed equally in our analysis.
Note that the activity matrix was z-scored across
A.2 TESTABLE PREDICTIONS & EXPERIMENTAL. PROPOSALS
A.2.1 A CONCRETE PROPOSAL FOR PIRIFORM CORTEX
While reanalysis of existing data suggests that rich learning during overtraining is plausible, reanal- ysis on its own insufficient on their own to confidently establish a new empirical phenomenology. In particular, since the focus of (Berners-I et al. 2023) is studying increase in target-selectivity and how training history affects the form of the learned representations, targeted experiments at- tempting to falsify our hypothesis may be helpful. We outline what one such experiment in piriform cortex may look like. Note that such an experimental setup can be applied to any part of sensory cortex in which at least hundreds of neurons can be recorded over several (10+) contiguous days of overtraining.
We propose an experiment of similar flavor that of (Berners-Leee al. 2023). with several important changes. These changes include:
Using several, as opposed to one, target odors would prevent the mice from memorizing the target and thus would make the task require more learning of odor identity, leaving more space for rich learning during overtraining.
We would include spectrum of difficult, held-out, examples, of n types if odorant is an n-hot vector of chemicals. Then, we would define j-probe trial as those that would share j elements with the target, allowing us to modulate distance in both Hamming and projected space, and therefore difficulty of classification as measured by distance to boundary.
Out of every 200 trials on a given mouse-day. type. This would be included during the initial training period of 8 days, as well as during and after overtraining, which is not done in (Berners-Lee et al.) 2023). This would ensure the test set has minimal impact on learning compared to the training set (which would be 200/n times more frequent).
would be probe trials, one for each
We would also ensure mice have much time to recover between overtraining sessions as learning sessions, so the same amount of rich learning can potentially take place, and that conditions during overtraining kept similar those seen during learning, as possible, to compare feature learning on equal footing.
A.2.2 IMPORTANT EXPERIMENTAL DETAILS FOR GENERAL SENSORY DISCRIMINATION TASKS
In seeking to test our hypothesis, there are few key experimental details that apply broadly to experiments on sensory cortex beyond olfaction.
Berners-L.cc al. 2023) include these unreliable decoding trials in their analysis.
15

---

Published as a conference paper at ICLR 2025
The task needs to involve diverse and novel stimuli. This can be measured by ensuring the ability to decode stimuli from population activity is not significantly higher than change before learning.
The task needs to be difficult, but possible to master. This is a tricky requirement: here, mice reach up to 90% behavioral performance, which is close to the upper bound possible given noise and fluctuating conditions. Conversely, the otherwise relevant work in vision of (Goltstein al. 2021) trains mice to discriminate gratings based on frequency and ori- entation. This task is not fully learnable, as the mice reach ceiling of 70%. Since a primary part of our hypothesis is that this late-time rich learning can persist when task performance is near-ceiling and thus in the absence of large error signal, an error rate of 30% is too high to test our hypothesis.
It is crucial to separate a part of the data that is dedicated toward learning the task, namely the "training" set, and a part of the data dedicated toward testing generalization, namely the "test" set. As in the piriform setting, constructing some test examples to be partic- ularly challenging by construction is likely to elucidate whether useful yet hidden repre- sentation learning takes place in sensory cortex during overtraining. Test trials should be infrequent so their effect on learning is small, but should be included throughout learning that notions of training and test performance can be determined for the entire training- overtraining-tee timeline.
Chronic recordings should include the evolution of population activity both during training and overtraining. In this work, we cannot examine whether representations begin lazy or rich during learning, since data and probe trials are only available during the overtraining, and not the training, period. We note that mice are good organisms for such a task be- cause larger animals, like macaques, take much longer and are more expensive train, and chronic and invasive recordings during training can run the risk of causing serious damage the organism. Conversely, smaller organisms like fruit flies cannot be trained solve difficult perceptual discrimination tasks with in-oivo neural recordings, in any meaningful sense.
On such setups, our theory predicts continued increases in decoding accuracy for category label for several days after training performance saturates and stops changing noticeably, and separation between task-relevant class representations.
A.3 CONTINUED INTRODUCTION
A third difficulty, in addition to the two mentioned in the main introduction, is that to see genuine representation learning during overtraining, the task must be difficult, but possible to master. In the deep learning setting of grokking, for instance, this is made precise by specifying that grokking occurs under a certain regime of task model alignment (Kumar et al. (023). Many settings in exper- imental neuroscience either study passive exposure (no perceptual task learning) (Rust & DiCarlo, 2010 FFreemann et al. 2003), use tasks which are easy (high initial decoding accuracy) for sensory cortex (Wang al.] 2020), or use tasks that are too hard in the sense that the animal does only slightly above chance by the end (Goltstein et al. 2021).
Closely related, but distinct from our desired phenomenology of study is that of representational drift. In neuroscience, our understanding of the stability of neural network representations over time has also been revised. Classical work in neuroscience operated under the implicit assumption that neural representations were supported by stable neuronal activity (Barnes et al. 1997] Thompson & Best 1990). While many neurons do show stable responses to stimuli, the recent discovery of "representational drift" shows that the population activity underlying some task-relevant stimulus can change over time (Kentros et al. 2004, Mankin et al., 2012 Ziv et al. 2013 Rule et al. 2019 Schoonover et al. 2021). Representational drift is commonly conceived
diffusive process (Qin et 2023), ) and there are arguments that such drift can offer advantages in cortical processing. or confer robustness (Nakamura et al. 2021 Buesing et al. 20011. Our focus is instead on the learning of representations during overtrainigg on a task, and therefore our phenomenology of interest is orthogonal to, but may potentially co-occur with, representational drift.
16

---

Published as a conference paper at ICLR 2025
A.4 MATHEMATICAL DETAILS ON FISHER DISCRIMINANT, LDA, AND MARGIN
Here we provide some background on the key statistical quantities we track, both in piriform cortex and synthetic model, and their relations each other. The Fisher Linear Discriminant (FD) a statistical quantity that measures the ratio of cluster center separation to intra-cluster variance. Informally, quantifies the signal noise (SNR) in binary classification problem in representation space, and is a quantity that Linear Discriminant Analysis (LDA) relies on for classification. It is particularly useful when the goal is reduce the dimensionality of data while preserving as much information as possible.
Consider dataset consisting of n samples, each of which belongs to one of two classes: C1 and C2- Let x; E R denote the d-dimensional feature vector of the i-th sample. The FD seeks projection vector w IR such that when the data projected onto this vector, the separation between the projected means of the two classes is maximized relative to the projected variance within each class.
Formally, the projection
data point x onto the vector w given by yı = w Iz-
The objective of the FD is to find w that maximizes the SNR
(11 222
J(w)
si + s2
where μr = w wwmm m1 and #22 w m2 are the means of the projected points for classes C1 and C2 respectively, with mi nı Ex.ec, F3 and m2 n2 Ex.ec, 2, being the mean vectors of the classes in the original feature space. The terms of and si represent the variances of the projected points for classes C₁ and Oz, respectively.
To express (ww) in terms of the original data, we can define the within-class scatter matrix S., and the between-class scatter matrix Sp as
S.
Σ (z, m1) T + Σ (x. m2) m2) 1CE01
T
E2
S (m² m2) m2)
Then, we have that the Fisher criterion can then be rewritten as
w T S.w J(w)
w Sww
Then to maximize (ww, we differentiate it with respect to w and set the derivative to zero. This leads to the generalized eigenvalue problem given by
S.w W.
The solution w that maximizes Jww) is the eigenvector corresponding the largest eigenvalue of the matrix S-1 Sy.
The FD closely related Linear Discriminant Analysis (LDA) in the following sense. In LDA, the assumption is that the data from each class is normally distributed with common covariance matrix Σ. The LDA decoding rule assigns new observation the class C1 or C2 based on which class's 's linear discriminant function is maximized with
1 T Σ E m. log TA: -2" -
(x(x) = T
where Hy is the prior probability of class Cs. and & E {1,2). The LDA solution involves projecting the data onto a domer-dimen space using the same criterion as the Fisher Linear Discrimi- nant, making the FD effectively equivalent to LDA when the data follows Gaussian distribution
17

---

Published as a conference paper at ICLR 2025
Mouse Q
House
House 5
Mouse
0.222
3.023
13
22-
Ped Kien
Bay
00
Cormitation
20.4 Cermiiation
-0.3
Cormiation
E35
NT Comelattes
Figure 8: Cluster separation (anticorrelation between representations for classes) over 500 random firing rate matrices More negative is better.
with our z-scored firing rate matrix. a denotes standard deviation.
with equal covariance matrices across classes. This is why track the Fisher Discriminant in our synthetic model: because we see mean and median LDA posterior probability on the correct class increase during overtraining, suggesting that SNR measured by Fisher Discriminant is also increasing.
The connection between FD and margin maximization be understood in simpler, more quali- tative terms: by examining the concept of separating hyperplanes in the context of binary classifi- cation. In margin-based methods such SVMs, the goal is find hyperplane that maximizes the margin, which is the distance between the hyperplane and the nearest points from each class (support vectors). The FD also seeks to maximize the separation between classes, but it does so by maximizing the ratio of the between-class variance to the within-class variance. It can be thought of as "soft margin" of sorts.
If the data is linearly separable, the direction w found by FD will tend to align with the direction that maximizes the margin between classes, although FD does not explicitly enforce the maximum margin criterion SVM does. However, in many cases, the Fisher criterion leads to a solution that approximates the maximum margin direction, particularly when the distributions the classes are approximately Gaussian with equal covariances, which is why practice these two quantities often tend to vary together, as indeed they do both in our neural data and our synthetic model. Regardless, they measure different things.
A.5 REPRESENTATIONAL SEPARATION ABLATION
Since Z-score the firing rates, the diagonals of the correlatation matrices will be positive by definition as they take the form || However, the expectation of the off-diagonals zero for set of random firing rates, with variance depending on dimension. If we are claiming there is significant separation in representations, we must check that the anticorrelation between the two firing rates far from chance for our dimension D 15. In Figure g we plot the distribution of anticorrelations for a mean-matched firing rate matrix with firing rate vectors drawn from a Gaussian for each mouse, and draw line represent the anticorrelation (representational quality, since the task is to discriminate two classes) both before and after the overtraining period. We plot these in Figure
We can see generally that 1) overtraining causes separation, in the sense that representations are more anticorrelated after overtraining compared to training, and that 2) even at the beginning of overtraining (red lines) the separation is nontrivial compared to chance. This makes sense because the mice can perform the task at ceiling, so we expect representations at the beginning of overtrain- ing (after week of overtraining) to be much better than chance. The caveats are that the data is noisy: for instance, Mouse has anticorrelation slightly decrease compared to the beginning of overtraining. Such limitations of the data emphasize that our findings suggestive rather than conclusive, and we aim to inspire further experimental work and pose our findings as hypotheses that this data suggest are plausible rather than definitively validated.
18

---

Published as a conference paper at ICLR 2025
Thiriing Los
Nergh Theiring
Trsh
Hersh
Poooue SMA
0.2
0.5
Opoc
Loecs 128
Eact Li0
Vrge
Taye
Figure 9: Loss function ablation for synthetic model that traces asymptotic margin maximization induced by cross-entropy the cause for late-time decrease in test loss/late-time separation of rep- resentations. "Late-time" refers to when training loss plateaus at a low value.
A.6 HARD MARGIN ABLATIONS
We begin with review of the Cross-Entropy vs Hinge loss definitions. Let y {0, 1) represent the true class label, and ÿ E [0, 1] represent the predicted probability for the positive class (i.e., Ply = 1 x) ). The cross-entropy loss LCE for a single example is defined as:
LCE(Y. 9) log + (1 v) log(1 9)).
In hinge loss, the labels y are typically encoded y E {-1, +1], and the model outputs a score f() E R. where the sign of f(x) determines the class prediction. For simplicity, let's assume f(x) the output linear model, f(z)
x b.
The hinge loss Luinge for a single example is defined as:
Lhinge (y. f(x)) = max(0, f()),,
where C' is the hard margin parameter to be specified, which we take as C = 1. The key point is that in Hinge objective, loss vanishes after the learned classifier classifies each training example with margin at least C, whereas loss does not vanish in this discontinuous way for the Cross-Entropy objective, driving continued late-time learning. We can see in Figure Iihat test loss stops improving when train loss does, and this is coincedent with an abrupt ceasation in the increase in margin, in contrst to the continued margin maximization driven by Cross-Entropp the main text.
A.7 BIOLOGICALLY PLAUSIBLE MODEL
We now repeat the above experimental setup of our synthetic model with an architecture that takes piriform cortex more seriously (Stern et al. 2018 Krishnamurthy et al. 2022 Mathis al., 2016).
Our biologically plausible model consists of three main processing layers plus an output layer, with additional feedback connections to model recurrent dynamics. The first layer serves a sensory input layer, analogous to inputs from the olfactory bulb. projects the input to a higher-dimensionall space, using ReLU activation followed by dropout with probability 0.2 to maintain sparse coding, a key feature of sensory representations in the piriform cortex. The second layer acts as a dense associative layer, modeling the extensive recurrent connectivity between pyramidal cells. This layer maintains the expanded dimensionality from layer one and includes a parallel feedback pathway. TTee feedback implemented through separate linear transformation followed by ReLUU activation,
19

---

Published as a conference paper at ICLR 2025
Test
Kargin Time
14-
Nups
04
02
3.5
361
341
10"
36
361
30 tpoch
101
10
Spock
Epoch
Epach 400
Epoch 10000
Taay Prodes
Daye hobra
Degat Fabe
20
22
PC1
PC3
P12
Figure 10: Architecture ablation for biologically plausible model of piriform cortex trained on the same binary classification task to separate target and nontarget odors. We see loss on held out probe odors continues to decrease after low train loss is achieved, since cross-entropy loss is again used as the objective.
and the result added back to the main pathway via a skip connection. Both the main and feedback pathways use the same dimensionality layer implements modulatory control, inspired by inhibitory interneuron circuits. This layer reduces dimensionality compared to layer two and uses LeakyReLU activation followed by a higher dropout rate (0.3) to model inhibitory effects. The reduction in dimensionality reflects the modulatory rather than representational role of this layer. The output layer performs final linear transformation single scalar value (since we're training on binary classification). We train without weight decay and learning rate le-2. We also add Gaussian noise to the forward pass to simulate stochastic activity in sensory cortex, hence why there are multiple target odor representations.
preserve the associative nature of this layer. The third
Results are shown in Figure 10 The persistence of the phenomenology identified in the main text suggests that the objective function and task structure, rather than architectural details, are the key here.
20


Reviews:
[{"review_id": "Ze0ZFyWCor", "reviewer": "Reviewer_KXYs", "summary": "The authors examine the phenomenon of overtraining in posterior piriform cortex of mice.  They show that class separation continues after performance saturates.  They further show that the margin of the maximum margin classifier increases in the overtraining period.  They construct a simple model, which is a one-hidden layer MLP, that shows this overtraining behavior.  They propose that this model provides an explanation for over-training reversal in animal learning.", "questions": "This seems like it would be a very general phenomenon, in particular because natural gradient descent dynamics should keep lowering the loss, which by construction has to maximize some discriminant in the data being classified.  At the same time, the training samples are discrete so you are guaranteed to saturate the training error (i.e. you can’t get better than perfect on a finite set).  Your kernel argument at the end seems to back up this intuition.  What properties of networks would you envision would prevent this behavior?  Or what kinds of tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is very clear and well presented.  The basic result is simple and backed by both empirical data and a simple model.  Most of the figures are very clear and concise.  That the observation can be explained by so simple a model is nice.  The explanation of reversal training is nice.", "weaknesses": "I think there are a few improvements that could be made to the figures:\n\nFigure 1 could use a quantitative indication of performance, so that it is clear when overtraining starts (as opposed to this being buried in the caption).  \n\nFigure 2 is slightly confusing in terms of which column in each row represents what.  Do I understand correctly that the first column is the average and the others are the first and last 3 days, respectively?  If so, this could be labeled better.  If not, this could be labeled much better.  \n\nIn Figure 3, maybe add an indication of the cluster mean with a line separating the clusters to make the separation more clear?  It’s not bad now, but some of these are rather close.  \n\nIn Figure 5, panel labeled “Epoch 1000”, the target dot is obscured.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2sAyNLFSqt", "reviewer": "Reviewer_aDZL", "summary": "This study examines grokking, an interesting phenomenon observed in both deep learning models and in animal learning. The authors re-examine a published dataset that tracks neural activity in the posterior piriform cortex (PPC) of mice trained to perform a multi-odor discrimination task. The authors found that the class margin measured in the neural activity space of PPC continued to increase after performance accuracy has reached a ceiling. In a simple MLP with one hidden layer trained to perform an in silico “odor” discrimination task, the authors found continued improvement in class margin measured in latent space, and “grokking” phenomenon in probe trial accuracy. Using this simple model, the authors also provide theoretical explanation and empirical demonstration of overtrained reversal, a phenomenon long reported in experimental psychology.", "questions": "Could the authors comment specifically on:\n1)\tHow results from the MLP model would generalize to more biologically constrained models of the PPC (or other olfactory processing circuits)?\n2)\tHow results from MLP could generalize to more complex DL models for classification?\n3)\tDemonstrate a causal link between margin maximization and the emergence of grokking in the MLP model?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The study draws an interesting connection between the grokking phenomenon in deep learning and to the gradual evolution of neural representation in mice overtrained on classification tasks. By using a simple MLP model to demonstrate the link between increase in class margin and the emergence of “grokking”, the authors highlight a promising mechanism by which grokking may emerge in biological and artificial learning systems.", "weaknesses": "1)\tThe architecture of the MLP model is likely far removed from the actual structure and connectivity of the PPC, hence it appears more like a toy model to demonstrate the cooccurrence of margin maximization and grokking. Whether similar phenomenon would emerge in more sophisticated models of olfactory cortex (or olfactory circuits in lower animals) remain to be tested;\n2)\tWhile the emergence of grokking in the MLP model is interesting, the authors have yet to test whether the observed increase in class margin is causally related to grokking. The authors cite related theory, e.g. setting margin maximization as an optimization objective could drive the emergence of grokking. This proposed causal link could be conveniently tested in the MLP model.  \n3)\tWithout further analysis, it is unclear how the link between class margin maximization and grokking in more complex DL models, e.g. models trained to classify images. While the authors provide a theoretical basis for how overtrained reversal occurs, a theoretical advance on why grokking occurs remains lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lHi7DAQOFW", "reviewer": "Reviewer_CDUr", "summary": "This paper examines an intriguing phenomenon where neural representations in mouse sensory cortex continue to evolve and improve even after behavioral performance has reached ceiling levels (Berners-Lee et al. 2022). The authors reanalyze neural recording data from mice learning an odor discrimination task and find evidence for \"hidden progress\" in how odor representations separate in piriform cortex during overtraining, similar to the \"grokking\" phenomenon observed in deep neural networks. They propose that this continued refinement of representations reflects implicit margin maximization and demonstrate this using a simplified neural network model. The paper makes novel connections between neuroscience and machine learning while offering new insights into both domains.", "questions": "Suggestions for Improvement:\n1. Tone down claims about biological fidelity and instead emphasize that the model captures key computational principles at an abstract level\n2. Add discussion of connections to deep learning literature on perceptual learning, which provides complementary perspectives on how representations evolve during training\n3. Clarify the precise definition of margin maximization being used, particularly in the context of evolving representations.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Makes an important connection between seemingly disparate phenomena in biological and artificial neural networks (hidden representational changes during overtraining, grokking, reversal learning)\n- Provides a compelling mathematical framework (margin maximization) that helps explain the neurobiological data\n- Offers a novel explanation for the classic \"overtraining reversal effect\" from psychology in terms of rich feature learning\n- Good empirical validation through neural data analysis and synthetic modeling\n- Mostly clear writing and effective visualization", "weaknesses": "- The characterization of their model as \"biologically faithful\" feels overstated, given the significant abstractions from real neural circuits\n- The paper should cite relevant literature on deep learning approaches to perceptual learning, particularly work by Wenliang and Seitz (2019) that examines how perceptual learning can be explained as changing readouts from complex features\n- The explanation of margin maximization was a bit inscrutable, particularly the part about distinguishing between different notions when a decoder is being retrained on evolving features. It could use a rewrite and tightening up, perhaps a subfigure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tDrqQo8YYz", "reviewer": "Reviewer_CcRH", "summary": "The paper demonstrates that animals can learn tasks and improve generalizability through over-training. The study begins by reproducing and reanalyzing experiments conducted on mice (Berners-Lee et al., 2023) and then proceeds to run toy experiments as proof-of-concept illustrations.", "questions": "I am unclear about the contribution of the section \"A mathematical model based on rich learning\" on page 9. If the authors use a kernel method, isn’t the result somewhat trivial? Furthermore, the logical connection between the content of this section and the conclusion — “An animal overtrained on a task will have better task-model alignment and a richer feature set than an animal whose learning was stopped after behavioral plateau” — is unclear. Could the authors elaborate on this section?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper addresses interesting topics at the intersection of foundation models and cognitive science, particularly within the realm of behavioral psychology.", "weaknesses": "**1. Writing Quality:**\n\n* *Citation:* The poor writing significantly hinders the evaluation of this work. For instance, the authors misuse \\citet and \\citep throughout the paper. The frequency of these errors affects the paper’s professionalism; please revise these accordingly.\n\n* *Confusing Notation:*\nAlthough Berners-Lee et al. (2023) represent each odor as a vector in their figures, the description \"n-hot vector of $k$ possible odorants\" may confuse readers. Given that ICLR’s primary audience is in machine learning, they may interpret this as a traditional vector representation, though in the authors' experiment it represents a combination of odorants rather than a vector.\n\n* *Undefined Notations:*\nSeveral notations lack definitions (e.g., [N, T] in line 755) or exact values (e.g., $n$ in $n$-hot vector).\n\n**2. Validity of Experimental Setting:**\n\nThe results could be heavily dependent on the experimental setting. In the default setup, each odor is represented as a combination of odorants, defined by an $n$-hot vector, with limited overlap between odorants. Given the network’s simple architecture (a single fully connected layer), each odorant is almost linearly separable, meaning there is minimal interference among components. This could explain why the model resists overfitting and maintains increasing test accuracy even during overtraining. Conversely, as seen in Figure 6, when odor overlap is greater, neither the Fisher Linear Discriminant nor the test accuracy increases significantly beyond 100 epochs, at which point the model reaches perfect training accuracy.\n\n**3. Loss Interpretation:**\n\nThe authors claim in line 370 that after achieving perfect training accuracy, the loss \"plateaus.\" However, it appears to continue decreasing (though this could be subjective). Additionally, perfect training accuracy does not imply the model has fully learned from the training data, especially since cross-entropy loss is used, and accuracy is based on the argmax classification.\n\n**4. Miscellaneous**\n\n* Lines 457–484: This section needs refinement. It is unclear how the original features differ from pre-trained features (L461–463), the relationship between $K$ and $K_0$ (L465), and the contrast between the original model’s readout and the randomly initialized network (L482).\n\n* Figure 1: The caption refers to (a) and (b), but these labels are absent in the figure.\n\n* Line 355: “are are” should be corrected to “are.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)