Understanding the Learning Phases in Self-Supervised Learning via Critical Periods
Authors: JangHyeon Lee
Philipe Ambrozio Dias
Yao-Yi Chiang
Dalton Lunga
Venue: ICLR 2026 Poster (2026)
Keywords: Learning Phases
Critical Periods
Self-Supervised Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Self-supervised learning (SSL) has emerged as a powerful pretraining strategy to learn transferable representations from unlabeled data. Yet, it remains unclear how long SSL models should be pretrained to yield such representations. Contrary to the prevailing heuristic that longer pretraining translates to better downstream performance, we observe a transferability trade-off: across diverse SSL settings, intermediate checkpoints can yield stronger out-of-domain (OOD) generalization, whereas additional pretraining primarily benefits in-domain (ID) performance. From this observation, we hypothesize that SSL progresses through learning phases that can be characterized via the lens of critical periods (CP). Prior work on CP has shown that supervised models exhibit an early phase of high plasticity, followed by a consolidation phase where adaptability declines but task-specific performance increases. Since traditional CP analysis was developed for supervised settings, we rethink it for SSL in two ways. First, we inject deficits to perturb the pretraining data and assess their lasting impact on representation quality via downstream tasks. Second, we compute the Fisher Information on pretext objectives to track plasticity, quantifying how sensitive model parameters are to the pretext task. Our experiments suggest that SSL models may exhibit their own CP, with CP closure coinciding with a sweet spot for broad downstream transferability. Leveraging these insights, we introduce CP-guided checkpoint selection as a strategy for selecting checkpoints that offer stronger OOD transferability. Finally, to balance the transferability trade-off, we present CP-guided self-distillation, which selectively distills layer representations from the intermediate checkpoint into their overspecialized counterparts in the final checkpoint.

Published as a conference paper at ICLR 2026
UNDERSTANDING THE LEARNING PHASES IN SELF- SUPERVISED LEARNING VIA CRITICAL PERIODS
JangHyeon Lee'
Philippe Ambrozio Dias²
Yao-Yi Chiang'
Dalton Lunga?
University of Minnesota
"Oak Ridge National Laboratory
ABSTRACT
Self-supervised learning (SSL) has emerged as a powerful pretraining strategy to learn transferable representations from unlabeled data. Yet, it remains unclear how long SSL models should be pretrained to yield such representations. Con- trary the prevailing heuristic that longer pretraining translates to better down- stream performance, we observe transferability trade-off: across diverse SSL settings, intermediate checkpoints can yield stronger out-of-domain (OOD) gen- eralization, whereas additional pretraining primarily benefits in-domain (ID) per- formance. From this observation, we hypothesize that SSL progresses through learning phases that can be characterized via the lens of critical periods (CP). Prior work on CP has shown that supervised models exhibit an early phase of high plasticity, followed by consolidation phase where adaptability declines but task- specific performance increases. Since traditional CP analysis was developed for supervised settings, rethink for SSL in two ways. First, inject deficits to perturb the pretraining data and assess their lasting impact on representation quality via downstream tasks. Second, we compute the Fisher Information on pretext objectives to track plasticity, quantifying how sensitive model parameters are to the pretext task. Our experiments suggest that SSL models may exhibit their own CP, with CP closure coinciding with a sweet spot for broad downstream transferability. Leveraging these insights, we introduce CP-guided checkpoint se- lection as strategy for selecting checkpoints that offer stronger OOD transfer- ability. Finally, to balance the transferability trade-off, we present CP- CP-guided self- distillation, which selectively distills layer representations from the intermediate checkpoint into their overspecialized counterparts in the final checkpoint.
INTRODUCTION
Self-supervised learning (SSL) leverages pretext tasks (e.g., contrasting views predicting masked inputs) to learn representations from unlabeled data that transfer well to downstream tasks (Liu et al.) Ozbulak et al. 2023 Gui et al. 2024). While prior work has studied how well SSL models transfer (Newell & Deng 2020 Ericsson et al. 2021a Hu et al., [02I), why they transfer (Saunshi al. 2019 Ericsson al. 2021b), and under what conditions they succeed (Tian et al. 2020 Zhao et al.
2021
2020: Cole et al.) 2022; Dubois et al. 2022 2023). it remains unclear how long to pretrain
SSL models for transterable representations to emerge.
Without knowing when the SSL model has learned enough from its pretext task, pretraining risks both under- and over-training. Stopping too early yields underdeveloped representations, leading to the common heuristic that longer pretraining is better (Chen et al. 20200] He et al.) 2022). However, pretraining for too long increases computational costs and risks overfitting to the pretext objective, potentially at the expense of broad downstream transfcrability.
Determining the optimal pretraining duration difficult because SSL objectives are only implicitly aligned with downstream transferability (Balestriero et al.) 2023 Reizinger al. 2025). Typically, the quality of SSL representations is assessed after pretraining via linear probing or finetuning (Chen et al. 20200 Kumar et al. 2022 Balestriero al. 2023). Such post-hoc evaluation is costly repeat across tasks and, more importantly, provides no guidance during pretraining about whether learned representations underdeveloped or already overspecialized the pretext task.

---

Published as a conference paper at ICLR 2026
(Oue-at-Domain pask)
(In Un--oomain paaki)
Downstream transter Performance
Fisher Information
Cittical
Planticht
Consolidation
Dvarapecialization
Checkpoint Sequence
Protraining Epech
Figure 1: (Left) Conceptual schematic of downstream performance across pretrained checkpoints. In-domain (ID) performance increases with pretraining, whereas oot-of-domain (OOD) transferabil- ity peaks at an intermediate checkpoint. (Right) Conceptual schematic of Fisher Information (FI) dynamics during SSL pretraining. The curve shows three phases. Phase I (Plasticity) shows a rise in Fl when representations are highly sensitive to changes. Phase (Consolidatiin) shows decline in FI as representations stabilize. The critical period (CP) closes once FI stabilizes, before Phase (Overspecialization), where OOD performance declines. Red shading highlights the loss of trans- ferability beyond CP closure. (Takeaway) CP closure may mark a sweet spot for broad transfer.
By evaluating checkpoints across the SSL pretraining trajectory (Fig. left) observe a trans- ferability trade-off:: intermediate checkpoints tend to achieve better out-of-domain (OOD) transfer than later checkpoints, whereas extended pretraining primarily benefits in in-domain (ID) performance. Here, ID denotes finetuning on a labeled version of the pretraining dataset, while OOD denotes finetuning on labeled datasets drawn from different distributions (Marks et al. 2025). This transfer- ability trade-off pattern implies that pretraining does not simply yield representations that improve uniformly across settings. Instead, we hypothesize that SSL progresses through learning phases whose representations differ in transferability: earlier phases may favor OOD generalization, while later phases specialize toward the pretraining distribution.
Prior work (Achille et al.] (018)) shows that, much like biological systems, neural networks exhibit CP: an early window of high plasticity followed by consolidation phase where representations stabilize and adaptability declines. In supervised learning, these phases were revealed through perturbation experiments, where temporary distortions of the training data permanently impaired generalization when applied during early epochs but had little effect later. This temporal sensitivity can be ex- plained by Fisher Information (FI) (Fisher 1925), which quantifies how strongly small parameter changes affect model predictions and serves as proxy for information plasticity (Achille et al.] 2018 Berariu al. 202I). Early in training, Fl rises and plasticity high, input perturbations strongly reshape representations and leave lasting effects. As training continues, Fl declines and representations consolidate, so later perturbations have little impact. Overall, CP analyses reveal when representations are adaptable or rigid, which may offer insights into transferability.
To build intuition, we draw on the notion of critical periods (CP) (Hensch 2004
Yet critical periods (CP) have not been studied SSL, where transferability between pretraining and downstream tasks is key. Unlike supervised learning, SSL derives its supervisory signal from the data itself rather than explicit labels, making prior CP analyses not directly applicable. We therefore reformulate CP analyses to track information plasticity during SSL. This is achieved in two ways: (1) applying perturbations during pretraining to test stage-Wise effects on downstream transferability, and (2) redefining Fisher Information (FI) with respect to pretext tasks.
We find that SSL pretraining may also exhibit a structured progression (Fig. right). During pre- training. Fl rises and then declines, marking transition from plasticity consolidation. We observe CP closure at the end of consolidation, where OOD performance tends to peak. Beyond CP closure, OOD generalization degrades, suggesting phase of overspecialization. This pattern aligns with empirical finding that intermediate SSL checkpoints can transfer better OOD than later ones.
Building these observations, transferable SSL. We introduce two CP-guided strategies. CP-guided checkpoint selection uses CP closure to select checkpoints that favor OOD transfer, while pretraining beyond closure benefits II)
explore whether critical periods (CP) inform efficient and
2

---

Published as a conference paper at ICLR 2026
performance. To balance this trade-off, CP-guided self- diffilistillation distills early-layer features from the CP closure checkpoint into the early layers of the final checkpoint during finetuning, while leav- ing later layers intact. We target early layers because they are widely understood to encode general features YYsiiskii et al.) 2014: Skean et al., 2025). which may erode with prolonged pretraining, while later layers encode task tasc-spe information tied to the training objective, which in SSL is the pretext task (Bordes et al. 2022 2023 Ouyang et al. 2025).
Our contributions can be summarized as follows:
We observe transferabiliity trade-off in SSL pretraining. Intermediate checkpoints can yield stronger out-of-domain (OOD) transferability, while models pretrained longer tend to improve in-domain (ID) accuracy. This calls for rethinking the standard practice in SSL that longer pretraining translates to better representations for downstream tasks ($2)
We connect this phenomenon the notion of critical periods (CP), providing the first study of CP in SSL and their impact on transferability. Since SSL objectives differ from su- pervised settings, we reformulate CP analyses for SSL by introducing perturbations into pretraining and redefining Fisher Information in terms of pretext tasks. These analyses suggest that SSL models may exhibit their own CP ($3
We characterize an overspecialization phase, where prolonged pretraining reduces OOD generalization. Building on this insight, we introduce two interventions: CP-guided check- point selection, which uses CP closure to select checkpoints with stronger OOD transfer- ability, and CP-guided self-distillation, which distills early-layer features from CP check- points into later checkpoints to recover OOD performance while retaining ID strength ($4).
DOES LONGER SELF-SUPERVISED PRETRAINING ALWAYS IMPROVE DOWNSTREAM TRANSFERABILITY?
Prior work in self- self-supervised learning (SSL) reported that longer pretraining improves downstream performance (Goyal et al., 2019; Chen et al. 2020 He et al. 2022). This has led to the de facto prac- tice of pretraining SSL models for as long as compute budgets allow. We find that this improvement does not universally hold. Instead, we observe a transferability trade-off: while extended pretrain- ing improves in-domain (ID) performance, it can diminish out-of-domain (OOD) transferabiliiy.
2.1 EXPERIMENTAL SETUP
To study how pretraining duration affects downstream transferability. evaluate two families of SSL: discriminative and generative (Ozbulak et al., 2023). For discriminative SSL, we include a contrastive method, SimCLR (Chen et al. 2020), and non-contrastive methods, VICReg (Bardes) 202T) and DINO (Caron et al. 2021) ). For generative SSL, we use MAE (He et al. 2022). For architectures,
ct al.
canonical ResNet (He et al. 2016) for SimCLR and VICReg. and vision transformer Dosovitskiy et al., 2020) for DINO and MAE. We pretrain on fMoW-RGB (Christic) 2018). large-scale remote sensing dataset with well-defined distribution shifts (Koh al. Rolf et al. 2024). making it a natural testbed for ID and OOD transfer.
al.
2021
We pretrain each model from scratch for 1000 epochs, saving checkpoints every 50 epochs. Down- stream transfer is evaluated along two dimensions (Marks et al. (2025). In-domain (ID) performance measured by finetuning and evaluating on a labeled version the pretraining data. Out-of-domain (OOD) performance is measured on datasets whose distribution differs from the pretraining data. For each checkpoint, we finetune and compare against the model pretrained for 1000 epochs, which we refer to the final checkpoint. Details on pretraining and downstream evaluation are in AA and
additional ImageNet (Deng et al. 2009) experiments B
2.2 RESULTS
Fig. shows downstream performance of SSL models across pretraining durations. OOD transfer peaks at intermediate checkpoints, while ID performance tends to increase. VICReg (Bardes et al.] DINO (Caron al. 2021), and MAE (He al. 2022) all reach peak OOD performance
2021,
3

---

Published as a conference paper at ICLR 2026
intermediate checkpoints, while SimCLR (Chen et al., 2020) follows the same pattern but peaks siderably later. Although the exact timing varies, this divergence between OOD and ID is evident, suggesting that intermediate checkpoints can yield more broadly transferable representations.
SimCUR-RW50-PMA
MICReg-RN550-MMO
DINO-VIT-016-MMY
NAE-VIT-B16-MMo
0.19
Perfermance
-0.05
0.00
0.03
0.00
0.00
0.00
-0.13
250
530
-0.00
-0433
1000
-0.05
750
100n
Checkpoint Sequercee
Checkpoint Sequence
Checkpoint Sequence
Checkipennt Sequence
TMoW-val (I)
(MoW-WILDS (OOD)
EuroSAT (OOD)
EuroSAT-Spatial (OOD)
Figure 2: Transferability trade-off in SSL. The x-axis denotes pretraining epoch (checkpoints saved every 50 epochs), and the y-axis shows the change in performance relative to the final checkpoint.
3 CRITICAL PERIODS IN SELF-SUPERVISED LEARNING
Insights from $2 raise the question: why do different stages of pretraining yield different transfer properties? We hypothesize that SSL pretraining progresses through structured learning phases. To examine this, we draw on the notion of critical periods (CP) (Hensch] (004). Prior work shows that neural networks exhibit phases of early plasticity, when representations are highly sensitive to change, followed by a consolidation phase with reduced plasticity (Achille et al., 2018) Kim al., 2023). Yet whether such phases exist in SSL, and how they relate to transferability. remains unexplored. If SSL models pass through similar transitions, probing when plasticity is present or lost during pretraining may help explain the mechanism behind the observed transferability trade-off.
To investigate this, we revisit traditional critical period analyses in supervised learning ($3.)) fol- lowed by their reformulation for SSL. via two approaches: perturbation experiments on unlabeled pretraining data ($3.2) and tracking Fisher Information pretext tasks 3.3
3.1 PRIOR CRITICAL PERIOD ANALYSES REQUIRE RETHINKING FOR SSL
How critical periods (CP) have been studied in supervised learning (SL). Prior work identifies CP two ways (Achille al.) 2018). First, perturbation experiments probe whether the timing of perturbations matters. If altering the input data distribution early in training degrades overall task performance, while the same change later has little effect, this marks critical early phase. Second, Fisher Information (FI) (Fisher 1925) analysis provides continuous marker of plasticity. Put sim- ply, Fl quantifies how sensitive the model's predictions are to changes in its parameters (van de Ven, 2023). Achille et al. (2018) show that during training, FI rises then declines. Deficits introduced early, while is high, leave lasting effects because the model is actively forming its representations. Once Fl declines, the model has consolidated and the same deficits have little effect.
Why critical period analyses must be rethought for self-supervised learning (SSL.. The two probes above rely on supervised downstream signals. However, SSL pretraining is decoupled from downstream labels and optimizes proxy objectives on unlabeled data. One could study critical pe- riods during finetuning, when downstream labels are available, but this only reveals how fixed representation adapts to one task, not how transferable representations emerge during pretraining. Our focus is SSL pretraining itself, since this stage defines the representations that constrain all downstream learning. In particular, pretraining on unlabeled data DA can be viewed as producing posterior p(0 DA) that serves as the prior for any downstream task DB (Shwartz-Ziv et al. 2022). To understand how this prior evolves, critical periods must be analyzed during pretraining. not after.
Probing critical periods in SSL. To study critical periods in SSL, we introduce two probes during pretraining. (1) Deficit injection on the unlabeled pretraining data perturbs the input distribution. The pretext task remains unchanged, but the self suf-suuperviiion signal is degraded (e.g., input pertur- bations remove fine-graiig men-grained cues, making data pairs harder to align or reconstructions less informa- tive). By varying when deficits are introduced and measuring their impact on downstream transfer, can identify phases when representations are more or less sensitive to change. (2) Fisher Infor-
4

---

Published as a conference paper at ICLR 2026
mation (FI) on pretext objectives quantifies the sensitivity of model parameters to the supervisory signal defined by the pretext tasks. Tracking FI over pretraining reveals when parameters remain adaptable and when they consolidate, which is crucial in SSL since the value of pretraining lies in producing transferable representations. Identifying when representations are still malleable versus when they have stabilized can help explain when they effective for downstream transfer.
3.2 PROBE 1: DEFICIT INJECTION ON PRETRAINING DATA
The central question is: does the impact of input perturbations during SSL pretraining depend on when they occur? If perturbations early in pretraining change the final representations, as reflected in downstream task performance, while the same perturbations later in pretraining have little effect, this would indicate a critical period in SSL.
Let D {x. denote samples from clean distribution p(z). A model learns representation function fe : X + Id with parameters 0, trained with a self-supervised loss (f)())) To inject deficits, we replace clean training data with data drawn from perturbed distribution p (r) starting at onset epoch to and lasting for duration of AL epochs. After this window, training resumes on clean data until epoch T, where T t0- + AL.
We denote the encoder trained entirely on clean data as fa (baseline) and the encoder trained with a deficit window as før To quantify the effect of the intervention, we compare downstream task performance between these models. Let o(-) denote downstream evaluation metric (e.g., classifi- cation accuracy). The sensitivity score is defined as
S(to) (ffe) (for).
This score reflects the relative degradation in downstream task performance caused by the perturba- tion. critical period exists if early perturbations yield higher sensitivity than later ones.
Deficit settings. Following prior work, we simulate sensory deprivation by replacing inputs with Gaussian noise (Achille et al. 2018). For SSL methods, the pretext objectives (e.g., contrastive alignment (Chen al. 2020) or masked reconstruction (He et al.) 2022.) continue updating dur- ing the deficit window, but the supervisory signal now comes from noise rather than meaningful images. As result, the model may learn nuisance features that are not useful for downstream trans- fer (Soatto & Chiuso 2014; Achille & Soatto| 2018). Each deficit is applied for a fixed window (5, 30, 50 epochs) at varying onset times lo: early (epoch 0), middle (epoch 450), and late (epoch 750), following (Kleinman et al. 2024). After the deficit window, training resumes on clean inputs (until epoch T' 1000). We use the same evaluation settings as $2.1
Early SSL pretraining phases are sensitive. Fig. 3 shows the sensitivity S(to) of learned repre- sentations noise deficits introduced at different times during pretraining. Across the evaluated SSL settings, we find that deficits applied at the start of pretraining lead to larger degradation than when the same deficits are introduced later. While the absolute magnitude of sensitivity varies by method, a similar pattern emerges: the beginning of pretraining appears to be sensitive window where perturbations to the data distribution can leave lasting effects on learned representations.
SimCLR RN50-MMeW
VICReg RN50-MMe
DINO DINO-VIT-11--MMU
MAE WTT-866-MMM
Sensitiitty
1.00/
1.00
0.75
0.50
050
0.50
0.50
0.25
0.25
0.25
0.25
0.00
Miadde
Early (0nsec 01
Meddie (Orse: 450)
Early [Onset
Ridde
Ewrl (Onset
Midsi
Lake (0rse: 750)
(0)ce10)
10nset 452)
(Onset 7501
10nset 7500
(Onset 4501
(Ōnse: 750)
[0nset 450)
duration
epochs
30 epochs
50 opochs
Figure 3: Sensitivity S(1)) input perturbations introduced at different stages of SSL pretraining on IMoW. Checkpoints are then finetuned on IMoW-train and evaluated on held-out iMoW-val, Higher values indicate stronger lasting degradation in downstream accuracy relative clean baseline.
5

---

Published as a conference paper at ICLR 2026
3.3 PROBE 2: TRACKING FISHER INFORMATION ON PRETEXT OBJECTIVES
Perturbation experiments reveal whether temporary interventions have lasting effects, but they do not explain why sensitivity varies across pretraining. To provide an analytical view, we study the evolu- tion of Fisher Information (FI) (Fisher 1925) during SSL pretraining. FI measures the sensitivity of the model's s predictions to perturbations in its parameters and has been used to quantify parameter importance (Amari 1998; Kirkpatrick et al. 2017). As a positive semi-definite approximation of the Hessian Martens 2020), Fl also reflects the local curvature of the loss landscape (Lewandowski ct al. 2023). Unlike the full Hessian, its trace can be estimated efficiently (Achille et al. 2018).
In SSL, pretext tasks define supervisory targets y derived from the input æ (Balestriero et al. 2023]
Balestriero & LeCum 2024). For instance, in contrastive learning, y specifies positive and negative pairs from augmentations of z, while in masked image modeling, y denotes masked input regions to be reconstructed. More generally, these pretext objectives can he formulated optimizing conditional distribution po(y(a)) (Wang et al. 2024a Bizeul et al. 2024 Alshammari et al.) 2025).
From this perspective, Fl computed on py ( (:) quantifies how sensitive parameters are to the pretext supervisory signal, and tracking this quantity over training reveals how that sensitivity evolves. Prior work has linked FI dynamics to plasticity (Achille et al. 2018 Berariu et al., 202T): Fl rises as the model traverses high-curvature regions during early training, then declines as representations consolidate into flat minima. Since downstream tasks inherit these learned representations, plasticity dynamics can influence transferability. Specifically, models that lose plasticity risk settling into representations that are harder to adapt, while those that retain plasticity can readily adjust to new tasks (Achille et al. 2018 Lyle et al. 2023 Dohare et al., 2024 Springer et al.) (025).
Rdd trained on inputs æ ~ p(x)) where p(x) is the empirical distribution of D. To quantify local sensitivity, we consider infinitesimal perturbation of the parameters, = 0 + 80. The effect of this perturbation measured by the Kullback-Leible (KL)
Consider a model with parameters
divergence between per (y) (z) and (v())z). A second-order Taylor expansion gives
Ex~ (L(po x) por (v/=)) = 1 T F F80 + o(180112)
(2)
where the Fisher Information Matrix (FIM) is
F= Exp((e) E Eyypollls)
V logpo(y(=))
V log logpo(ul=)"]
(3)
The matrix F characterizes how perturbations to the parameters 0 influence the model's predictive distribution. Parameter-space directions with large eigenvalues of F corresponn to high sensitivity, whereas directions with small eigenvalues can be altered with minimal impact on model behavior.
Since computing the full FIM is intractable, particularly when tracked across pretraining epochs, we use its trace as scalar measure of total sensitivity (Achille et al., 2018 Jastrzebski et al. 2021)
Lr(F)= Ez-p(u) Eyyspoyla)) [I po [+]
The trace of F' is the expected squared norm of the score function. In practice, since compute this quantity at every epoch throughout pretraining, we approximate tr(F) using gradients of the self-supervised loss evaluated at observed targets from the pretext task. Despite known differences from the true Fisher (Kunstner et al., 2019), this approximation has been shown to effectively track
trends in learning dynamics and parameter sensitivity (Lewandowski et al. 2023 Li et al. 2025).
Plasticity rises, peaks, and stabilizes during SSL pretraining. Fig. 4 shows Fisher Information (FI) trajectories during SSL pretraining, providing quantitative view of how plasticity evolves over time. Across methods, FI rises early, peaks, and then declines before stabilizing.
SiMCLR-NNSO-MMWW
MICReg-RN50-TMan
DINO-VT-816-MMeW
MME-IT-816-1MoW
Norm. fisher info
1o0k-
1.00
1.05
1.00
0.75-
0.75
0335
0.75
0.25
5:25
025
0.25
0.02
252
300
750
1000
0.00
730
1050
0.00
233
303
1029
0.00
230
550
1000
Pretrain Epach
Protrain Epach
Pretrain Epoch
Protrain Epach
Figure 4: Fisher Information tracked during pretraining.
6

---

Published as a conference paper at ICLR 2026
We define the critical period (CP) the sequence of epochs before FI stabilizes. During this phase, the model is highly plastic and representations can be readily adapted to new information. Once FI stabilizes, the CP closes: the model discards variability uninformative for the pretext task, and representations become less sensitive to change. These dynamics align with the perturbation experiments in Fig. [] Deficits introduced during the early phase had lasting effects representation quality, whereas deficits introduced after the CP produced only minor effects, suggesting that the model had consolidated and become less responsive perturbation. Overall, the rise and decline of Fl may reflect an asymmetry in how SSL pretraining responds to new information over time.
4 CRITICAL PERIODS AS A GUIDE FOR EFFICIENT AND TRANSFERABLE SSL
In the previous section, our analyses revealed that SSL pretraining can exhibit its own critical period (CP) dynamics. Here, we investigate how these dynamics relate to downstream transferability and introduce two simple CP-guided interventions for efficient and transferable SSL.
4.1 ALIGNING CRITICAL PERIODS WITH DOWNSTREAM TRANSFERABILIT
To examine whether critical period (CP) dynamics relate to transferability trade-offs ($2). we align
Fisher Information (FI) trajectories (Fig- with downstream performance (Fig. 2 across pretraining epochs. Fig. shshows a recurring pattern across SSL methods: out-of-domain (OOD) transferability peaks near the point where FI stabilizes (grey shading marks CP closure), then declines even as in-domain (ID) accuracy continues to rise.
SimCLR. 4MoW
VICReg
VIT
MAE
0.02
0.12
0.10
0.10
Performance
0.01
0.56
0.05
0,05
0.00
0.03
0,00
-0.03
250
303
750
100
-0.26
ztc
500
750
1070
25
5J0
7S0
1000
-0.05
257
500
750
Pretrain Epech
Pretrain Epoch
Pretrain Epoch
Pretrain Epoch
(MoWival
IMOW-WILDS (OOD)
EuroSAT (OOD)
EuroSAT Spatial (OOD)
Fisher Info
Figure 5: Relation between Fisher Information (FI) dynamics and downstream transferability. FI trajectories (black) are aligned with downstream performance (colored lines) across checkpoints.
The overspecialization phase. We define the divergence between rising ID and declining OOD performance as the onset of an overspecialization phase. After CP closure, representations continue to specialize on the pretext distribution by discarding variability irrelevant to the pretext task. While this pruning benefits ID performance, it also discards information that may be useful for OOD transfer, leading to a divergence between the two. This suggests that CP closure provides a sweet spol where representations sufficiently learned but not yet overspecialized the pretext task.
Fisher Information (FT) dynamics help explain delayed trade-offs. As noted in L SimCLR's transferability trade-off emerges later than other SSL methods. Fl trajectories suggest that Sim- CLR's critical period (CP) closes later, which may delay overspecialization. This aligns with prior findings that contrastive losses converge slowly (Shah al. 2022 Tong al. 2023). key distinc- tion is that SimCLR (Chen et al., 2020) is the only method among those we study whose objective depends on both positive and negative pairs. Since the gradients of the contrastive loss depend on each sample's position relative to all other negatives, the optimization objective shifts as the minibatch changes (Chen et al., 2002). This requires the model to repeatedly reorganize the global structure of the representation space to keep positives aligned while pushing all other samples apart. We conjecture that SimCLR's delayed CP-closure reflects this repeated global reshaping.
In contrast, VICReg (Bardes et al. 2021). DINO (Carnn al. 2021). and MAE (He et al., 2022) do not rely on negatives. VICReg regulates ber-batch variance and covariance, DINO enforces alignment with a slowly updated teacher, and MAE reconstructs masked parts of single image. These objectives do not require maintaining relationships to all other samples in the batch, unlike SimCLR (Chen al. 2022). which can partly account for the faster CP closure observe.
7

---

Published as a conference paper at ICLR 2026
4.2 CRITICAL PERIOD-GUIDER CHECKPOINT SELECTION (CP-CS)
Selecting the optimal SSL checkpoint is non-trivial, as earlier checkpoints risk underdeveloped rep- resentations while later ones overspecialize to the pretext task. The finding that OOD transferability peaks near the end of the critical period (CP) suggests a practical strategy. Rather than default- ing the conventional final checkpoint, we introduce Critical Period-guided Checkpoint Selection (CP-CS), which leverages Fisher Information (FI) dynamics to identify CP closure checkpoints.
CP-CS requires no additional cost beyond pretraining and no labels. One can (i) monitor FI trace across epochs, (ii) identify the range of epochs where the FI curve stabilizes, and (iii) select the nearest checkpoint for downstream. This rule-of-thumb can narrow the search space: CP closure provides a reasonable starting point when OOD transfer is important, while pretraining beyond CP closure remains beneficial when accuracy is the priority, Additional results are provided in $C
4.3 CRITICAL PERIOD-GUIDED
(CP-SD)
While intermediate checkpoints exhibit stronger out-of-domain (OOD) generalization, later check- points continue to achieve higher in-domain (ID) accuracy. This trade-off reflects complementary properties: CP checkpoints can capture broadly transferable features, while post-CP checkpoints specialize toward pretext-specific signals, increasing alignment with the pretraining distribution.
To mitigate this loss OOD transferability, we present CP- CP-guided self-distillation (CP-SD), a light post-pretraining strategy that reuses existing checkpoints. The idea is simple: use the CP checkpoint as a teacher for the intermediate layers of the post-CP checkpoint (student). During downstream finetuning, optimize the task loss Lusk (e.g., cross-entropy for classification) together with distillation loss applied only to intermediate layers L. The overall objective is
L = Lusk Σ
|1 Issudde
fleacher
(5)
IEC
where À is hyperparameter and the later layers are optimized only with Lusk-
The intuition comes from our layer-wise probing analysis (Fig. 6) CP checkpoints achieve stronger OOD performance across the network, with the gap largest in the early layers. In contrast, post-CP checkpoints provide higher ID accuracy in the later layers, reflecting the benefits of extended pre- training when downstream tasks are aligned with the pretraining distribution. While early layers are widely understood encode general features YYosinski al. 2014). this property can weaken throughout training. As pretraining progresses, the model may increasingly compress represen- tations toward information relevant the task-specific objective (Shwartz-Ziv & Tishby (017). including in the early layers. CPSD addresses this trade-off by restoring the early layers of the final checkpoint toward their CP state to recover OOD transferability, while preserving the later layers of the post-CP checkpoint to maintain the ID strength gained through extended pretraining.
test (OOD)
0.8
0.5
0.4
0.6
Test Acc.
0.3
0.4
0.2
CP Checkpont 350)
CP Checkpomt 350)
Post CP Checkpoink (sp. 1000)
Post CP Cheskpaink (op. 1000)
Stage
Stage
Stage
Stage
Stage
Stage 2
Stage
Stage
Figure 6: Layer-wise probing for OOD transfer (left: VICReg pretrained on fMoW, evaluated on EuroSAT-Spatial and for ID) performance (right: VICReg pretrained and evaluated fMoW).
Results. Tab. [ reports top-1 downstream classification accuracy. The final pretrained checkpoint achieves the strongest ID performance but suffers OOD degradation. The CP-guided checkpoint (CP-CS) shows the opposite pattern: it trades small amount of ID accuracy for large OOD gain. CP-guided distiillaatno (CP-SD) aims to combine both properties: distilling early-layer features from the CP checkpoint into the final checkpoint yields balanced overall performance. Distilling
8

---

Published as a conference paper at ICLR 2026
all layers, however, performs worse than early-layer distillation. We suspect this is because pulling the entire model toward the CP checkpoint restores some generality from the intermediate model but also overwrites the useful ID- ID-specific refinements learned in later layers.
Table 1:
P-M5S0--MWWW.. W. Results are averaged over runs. Bold: best, underline: second best. Distillation settings are provided in $D
Downstream classification results for VICReg- VICReg-RN5O-- 1000--N500-MM--
Model
Maol-val (ID)
[MOW-WILDS (OOD)
EuroSAT (OOD)
EuroSAT-Spatiol (OOD)
Final checkpoint
0.621 0.01
0.341 +: 0034
0.917 20017
0.894 0.028
CP checkpoint
0.610 i 0.025
0.430 0001
0,931 1:0133
0,912 0.022
CP-SD (early layers)
0.617 1 0.018
0,445 0.029
0.944 1:011
0.925 0,019
CP-SD (all layers)
0.611 1 0.019
0,421 1 0.023
0.929 0.00
0.908 0.012
5 DISCUSSION & RELATED WORK
In this work, we studied a simple yet underexplored question: how long should we pretrain self- supervised learning (SSL) models? Contrary the prevailing heuristic that longer pretraining trans- lates to better downstream performance (Chen et al. 2220; He et al. (022), we find that the answer is more nuanced. Specifically, earlier checkpoints achieve stronger of--oann (OOD) transfer than later ones, while the latter improve in-domain (ID) performance. The transferability trade-off across pretraining duration suggests that SSL may undergo a phase transition, akin to critical periods.
Critical early learning phases. Originating in biology, critical periods refer to windows of height- ened plasticity during which neural circuits are particularly sensitive to early experience (Kandel ett al. 2000; Henschh 2004] Knudsen 2004). A similar effect has been reported in artificial neural networks: changes in the early training phase shape the final representation, whereas changes later have limited impact. Input perturbations applied early permanently reduce generalization, while the same perturbations applied later are recoverable (Achille et al.) 2018 Kleinman et al.] 2023 2024]
Altinta al. 2025). Moreover, regularization methods (e.g., weight decay) only have large effects when applied early in training (Golatkar et al., 2019] Liu et al. 2020; Kalra & Barkeshli] 2023).
Conceptually, critical periods mark a transition from high-plasticity stage, where representations are rapidly formed, to consolidation stage, where representations stabilize and task iiseeirtele in-
formation is discarded (Shwartz Ziv & Tishby, 2017 Achille et al., 2018 Zhou et al., 2025).
Exploring critical periods in SSL. Whether SSL exhibits critical periods (CP) similar to supervised learning, and how these phases affect downstream transfer, remains unexplored. Building on recent calls for a temporal perspective on SSL (Simon et al. [023 Reizinger et al. 2025), we investigate the emergence of CP in SSL and their impact on transferability. Our results suggest that SSL pretraining undergoes structured phases: early epochs exhibit high plasticity, while later epochs consolidate the model into patterns dictated by the pretraining setup. Beyond plasticity and consolidation, we identify subsequent phase of overspecializatiinn During overspecialization, OOD generalization declines, suggesting that representations become increasingly bound to pretraining data and pretext task. These dynamics shed light on when representations are broadly transferable, complementing prior work that investigated SSL transferability after full pretraining (Ericsson et al.) 2021a ()).
Several implications follow. SSL is often positioned as pathway task-agnostic representa- tions (Qiang et al.] 2024; Reizinger et al., 2025). This has fueled the rise of foundation models, whose general-purpose representations transfer across tasks (Bommasani] 2021). From this view, SSL pretraining defines a distribution over parameters that serves as a prior for all possible down- stream tasks. This prior only useful to the extent that it supports adaptation, yet SSL is not devoid of specialization. Even without labels, every pretext objective imposes implicit supervisory sig- nals (Balestriero LeCun, 2024 Wang al., 2024b), shaping the invariances and biases the model encodes. Since downstream tasks are unknown at pretraining time, SSL. has no guidance for dis- tinguishing task-relevant from task-irrelevant variation (Kleinman et al. 2021). Thus, models may capture nuisances alongside useful features (Xiao et al. |020 Robinson et al. 2021 Wang et al.) 2022 Bandara et al. 2023] Rabin et al., 2024 Qiang et | and with extended pretraining, the prior may increasingly align with the pretext task, potentially reducing broad transferability. This tension is salient for foundation models, whose utility depends on SSL producing broadly adaptable
priors. We briefly discuss this tension through an information-theoretio lens in :
9

---

Published as a conference paper at ICLR 2026
6 LIMITATIONS & FUTURE WORK
Our analysis centers on unimodal vision models and a subset of SSL methods. Whether similar phase-like behavior occurs in other datasets and domains (e.g,, language, time-series, audio) or in other SSL settings (e.g., joint embedding predictive architectures (Assran al. (023). multimodal SSL ZZng et al. 2024g) remains an open question. Additionally, the quality of the SSL represen- tations is shaped by the interplay between training configurations such augmentation choices in discriminative methods or masking logie in generative methods (Cabannes et al., 2023). Varying
these configurations may uncover further insights into the dynamics of SSL (Reizinger al. 2025
Our experiments also operate within a fixed epoch budget, which may preclude observing phenom- ena such epoch-wise double descent (Nakkiran et al. 2021; Stephenson & Leel 2021 where test performance can worsen and then improve again with prolonged training. This is particularly relevant as SSL methods are known to be rather inefficient learners (Wang et al., 2021 Tong al. 2023), and extending pretraining may reveal additional non mom-oooto dynamics whose shape could vary across downstream settings.
Another promising direction is integrating label-free representation quality metrics such a- ReQ (Agrawal al. 2022). RankMe (Garrido et al.] [023), and LiDAR (Thilak et al.) (023) into critical period analysis. Since single metric fully captures transferability (Agostinelli et al. 2022),
combining complementary metries may provide a richer view: a-ReQ characterizes the eigenspec- trum decay of the feature covariance, RankMe measures its effective rank, and LiDAR estimates linear probing performance by measuring aligmment between learned features and discriminative directions inferred from augmentations. Tracking such metrics during pretraining may offer com- prehensive picture of when structured representations begin to form and how they evolve temporally.
While our exploration is by means exhaustive, our findings show that transferability in SSL can follow a non-monotonic trajectory. Analyses that rely solely on the final pretrained checkpoint therefore risk missing the stages which broad transferability is acquired, altered, or lost. Under- standing these transient phases may be as important as evaluating the converged model, as the final checkpoint not necessarily the most transferable one.
ACKNOWLEDGMENTS
This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-ACOS ACC0-0000222725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains non-exclusive, paid- up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or al- low others to do so, for United States Government purposes. The Department of Energy will provide pub- lic access to these results of federally sponsored research in accordance with the DOE Public Access Plan htttp://eerry.gov/dowwwlee hiloow/ladddee--pbii--acaee ship Computing Facility the Oak Ridge National Laboratory, which supported by the Office of Science of the U.S. Department Energy under Contract No. DE-AC00000RR2277
This research used resources of the Oak Ridge Leader-
The results and models presented in this work used compute resources from the National Al Research Resource Pilot (NAIRR), with support from NVIDIA, including NVIDIA's DGX Cloud product and the NVIDIA AI Enterprise Software Platform.
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. Journal of Machine Learning Research, 19(50) 1-34, 2018.
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep networks. In Inter- national Conference Learning Representations, 2018.
Andrea Agostinelli, Michal Pándy, Jasper Uijlings, Thomas Mensink, and Vittorio Ferrari. How stable are transferability metries evaluations? In European conference computer vision, pp. 303-321. Springer, 2022.
Kumar K Agrawal, Arnab Kumar Mondal, Amaa Ghosh, and Blake Richards. a- ReQ: Assessing representation quality in self-supervised learning by measuring eigenspectrum decay. Advances in Neural Information Processing Systems, 35:17626-17638 2022.
10

---

Published as a conference paper at ICLR 2026
Shaden Alshammari, John Hershey, Axel Feldmann, William T Freeman, and Mark Hamilton. I-con: unify- ing framework for representation learning. arXiv preprint arXiv:2504. 16929, 2025.
Gül Sena Altunta,, Devin Kwok, Colin Raffel, and David Rolnick. The butterfly effect: Neural network training trajectories are highly sensitive to initial conditions. In Forty-second International Conference on Machine Learning, 2025.
Shun-Ichi Amari Natural gradient works efficiently in learning. Neural computation, 10(2):211-276, 1998.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann Lc- Cun, and Nicolas Ballas. Self- supervvse learning from images with coint-embedding predictive archi- tecture. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15619-15629, 2023.
Randall Balestriero and Yann LeCun. The birth of self supervised learning: A supervised theory. In NeurlPS 2024 Workshop: Superviiee Learning-Theon and Practice, 2024.
Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bor- des. Adrien Bardes, Gregoire Mialon, Yuandong Tian, et al. A cookbook of self-supervised learning. arXiv preprint arXiv:2304. /22 2023.
Wele Gedara Chaminda Bandara, Celso M De Melo, and Vishal M Patel. Guarding barlow twins against overfitting with mixed samples. arXiv preprint arXiv:2312.02151, 2023.
Adrien Bardes, Jean Ponce, and Yann LeCun, Vicreg: Varinne-iivarianee-coaainne regularization for self- supervised learning. arXiv prepritt orXiv22055.04900 2021.
Tudor Berariu, Wojeiech Czarnceki, Soham De, Jorg Borschein, Samuel Smith, Razvan Pascanu, and Claudia 2021.
Clopath. A study on the plasticity of neural networks. arXiv preprint
Alice Bizeul, Bemhard Schölkopf. and Carl Allen. A probabilistic model behind self-supervised learning. arXiv preprint arXiv:2402.01 2024.
Rishi Bommasani. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 2021.
Florian Bordes, Randall Balestriero, Quentin Garrido, Adrien Bardes, and Pascal Vincent. Guillotine regular- ization: Why removing layers is needed to improve generalization in self-supervised learning. arXiv preprint 2.XXv:2206./3772 2022.
Florian Bordes, Samuel Lavoic, Randall Balestriero, Nicolas Ballas, and Pascal Vincent. A surprisingly simple technique to control the pretraining bias for better transfer: Expand narrow your representation. arXiv preprint arXiv:2304.0533: 2023.
Vivien Cabannes, Bobak Kiani, Randall Balestriero, Yann LeCun, and Alberto Bietti. The ssi interplay: Aug- mentations, inductive bias, and generalization. In International conference machine learning, pp. 3252- 3298. PMLR, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski. and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings the IEEE/CVF inter- national conference on computer vision, pp. 9650-9660, 2021.
Changyou Chen, Jianyi Zhang, Yi Xu, Liqun Chen, Jiali Duan, Yiran Chen, Son Tran, Belinda Zeng, and Trishul Chilimbi. Why do we need large batchsizes in contrastive learning? gradient-bias perspective. Advances in Neural Information Processing Systems, 35:33860 33875, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1696666 1607. PmLR, 2020.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6172-6180, 2018
Elijah Cole, Xuan Yang, Kimberly Wilber, Oisin Mac Aodha, and Serge Belongie. When does contrastive visual representation learning work? In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 14755-14764, 2022.
Andong Deng, Xingjian Li, Di Hu. Tianyang Wang, Haoyi Xiong, and Cheng Zhong Xu. Towards inade- quately pre-trained models in transfer learning. In Proceedings of the IEFF/CVP International Conference on Computer Vision, pp. 19397-19408, 2023.
11

---

Published as a conference paper at ICLR 2026
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. lece, 2009.
Shibhansh Dohare, J Fernando Hermandez- Qingfeng Lan, Parash Rahman, A Rupam Mahmood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026))7688 774,, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint urXiv:2010.1192: 2020.
Yann Dubois, Stefano Ermon, Tatsunori Hashimoto, and Percy Liang. Improving self-supervised learning by characterizing reprenttton.,, Advances in Neural Information Processing Systems, 35:1 1279- 11296, 2022.
Yann Dubois, Tatsunori Hashimoto, and Percy Liang. Evaluating supf-supe learning via risk decomposi- tion. In International Conference Machine Learning, pp. 8779-8820. PMLR, 2023.
Linus Eriesson, Henry Gouk, and Timothy M Hospedales. Why supervied models transfer? investi- gating the impact of invariance on downstream tasks. arXiv preprint arXiv:2111.1133 2021a.
Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5414-5423, 2021b.
Ronald Aylmer Fisher. Theory of statistical estimation. In Mathematical proceedings of the Cambridge philo- sophical sociery, pp. 700- 725. Cambridge University Press, 1925.
Quentin Garrido, Randall Balestriero. Laurent Najman, and Yann Lecun. Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank. In International conference on machine learning. pp. 10929-10974 PMLR, 2023.
Aditya Sharad Golatkar, Alessandro Achille, and Stefano Soatto. Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little convergence. Advances in Neural Information Processing Systems, 32, 2019.
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and henchmarking self-supervised visual representation learning. In Proceedings of the ieee/cvf International Conference on computer vision, pp. 6391-6400, 2019.
Jie Gui, Tuo Chen, Jing Zhang, Qiong Cao, Zhenan Sun, Hao Luo, and Dacheng Tao. A survey self- supervised learning: Algorithms, applications, and future trends. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):9052-9071 2024.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Xinlci Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16000-16009, 2022.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Takao K Hensch. Critical period regulation. Annu. Rev. Neurosci, 27(1):549-579. 2004.
Dapeng Hu, Shipeng Yan, Qizhengqiu Lu, Lanqing Hong, Hailin Hu, Yifan Zhang, Zhenguo Li, Xinchao Wang. and Jiashi Feng. How well does self-supervised pre-training perform with streaming data? arXiv preprint arXiv:2104.1 120001, 2021.
Tao Huang, Yanxiang Ma, Shan You, and Chang Xu. Learning mask invariant mutual information for masked image modeling. arXiv preprint arXiv:2502. 19718, 2025.
Stanislaw Jastrzebski, Devansh Arpit, Oliver Astrand, Giancarlo B Kerg. Huan Wang. Caiming Xiong, Richard Socher, Kyunghyun Cho, and Krzysztof J Geras. Catastrophic fisher explosion: Early phase fisher matrix impacts generalization. In International Conference on Machine Learning, pp. 4772-4784. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2026
Dayal Singh Kalra and Maissam Barkeshli. Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and width. Advances Neural Information Processing Systems, 36:51621- 51662, 2023.
Eric R Kandel, James H Schwartz, Thomas M Jessell, Steven Siegelbaum, A James Hudspeth, Sarah Mack, et al. Principles of neural science, volume McGraw-hill New York, 2000.
Youngeun Kim, Yuhang Li, Hyoungseob Park, Yeshwanth Venkatesha, Anna Hambitzer, and Priyadarshini Panda. Exploring temporal information dynamics in spiking neural networks. In Proceedings of the AAAI conference on artificial intelligence, pp. 8308-831 10868 2023.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka BraaiskaBBrrwinsaa, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114((3):351-33522 2017.
Michael Kleinman, Alessandro Achille, Daksh Idnani, and Jonathan Kao. Usable information and evolution of optimal representations during training. In International Conference on Learning Representations, 2021.
Michael Kleimman, Alessandro Achille, and Stefano Soatto. Critical learning periods for multisensory inte- gration in deep networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14296-24305 2023.
Michael Kleinman, Alessandro Achille. and Stefano Soatto. Critical learning periods emerge even in deep linear networks. In The Twelfth International Conference on Learning Representations, 2024.
Konstantin Klemmer, Esther Rolf, Caleb Robinson, Lester Mackey, and Mare Rußwurm. Satelip: Global, general-purpose location embeddings with satellite imagery. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 4347-4355 2025.
Eric Knudsen. Sensitive periods in the development of the brain and behavior. Journal of cognitive neuro- science, 16(8):1412-1425, 2004.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani,
Wcihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the- wild distribution shifts. In International conference machine learning, pp. 5637-5664. PMLR, 2021.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arx: 10054, 2022.
Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations the empirical fisher approximation for natural gradient descent. Advances in neural information processing systems, 32, 2019.
Alex Lewandowski, Haruto Tanaka, Dale Schuurmans, and Marlos C Machado. Directions of curvature an explanation loss plasticity, arXiv preprint arXiv:2312.002466 2023.
YuXin Li, Felix Dangel, Derek Tam, and Colin Raffel. Fishers for free? approximating the fisher information matrix by recycling the squared gradient accumulator. arXiv preprint arXiv:2507. 18807, 2025.
Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Samandez-Grradee Early-learnins regular- ization prevents memorization of noisy labels. Advances in neural information processing systems, 33: 20331-20342, 2020.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian. Zhaoyu Wang, Jing Zhang, and Jie Tang. Self-supervised learning: Generative contrastive. IEEE transactions on knowledge and data engineering, 35(1)8557887) 2021.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Under- standing plasticity in neural networks. In International Conference Machine Learning, pp. 18319-233211 PMLR, 2023.
Markus Marks, Manuel Knott, Neehar Kondapaneni, Elijah Cole, Thijs Defraeye, Fernando Perez-Cruz. and Pietro Perona. A closer look at benchmarking self-supervised pre-training with image classification. Inter- national Journal of Computer Vision, pp. 1-13, 2025.
James Martens. New insights and perspectives the natural gradient method. Journal of Machine Learning Research, 21(146))1-76, 2020.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang Boaz Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003, 2021.
13

---

Published as a conference paper at ICLR 2026
Alejandro Newell and Jia Deng. How useful is self-supervised pretraining for visual tasks? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7345-7354, 2020.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernan- dez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.077 2023.
Zhuo Ouyang, Kaiwen Hu, Qi Zhang. Yifei Wang, and Yisen Wang. Projection head is secretly an information bottlemeck. arXiv preprint arXiv:2503.00500 2025.
Utku Ozbulak, Hyun Jung Lee, Beril Boga, Esla Timothy Anzaku, Homin Park, Arnout Van Messem, Wesley De Neve, and Joris Vankerschaver. Know your self-supervised learning: A survey on image-basee generative and discriminative training, arXiv preprint 2XXv:3305.13665 2023.
Wenwen Qiang, Jingyan Wang, Changwen Zheng, Hui Xiong, and Gang Hua. On the universality of self- supervised learning. arXiv preprint arXiv22405.01053, 2024.
Wenwen Qiang, Jingyao Wang, Zeen Song, Jiangmeng Li, and Changwen Zheng. On out-of-distribation generalization of self-supervised learning. arXiv preprint arXiv:2505. 16675, 2025.
Zachary Rabin, Jim Davis, Benjamin Lewis, and Matthew Scherreik. Overfitting in contrastive learning? arXiv preprint arXiv:2407. 15863, 2024.
Patrik Reizinger, Randall Balestriero, David Klindt, Wieland Brendel. Position: An empirically grounded identifiability theory will accelerate self supervised learning research. In Forty-second International Con- ference on Machine Learning Position Paper Truck, 2025.
Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? Advances in neural information processing systems,
2021.
Esther Rolf. Konstantin Klemmer, Caleb Robinson, and Hannah Kerner. Position: Mission critical-satellite data is distinct modality in machine leamning In Forty-first International Conference on Machine Learning, 2024.
Nikunj Saunshi, Orestis Plevrakis. Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoret- ical analysis of contrastive unsupervised representation learning. In International conference on machine learning, pp. 5628-5637. PMLR, 2019.
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12): 2019.
Anshul Shah, Suvrit Sra, Rama Chellappa, and Anoop Cherian. Max-margin contrastive learning. In Proceed- ings of the AAAI Conference on Artificial Intelligence, pp. 8220-8230, 2022.
Ravid Shwartz-Tiv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint 1003.0000000//// 10, 2017."
Ravid Shwartz-Ziv, Micah Goldblum, Hossein Souri, Sanyam Kapoor, Chen Zhu, Yann LeCun, and Andrew G Wilson. Pre-train your loss: Easy bayesian transfer learning with informative priors. Advances in Neural Information Processing Systems, 35:27706-27715,, 2022.
James Simon, Maksis Kautins, Liu Ziyin, Daniel Geisz, Abraham Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning. In International Conference on Machine Learning, pp. 31852- 31876. PMLR, 2023.
Oscar Skean, Md Rifat Aretin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz Ziv. Layer by layer: Uncovering hidden representations in language models. arXiv preprint arXiv:2502.02011 2025.
Stefano Soatto and Alessandro Chiuso. Visual representations: Defining properties and deep approximations. arXiv preprint 4rXiv:1411.7674 2014.
Jacob Mitchell Springer, Sachin Goyal, Kaiyue Wen, Tanishq Kumar, Xiang Yue, Sadhika Malladi, Graham Neubig, and Aditi Raghunathan. Overtrained language models are harder to fine-tune. arXiv preprint arXiv:2503. 2025.
Cory Stephenson and Tyler Lee. When and how epochwise double descent happens. arXiv preprint 108.22008/200 2021.
14

---

Published as a conference paper at ICLR 2026
Adam J Stewart, Caleb Robinson, Isaac A Corley, Anthony Ortiz, Juan M Lavista Ferres, and Arindam Baner- joc. Torchgoo: deep learning with geospatial data. Proceedings of the 30th international conference on advances in geographic information systems. pp. 1 1-12, 2022.
Damien Teney, Yong Lin, Seong Joon Oh, and Ehsan Abbasnejad. Id and ood performance are sometimes inversely correlated on real-world datasets, Advances in Neural Information Processing Systems, 36:71703 71722, 2023.
Vimal Thilak, Chen Huang, Omid Saremi, Laurent Dinh, Hanlin Goh, Preetum Nakkiran, Joshua M Susskind, and Etai Littwin. Lidar: Sensing linear probing performance in joint embedding ssi architectures. arXiv preprint arXiv:2312.0400 2023.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learnin?? Advances in neural information processing systems, 33:6827-668 -839, 2020.
Naftali Tishby, Fernando C Percira, and William Bialek. The information bottleneck method. arXiv preprint physies/0004055, 2000.
Shengbang Tong, Yubei Chen, Yi Ma, and Yann Lecun. Emp-ssl: Towards self-supervised learning in training epoch. arXiv preprint urXiv:2304.09977 2023.
Gido M van de Ven. On the computation of the fisher information in continual lcarning. arXiv preprint arXiv:2502.11755 2025.
Bokun Wang, Yunwen Lei. Yiming Ying. and Tianbao Yang. On discriminative probabilistic modeling for self-supervised representation learning. arXiv preprint arXiv:2410.09156, 2024a.
Guangrun Wang, Keze Wang. Guangcong Wang, Philip HS Torr, and Liang Lin. Solving inefficiency of self- supervised representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9505 9515, 2021.
Haoqing Wang, Xun Guo, Zhi-Hong Deng, and Yan Lu. Rethinking minimal sufficient representation in con- trastive learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 16041-16050, 2022.
Wenhao Wang, Muhammad Ahmad Kaleem, Adam Dziedzic, Michael Backes, Nicolas Papernot, and Franziska Boenisch, Memorization in self-supervised learning improves downstream generalization. arXiv preprint arXiv:2401. 2024b.
Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell, Characterizing and avoiding negative transfer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11293-1130 2019.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey zoo. In 2010 IEEE computer society conference on computer vision and puttern recognition, pp. 3485-3492. IEEE, 2010.
Tete Xiao, Xiaolong Wang, Alexci A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. arXiv preprint arxiv:2008.0 05659, 2020.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable features in deep neural networks? Advances in neural information processing systems, 27, 2014.
Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? arxiv preprint arXiv:2006.c 06606, 2020.
Zhanpeng Zhou, Yongyi Yang, Mahito Sugiyama, and Junchi Yan. New evidence of the two-phase learning dynamics of neural networks. In The Thirteenth International Conference on Learning Representations, 2025.
Yongshuo Zong, Oisin Mac Aodha, and Timothy M Hospedales. Self-supervised multimodal learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(7):5299-5318, 2024.
15

---

Published as a conference paper at ICLR 2026
A EXPERIMENTAL DETAILS ($2.1)
All experiments were run on NAIRR using NVIDIA A A100-SXM4-80GBB GPUs with mixed-precision training.
A.1 PRETRAINING SETUP
Hyperparameters. For SimCLR (Chen al. 2020 we a batch size of 1024 with SGD, momentum with 10-epoch linear warmup followed by cosine learning rate decay. The contrastive NT-Xeet loss uses temperature f = 0.1 with negatives gathered across all GPUs. The projection head is a a.2-laver MLP (2048 - 2048 + 128) with batch normalization and ReLU after the hidden layer. For VICReg (Bardes et al. 2021 we LARS with momentum 0.9. weight decay 1x10 and base learning rate 0.3 with linear scaling, with 10-epoch linear warmup followed by cosine learning decay. The loss uses invariance, variance, and covariance weights (A, f4, v) = (25, 25, 1). The projection head is a 2-layer MLP (2048 + 8192 + 8192) with batch normalization and ReLU after the hidden layer. For DINO (Caron et al. 2021 we use Adam W with base learning rate 5x 10 and linear scaling, with 10-epoch linear warmup followed by cosine learning rale decay. Weight decay is cosine-scheduled from 0.04 to 0.4. The student temperature is T. 0.1 and the teacher temperature is fixed at T. 0.04. Teacher momentum increases from 0.996 to 1.0 following a cosine schedule, with center momentum 0.9. The student backbone uses stochastic depth with drop path rate 0.1. The projection head is a 3-layer MLP (768 + 2048 + 2048 - 256) with GELU activations, followed by lu normalization and weight-normalized linear layer projecting 65,536 dimensions. The last layer of the student head is frozen for the first epoch. For MAE (He et al. 2022). we use AdamW with betas (0.9, 0.95), base learning rate 1.5 x 10 with linear scaling, and weight decay 0.05, with a 40-epoch linear warmup followed by cosine learning rate decay. The masking ratio is 0.75 and the reconstruction target is raw pixel values with mean squared error loss. The decoder has 8 Transformer blocks with embedding dimension 512, 16 attention heads, and MLP ratio 4.0.
0.9, hase learning rate 0.3, and weight decay 1
2020), we apply random resized crop x21 pixels (scale 0.08-1.0), random horizontal flip (p = 0.5), color jitter (brightness, contrast, saturation = 0.8, hue = 0.2; applicd with p = 0.8), random grayscale (p = 0.2), and Gaussian blur (a E 2.01, p = 0.5). For VI- CReg (Bardes et al. 2021), we use the same pipeline as SimCLR but with reduced color jätter strength (bright- 0.2, hue 0.1: applied with p 0.8) and random solarization 0.1). For DINO Caron al.] 202I). we use multi-crop with two global views (224 224 pixels, scale 0.4-1.0) and eight local views (96 96 pixels, scale 0.05-0.4). All views use random horizontal flip (p=0.5), color jitter (brightness = 0.4, contrast = 0.4, saturation = 0.2, huc = 0.1; applied with = 0.8), random grayscale (p 002)), and Gaussian blur (a E 10.1,2.01 p00:p=1.. 1.0 for the first global view, 000 for the second, 0.5 for local views). Solarization (p= 0.2) applied only the second global view. For MAE (He all we apply random resized crop to 224 x 224 pixels (scale 0.2-1.0) and random horizontal flip 0.5).
Augmentations. For SimCLR (Chen
ness 0.4. contrast
0.4, saturation
2022..
A.2 DOWNSTREAM SETUP
Datasets. We follow the definition of in-domain (ID) and out-of-domain (OOD) transfer from Marks et al. (2025). For models pretrained on IMoW-RGB, ID transfer is measured on the held-out IMoW-RGB validation split. We consider three OOD datasets. IMOW-WILDS (Koh et al. (02T) introduces temporal and regional distribution shifts over the same categories. AT (eelber et al. 2019) contains 27, 7,000 Sentinel-2 satellite images across 10 land-use classes. We use the RGB version 64x64 pixels, which differs from MoW-RGB sensor, spatial resolution, and scene semantics. EuroSAT- ApSttaiill (Stewart et all 2022) uses the same EuroS AT images but splits train, validation, and test sets by longitude (60/20/20) to induce a spatial distribution shift. For models pretrained on ImageNet, ID transfer is measured on the held-out ImageNet validation split. We consider three OOD datasets with increasing domain shift: SUN397 (Xiao et al. 2010) for scene recognition within the natural image domain, FuroSAT-Spatii (Stewart et al. 2022) for remote sensing transfer, and Camelyon WILDS (Koh et al. 2021) for histopathological tumor detection.
Evaluation. We follow standard evaluation protocols in SSL (Balestriero et al. 2023 All models are evaluated using top-1 accuracy via finetuning. We use official splits when available and adopt standard 80/20 split otherwise. For ResNet hackbones (SimCLR, VICReg), we finetune for 50 epochs using SGD with momentum 0.9 and base learning rate with linear scaling. For VIT backbones (DINO, MAE), we finetune for 50 epochs using AdamW with betas (0.9, 0.999) and base learning rate 1 x 10 -3 with linear scaling.
16

---

Published as a conference paper at ICLR 2026
B ADDITIONAL: IMAGENET RESULTS
SimCLR
VICReg RNSD. ImageNet
DINO NIT--16-lmin B16-ImageNet
MAE-VIT VIT Imag
0.12-
0.06
0.05
0.7
Performance
0.05
0.05
0.2
0.00
0.00
0.0
0.00
250
500
750
1003
0.03
1000
0.05
250
500
1:00
250
510
750
1000
Checipoint Sequence
Check sncitpnnttnnnm Sequence
Cherkpaint Sequuence
Checkpoint Sequence
ImageNeet-val (I))
SUN397 (OOD)
Camelyon17-WILDS (OOD)
EuroSAT-Spatial (OOD)
Figure 7: Transferability trade-off in SSL. The x-axis denotes pretraining epoch (checkpoints saved every 50 epochs), and the y-axis shows the change in performance relative the final checkpoint.
IN
VICReg RN50-IN
MAE V6
Sensitvitt
1.00
0.75
0.75
0.73
0.75
0:30
0.5
0.50
0.50
(0rdet
10roet
(Onset 7501
(0 01
(0160:4450) 450)
[0nset 750)
[Onset
(Onset 4501
(Onset 750)
(Onget
[0nset 450)
(Drcet 750)
Deficit duration tion
epochs
epochs
epochs
Figure 8: Sensitivity to input perturbations during pretraining reflected in downstream performance.
SimCLR-RN5O-IN
VICReg-RN50-IN
DINO-VIT-816-N
MAE-VIT-816-N
Nerm. fisher Info
3.001
1.001
1.000
0.59
0.50
0.5
0.50
0.25
0.25
025
0.25
1050
253
Epach
Epoch
Spoch
Epoch
Figure 9: Fisher Information tracked across pretraining epochs.
SimILL-RRND-IN
VICRag AN50-NN
DINO-VIT-B1-IN
MAE-IT-B16-N
à Performance
0.02
0005
Pretrain Epoch
Pretrain Epoch
Pretrain Epoch
Pretrain Epoch
ImageNeet-val (ID)
SUN397 (OOD)
Gamelyon17-WLDD (OOD)
EuroSAT-Spatial (OOD)
Fisher Info
Figure 10: Relation between Fisher Information (FI) dynamics and downstream transferability. FI trajectories (black) are aligned with downstream performance (colored lines) across checkpoints.
17

---

Published as a conference paper at ICLR 2026
C ADDITIONAL: CP-GUIDED CHECKPOINT SELECTION (CP-CS)
Additional SSL. method. We pretrain DINOv2 (Qquah all 2023 with EMA teacher and student trained using AdamW with cosine warmup schedule. Weight decay annealed from 0.04 to with cosine schedule. The loss combines an image-level DINO objective (class tokens), a patch-level iBOT objective (masked tokens, applied only on the student). and KoLeo regularizer (A = 0.1). Teacher momentum is scheduled from 0.992 to 1.0. We
ViT-S/16 backbone with DropPath 0.1, LayerScale 10 and standard DINO multi-crop augmentations. For evaluation, we follow A.2
Results. Fig. [ shows the Fisher Information (FI) dynamics during DINOv2 pretraining their relation to downstream transfer. Checkpoints around FI stabilization coincide with peak OOD performance.
0.08
A Performance
0.03
-0.03
-0.08
250
500
750
1000
Pretraining Epoch
vMl (ID)
MoWWWILDS. (OOD)
EuroSAT (OOD)
EuroSAT-Spatial (OOD)
Fisher Info
Figure 11: Relation between Fisher dynamics and transferability for DINOV2-ViT-SIG-INE
D ADDITIONAL: CP-GUIDED SELF-DISTILLATION (CP-SD)
Distillation setting. For distillation, first three Transformer blocks of ViT-B16 (aligning [CLS] tokens). Models are trained with Adam W
the first residual stage (layerl) of ResNet-50 (He al. 2016
for 100 epochs (batch 256, base LR 10 -4 with cosine schedule, weight decay 0.05), using À = 0.5 (Eq.
MMW
VICReg-RN5O Baiisg--550--M00
DINO-VIT-116PP fMoW
MMoW
Dowretream Performnce
2.00-
1.00
1.00
1.00
2.75
0.72
0.75
2.25
3.50
0.00
0.02
0.50
(500)
(982)
Rnal Checkpoitt
Selt Desiliation
Figure 12: We compare the final checkpoint, the CP checkpoint, and our CP self-distilled (CP-SD) checkpoint. CP-SD) consistently achieves the highest OOD performance while recovering ID).
A Performance (A=0.5)
0.0
-0.5
-10
B4cW-val (IB)
WILWWILDS (COD)
-15
EurcSAT
Spatia (60D)
Distillation weight
Figure 13: We use VICReg-RN50-IMoN and distill from the conv2 x block group (He et al. 2016 Performance is overall stable, though larger values (A 1.0) show diminishing returns.
18

---

Published as a conference paper at ICLR 2026
E AN INFORMATION-THEORRET LENS ON SSL SPECIALIZATION
Background. The Information Rottleneck (IB) (Tishby al. 2000) provides framework for characterizing optimal representations. Given an input X and a task variable the IB characterizes the representation Z that optimally balances compression of X against retention of information about Y:
min p(s])
(2: 3 (1(ZYY),
(6)
where controls the trade-off between the two terms. Prior work on the information dynamics deep net- works (Shwartz-Ziv & Tishby 2017) shows that supervised training can proceed in two phases: a fitting phase in which I(ZY)) increases rapidly, followed by a compression phase in which (Xx; 2) decreases, though the generality of this pattern depends architecture and estimation choices (Saxe al. 2019).
Although supe-suue learning (SSL) does explicitly optimize this IB objective, the IB perspective offers an intuition that may help explain the learning dynamics we observe. Our gool here not to claim a formal equivalence, but to use the IB framework as preliminary interpretive lens.
In SSL, downstream labels are unavailable during pretraining, Instead, each method defines a pretext signal Y Ypretess. such the positive-paie pair structure in contrastive learning (Chen al. 2020), teacher predictions in self- distillation (Caron al. 202T), masked regions reconstruct (He 2022). Under the IB perspective, optimization encourages the representation to increase TZ: Y Ypreee)) while discarding input variability that does not help predict Y Ypeses, reducing T(X; Z).
As optimization progresses, the representation specializes toward what Ypreteur treats as relevant. Since the pretext signal is derived from the data itself, this relevance inherits the statistics of the pretraining distribution. Features irrelevant Y Ypreue may be discarded even when they would he broadly useful, while distribution- specific correlations that happen to be predictive Yrrees may be retained even when they do not generalize.
In discriminative methods, Y Ypreeu encodes which augmented views originate from the same input. Recent unifying work (Alshammari et al. 2025) shows that SimCLR (Chen et al. 2020) and VICReg (Bardes et al. 2021) minimize the same divergence between supervisory distribution over augmented views and learned distribution over embeddings, and DINO (Caron et al. 2021) applies similar pressure through teacher-student alignment. Since the augmentations are applied to the pretraining distribution, the invariances the encoder learns are shaped by the statistics of that data. The pretext objective has no mechanism to distinguish between spurious and generalizable correlations: both are encoded so long they satisfy the view-matching criterion.
In generative methods, Ypetes defined by masked-region reconstruction. The decoder reconstructs individ- ual patches from nearby tokens using pixel-level loss, the encoder is primarily rewarded features that support local spatial prediction. Recent work formalizing MAE through the IB (Huang et al. 2025) provides further evidence: the reconstruction objective can produce suboptimal compression trade-off. retaining more low-level detail than necessary while failing to preserve cross-view information that proves useful downstream.
Takeaway. The pretext task and the pretraining distribution jointly determine what counts relevant infor- mation SSL. When downstream distribution diverges sufficiently, features shaped by this definition can impair performance (Wang et al. [2019] Teney et al. [0233 Deng et al. (003). This does not imply that longer pretraining is strictly harmful: when pretext and downstream objectives are aligned, additional training may continue to improve useful features. Overall, extended pretraining may increasingly specialize the representa- tion toward prefext-defined relevance, regardless of whether those features are transferable.
19

---

Published as a conference paper at ICLR 2026
F ABLATION STUDY
Performance
Performance
Perfomance
0.02
0.02
-0.02
0.00
-0.05
250
360
370
380
390
400
-0.55
750
1000
-0.05
250
Checkpoint Sequence
Pretraining Epoch
Pretraining Epoch
(a) Finer-resolution (10 epochs)
(b) Pretrained IMoW
(c) Pretrained on ImageNet
Figure 14: (a) Evaluating every 10 epochs within the CP window for VICReg R-50--555:: OOD performance (other than pink) peaks again nearby. (b, e) Linear probing across pretraining check- points: intermediate checkpoints outperform the final checkpoint OOD tasks (other than pink), while ID performance (pink) tends to improve.
1.00
Norm Senuibiity
0.75
1.00
0.50
Nerm, lisher Info
Norm. Fisher Info
1.0
VIT-Blas
0.7
0.82
Hdd
0.25
Delinit duration
0.00
25
50
75
100
(canine
esechs (ffand
Mixdd Mteed
Protraining Epoch
0.0
100
200
300
icosine
Datareuse
Data-reuse 5
Pretraining Epoch
(a) Fixed vs. Cosine L.R
(b) Minibatch reuse
(c) VIT-S VIT-B
Figure 15: (a) Under both fixed and cosine LR schedules, early deficits cause the largest degradation, suggesting the temporal sensitivity is necessarily artifact of the LR schedule. (b) Increasing minibatch reuse from k=1 to k=5 causes FI to peak earlier, consistent with reduced gradient di- versity accelerating CP closure. (c) ViT-Base shows slower FI stabilization than ViT-Small under DINO pretraining, suggesting larger capacity can delay CP closure.
2e 23
0.03
habee informattonn mommmtton
Loc: Encoder
1e:03
20.22
Reo
Be-04
-.07
atte
atd
ans Aton
$004000
$6,00
Stuet
5o Jopp
105 305
400
500
100 203 450
Chec Spant Para
Checkpoint Parr
Pretrainig Esoch
Checkpeint Sequence
(a) Consecutive rep, drift (b) Rep. drift from ep.
(c) SatCLIP Fisher
(d) SaICLIP downstream
Figure 16: (a) Representation drift for VICReg-RN5O-N consecutive checkpoint drift is large carly and decreases, (b) while drift relative to epoch 50 rises then plateaus, suggesting early pre- training exhibits greater representational plasticity. (c) Extending CP experiments to multimodal SatCLIP (Klemmer et al. 2025): Fl peaks early and stabilizes for both encoders during pretraining, (d) and the intermediate checkpoint outperforms the final checkpoint across geospatial tasks.
20


Reviews:
[{"review_id": "lywj8SYtuP", "reviewer": "Reviewer_U7ZN", "summary": "This paper aims to answer the following question: \"how long should SSL models be pretrained?\". They relate this problem to the notion of critical periods: where models exhibit high plasticity in early training stages, and go through a consolidation phase where OOD adaptability declines but ID performance improves. This phenomenon has been observed in supervised learning and this work claims to be the first systematic investigation of critical periods in SSL. They make three main contributions:\n- how to do CP analyses for SSL without requiring labels\n- how CP closure can guide the process of selecting intermediate checkpoints that show stronger OOD robustness\n- propose a distillation technique that uses sweet spot CP checkpoints as teacher and overspecialized networks as student, which leads to improved OOD generalization while maintaining ID performance.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. **Reformulation of CP for SSL**: This seems to be the main novelty of this work. To show critical periods in SSL, the authors have proposed two analyses techniques: (a) perturbations at different learning stages (b) FI matrix with respect to pretext tasks.\n2. **Practical Impact**: Contributions in Section 4 (CPCS and CPSD) can potentially have good practical impact providing guidance to select checkpoints based on transferability tradeoff, and improving OOD robustness. \n3. **Experiments**: This work is backed well with strong experiments. They have considered two real-world datasets (IM-1K and fMoW-rgb) and four methods (SimCLR, VICReg, MAE, DINO). There're some DINOv2 results as well but I am not sure why it's only mentioned in Appendix.\n4. **Reproducibility**: Hyperparameters and dataset details are well documented, and the methodology seems reproducible given the provided information.\n5. **Quality & Clarity**: Overall, the paper is well-written and has a logical flow. The schematic in Figure 1 is particularly effective in summarizing the conceptual framework, which is later supported by quantitative results.", "weaknesses": "1. **Lack of Mathematical Rigor / Theoretical Depth**: The study is primarily empirical. While the FI metric offers some analytical grounding, the paper does not provide a principled explanation for why SSL exhibits critical periods or overspecialization. Some lightweight theoretical reasoning could strengthen the argument. \n\n2. **Issues with CP-guided self-distillation**:\n\n    (a) L117: it is not explained why the authors chose to distill early layers only. It's only later in Section 4.3 where the rationale behind this choice is addressed. \n\n    (b) Figure 6 is not convincing enough to justify distilling early layers. I think it will be useful to verify their claims by distilling the entire network and comparing performance gains.\n\n    (c) **Missing Ablations**: The number of layers distilled (L) and the distillation weight (λ) are unspecified and unexplored. Both likely affect results and reproducibility.\n\n3. **Figure 6**: It is not clear what the authors imply by \"stage\". \n4. **SimCLR's behavior**: I appreciate the attempt to explain why SimCLR's critical period closes much later (L375-376). I think this section needs more clarification. What are the differences in objectives of all the methods taken into consideration that leads to this behavior?\n5. **Definition of CP closure**: The paper briefly mentions detecting CP closure “L388- when the FI slope stabilizes (e.g., below a tolerance for p consecutive epochs)” (in Sec. 4.2). However, it is unclear whether this rule was actually used to determine the CP checkpoints reported in the experiments. For reproducibility and clarity, it would be valuable for the authors to specify the exact tolerance threshold and window length used in practice (if any).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Avcaj4oIag", "reviewer": "Reviewer_12fS", "summary": "This paper studies the training dynamics of self-supervised learning methods and how performance evolves both in domain and out of domain during training. To understand why OOD performance drops even though ID performance increases during training the authors people to study this behavior through the lens of critical periods. But using two different criteria, the authors are effectively able to detect when overspecialization starts to happen which helps select the best model for OOD performance, or a more balanced model.\nFinally, the authors motivate and experiment with a distillation technique to help with overspecialization.", "questions": "1) For Probe 1, how much do you think that the sensitivity is correlated with a lower learning rate later in training ?\n2) Lines 254-255: when using noise during the deficit window, are data augmentation (jitter,crop,masking etc) still applied ? If not, how are the input image pairs constructed ?\n3) Figure 4: All methods except VICReg have their Fisher Information drop to zero, do you have any intuition why it does not for VICReg ?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The authors provide clear evidence of the studied problem, notably the drop in OOD performance during longer training\n\n- The proposed metrics, in particular the one based on Fisher Information, correlate really well with this behavior. This leads to an effective method to perform early-stopping\n\n- Experiments are performed at a good scale (R50/ViT-B, trained up to ImageNet for 1000 epochs) which adds to the relevance of the results\n\n- The proposed distillation method is useful and very relevant to current SSL research. A similar problem was shown to be present in DINOv3[1], which concurrently proposed another solution based on earlier checkpoint distillation.\n\n[1] Siméoni, Oriane, et al. \"Dinov3.\" arXiv preprint arXiv:2508.10104 (2025).", "weaknesses": "1) Focus on Satellite dataset in the main paper. The same results as in the main paper are performed on ImageNet in the appendix but should be emphasised more in the main paper to appeal to a broader audience.\n\n2) Throughout the paper, the considered evaluation is finetuning. However, the considered methods are more commonly used with lighter evaluations such as training a linear classifier. This would help make the results more relevant and may shed different insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kEPB7OcIuo", "reviewer": "Reviewer_HAzr", "summary": "This paper studies the question of how long to train a visual self-supervised learning model, by looking at the critical periods happening throughout training. The paper presents metrics that allow to identify these periods, and studies how they correlate to downstream tasks. Finally, from these metrics the paper describes methods to reconcile optimal in-domain and out-of-domain performance on downstream tasks. Experiments are conducted on the ImageNet-1K and fMoW-RGB satellite imagery datasets, and several popular SSL methods are studied (SimCLR, VICReg, DINO and MAE).", "questions": "Is there a link between critical periods and double descent ?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper highlights a very interesting phenomenon in SSL, the fact that there exists learning phases that correlate with different levels of performance in out-of-domain or in-domain tasks.  The paper calls this phenomenon, which is the counterpart of overfitting in supervised learning, but for self-supervised learning: “critical periods”. The paper clearly explains how they found these critical periods with precise metrics, how these metrics correlate to downstream tasks and how to leverage these metrics to derive SSL models with better generalization capabilities.\n\n- There is existing large-scale evidence of this phenomenon in the SSL literature. Most SSL practitioners have already encountered these critical phases without putting a name on it.\n\n- The paper is well-written, easy to follow and with good presentation, the message and finding is simple and presented with clear experiments.", "weaknesses": "- The paper focuses on two datasets: ImageNet, which make sense as a general pretraining datasets to learn visual representation, and fMoW-RGB, a satellite imagery datasets, so not generalist. But the use fMoW-RGB is not well motivated and does not help make the experiment convincing. Is the motivation to clearly identify what is ID and what is OOD ? I would appreciate more a focus on ImageNet data or even larger scale generalist data, with the objective to see if these critical periods are also observed in real-world or large-scale scenarios. There is some evidence in the literature that this is the case, in DINOv3, they use gram anchoring at the end of their pretraining to retrieve pic segmentation performance that the model has towards the beginning of training, this is very similar to what the authors describe line 78 “Intermediate checkpoints often achieve better out-of-domain (OOD) transfer than later checkpoints”.\n\n- The paper does not study critical phases with the prism of data overfitting, or doing too many passes on the same data. What would happen in an infinite data regime ? Would we still observe the same behaviour ? It would be great to have more experiments controlling the data distribution and studying the impact on these critical periods.\n\n- Other than the impact of data, the paper also misses the fact that a schedule is used on the learning rate and weight decay. Both are brought to 0 progressively throughout the training and that could be correlated with the critical period observed. I think some experiments with a fixed schedule should be conducted to remove this confounding factor.\n\n- Other similar metrics that have the same objective of characterizing SSL features are not acknowledged. For example, RankMe or LiDAR.\n\n- The introduction should say a little more about the experimental setup, in terms of dataset used and concrete results obtained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s6kLgnCX3w", "reviewer": "Reviewer_2iUr", "summary": "The paper studies how self-supervised learners evolve over training and argues that there exist “critical periods” (CP) during which representations are highly plastic and best for transfer. The authors (i) document a transferability trade-off: longer pretraining continues to improve in-domain accuracy while hurting out-of-domain generalization, (ii) operationalize CP analysis for SSL via two tools, and (iii) propose two practical mechanisms: CP-guided checkpoint selection and CP-guided self-distillation that transfers intermediate “sweet-spot” layer features into the final model to balance ID and OOD performance.", "questions": "1. Why do different SSL methods in Section 2.2 exhibit distinct trends, and why do models such as SimCLR not show a clear “critical period” phase?\n2. How is Fisher Information computed in practice? Do you use full, block-diagonal, or diagonal approximations, per layer or per parameter group, and how frequently is it recomputed during pretraining?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of rethinking critical periods without labels by analyzing deficits and Fisher Information on the pretext loss is conceptually elegant and broadly applicable.\n2. The CP-guided checkpoint selection provides a simple yet practical tuning mechanism. The selective self-distillation strategy offers a concrete approach to restore transferability while retaining late-stage in-domain performance gains.", "weaknesses": "1. The paper directly equates Fisher Information on the self-supervised pretext objective with parameter sensitivity to supervision signals to quantify plasticity, but it does not establish a theoretical connection between this proxy and transfer performance or generalization error.\n2. The paper posits a trade-off between sustained in-domain improvement and degraded out-of-domain transferability, which motivates the critical-period perspective, but lacks experiments linking this phenomenon to representation drift or task specialization.\n3. The use of broad phrases such as “across datasets or distributions” suggests generality, yet the paper neither defines nor categorizes the types of distribution shifts considered, nor decomposes which shift types benefit most, making the scope of conclusions unclear.\n4. The abstract claims coverage “across multiple SSL methods, architectures, and datasets,” but omits explicit statements on systematic coverage and boundary conditions. Without clarifying the circumstances under which the findings fail, the general conclusions risk overextension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eld1DC24de", "reviewer": "Reviewer_EgnN", "summary": "This paper investigates the temporal dynamics of self-supervised learning (SSL) and its effect on transferability. The authors identify a transferability trade-off: intermediate checkpoints during SSL pretraining often yield stronger out-of-domain (OOD) generalization, whereas longer training enhances in-domain (ID) accuracy. Authors characterize SSL pretraining into three phases—plasticity, consolidation, and overspecialization—tracked via Fisher Information dynamics and deficit injection experiments. They further propose 2 training scheme for improvement:\n1. CP-guided Checkpoint Selection (CPCS) for identifying checkpoints near CP closure, improving OOD transfer;\n2. CP-guided Self-Distillation (CPSD) to distill representations from CP checkpoints into later ones, mitigating overspecialization effects", "questions": "1. Can Fisher Information be replaced with a more stable unsupervised proxy?\n2. Do critical periods also exist in language or multi-modal SSL?\n3. How the model capacity will influence the CP closure?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Proposed that the learning process of SSL is stage-by-stage and provide evidences for verification.\n2. Proposed 2 training improvement scheme CPCS and CPSD without uesag of extra label.\n3. Propose novel explanation for OOD generalization decay at the end of training.", "weaknesses": "1. The paper defines Critical Period closure as the epoch when the Fisher Information (FI) curve stabilizes—operationalized as a near-zero slope. It better comes up with a formal metric consider many other variables like batch size, hyperparameters and even downstream tasks.\n2. Need to provide more insights about why the FI dynamic performance differently among different SSL architectures. Are these differences come from the loss term or the contrastive and reconstruction-based SSL methods？", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)