PhiNets: Brain-inspired Non-contrastive Learning Based on Temporal Prediction Hypothesis
Authors: Satoki Ishikawa
Makoto Yamada
Han Bao
Yuki Takezawa
Venue: ICLR 2025 Poster (2025)
Keywords: Non contrastive learning
Predictive coding
Eigenvalue dynamics
Temporal prediction hypothesis
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We introduce PhiNet, a model based on the temporal prediction hypothesis, which outperforms SimSiam in stability and adaptability within non-contrastive learning.

Abstract: Predictive coding has been established as a promising neuroscientific theory to describe the mechanism of information processing in the retina or cortex.
This theory hypothesises that cortex predicts sensory inputs at various levels of abstraction to minimise prediction errors.
  Inspired by predictive coding, Chen et al. (2024) proposed another theory, temporal prediction hypothesis, to claim that sequence memory residing in hippocampus has emerged through predicting input signals from the past sensory inputs.
  Specifically, they supposed that the CA3 predictor in hippocampus creates synaptic delay between input signals, which is compensated by the following CA1 predictor.
  Though recorded neural activities were replicated based on the temporal prediction hypothesis, its validity has not been fully explored.
  In this work, we aim to explore the temporal prediction hypothesis from the perspective of self-supervised learning (SSL).
  Specifically, we focus on non-contrastive learning, which generates two augmented views of an input image and predicts one from another.
  Non-contrastive learning is intimately related to the temporal prediction hypothesis because the synaptic delay is implicitly created by StopGradient.
  Building upon a popular non-contrastive learner, SimSiam, we propose PhiNet, an extension of SimSiam to have two predictors explicitly corresponding to the CA3 and CA1, respectively.
  Through studying the PhiNet model, we discover two findings.
  First, meaningful data representations emerge in PhiNet more stably than in SimSiam.
  This is initially supported by our learning dynamics analysis: PhiNet is more robust to the representational collapse.
  Second, PhiNet adapts more quickly to newly incoming patterns in online and continual learning scenarios.
  For practitioners, we additionally propose an extension called X-PhiNet integrated with a momentum encoder, excelling in continual learning.
  All in all, our work reveals that the temporal prediction hypothesis is a reasonable model in terms of the robustness and adaptivity.

Published as a conference paper at ICLR 2025
PHINETS: BRAIN-INSPIRED NON-CONTRASTIVE LEARNING BASED ON TEMPORAL PREDICTION HY- POTHESIS
Satoki Ishikawal:2, 1,2,,, Makoto Yamada 2.. Han B Yuki Takezawa
Science Tokyo, 2Okinawa Institute of Science and Technology, "Kyoto University
ABSTRACT
Predictive coding has been established as promising neuroscientific theory to describe the mechanism of information processing in the retina cortex. This the- ory hypothesises that cortex predicts sensory inputs at various levels of abstraction minimise prediction errors. Inspired by predictive coding, Chen et al. (2024) proposed another theory. temporal prediction hypothesis, to claim that sequence memory residing in hippocampus has emerged through predicting input signals from the past sensory inputs. Specifically, they supposed that the CA3 predictor in hippocampus creates synaptic delay between input signals, which is compen- sated by the following CAI predictor. Though recorded neural activities were replicated based on the temporal prediction hypothesis, its validity has not been fully explored. In this work, we aim to explore the temporal prediction hypothesis from the perspective of self-supervised learning (SSL). Specifically, we focus on non-contrastive learning, which generates two augmented views of an input image and predicts one from another. Non-contrastive learning is intimately related to the temporal prediction hypothesis because the synaptic delay is implicitly created by StopGradient. Building upon popular con-contratt learner, SimSiam, we propose PhiNet, an extension of SimSiam with two predictors explicitly corre- sponding to the CA3 and CA1, respectively. Through studying the PhiNet model, discover findings. First, meaningful data representations emerge in PhiNet more stably than in SimSiam. This is initially supported by our learning dynamics analysis: PhiNet more robust to the representational collapse. Second, PhiNet adapts more quickly to newly incoming patterns in online and continual learning scenarios. For practitioners, we additionally propose an extension called X-PhiNet integrated with momentum encoder, excelling in continual learning, All in all, our work reveals that the temporal prediction hypothesis a reasonable model terms of the robustness and adaptivity.
INTRODUCTION
How does learning and adaptivity emerge in biological system? It has been long-standing question in both neuroscience and machine learning. the neuroscience community, predictive coding has been a promising hypothesis support the flexibility of biological brains. While predictive coding was initially proposed to explain cortical functions, Chen et al. (2024) recently extended predictive coding propose the temporal prediction hypothesis, which claims that the hippocampus predicts future sensory inputs based on past experiences (Mumford, 1992; Rao and Ballard, 1999; Friston, 2005). Specifically, Chen al. (2024, Figure 1) modelled the hippocampus with the CA3 predictor followed by the CA1 predictor-whil the former yields synaptic delay to input signals, the latter compensates the time difference between the past and incoming signals by temporal prediction. This is the first attempt to explain the mechanism of sequence (short-term) memory from the viewpoint of temporal prediction. While they tested the temporal prediction hypothesis by using recorded neural activities, the validity of the hypothesis has not been explored sufficiently.
"Equal contribution "orresponning author, e-mail yww.oattto..yaaaaaaaaaa daiiaaaa@oaaa itsstttt........ -ip

---

Published as a conference paper at ICLR 2025
This work is aimed exploring learning model built upon the temporal prediction hypothesis to see when the hypothesis is reasonable. To this end, we shed light on self-suuer learning (SSL). SSL is paradigm to train a learner from input sensory patterns without supervised signals, which aligns to biological learning more closely. Over the past decade, machine learning researchers have developed number of SSL models. Popular SSL models are SimCLR (Chen et al., 2020a) and MoCo (Chen et al., 2020b), which are contrastive learning methods that learn data representations with two augmented views generated from an input image by minimizing the InfoNCE loss (van den Oord et al., 2018), requiring a tremendous number of negative samples to stably obtain representations. Thus, we focus on another SSL model, non-contrastive learning, which learns data representations from only the two augmented views without requiring negative samples. Specifically, SimSiam (Chen and He, 2021) is a natural model to study the temporal prediction typothesis-SinSian predicts one augmented view of an input image from another view, introducing an implicit time difference through the StopGradient operation. For this reason, we choose SimSiam, unlike the other non- contrastive models such as Barlow Twins (Zbontar et al., 2021). This implicit connection between SimSiam and the temporal prediction hypothesis is an appealing test bed to computationally verify how memory-based prediction processes in the brain behave in different scenarios.
Building upon SimSiam, we propose the brain-inspired SSL model PhiNer for investigating the effectivity of the temporal prediction hypothesis in the context of machine learning. PhiNet extends SimSiam by incorporating an additional predictor after the original predictor. We associate the original and additional predictors with the CA3 and CAI regions in the hippocampal model (see Figure in Chen et al. (2024) and Figure lb). We leverage PhiNet as a computational model to implement the temporal prediction hypothesis and study when it effectively learns sensory inputs.
Our first discovery is that PhiNet is less prone to the representational collapse, which leads to stable learning. Non-contrastive learning intrinsically faces the challenge of collapsing into a trivial representation because eliminates explicit negative signals. In Section 4, we theoretically analyse the learning dynamics of PhiNet to reveal that PhiNet is less sensitive to initialization and the weight decay hyperparameter, and has a wider retraction basin to a non-trivial representation (in (CI)), relative to SimSiam. This supports the empirically better linear probing performance and hyperparameter robustness of PhiNet across different image datasets. Our second discovery is that PhiNet empirically performs better in online and continual learning, in particular. We tested PhiNet and baseline non-contrastive learners by using the CIFAR-5m dataset (Nakkiran et al., 2021), exposing learners to a gigantic amount of input images but with only significantly fewer epochs. In this scenario, effective memory functions are necessary to lead the learning to success. As a result, PhiNet exhibits better accuracy with less forgetting than SimSiam. Therefore, the effectiveness of the temporal prediction hypothesis is witnessed from the perspective of the robustness and adaptivity.
For practically better performance, we extend PhiNet to additionally propose X-PhiNet, which incorporates momentum encoder, inspired by the Complementary Learning Systems (CLS) the- ory (McClelland et al., 1995). This extra momentum encoder represent long-term memory in the neocortex, storing information derived from the hippocampal model. X-PhiNet maintains good performances especially in online and continual learning scenarios.
Contributions.
Section 3: we propose new non-contrastive learning model called PhiNet, which is inspired by a hippocampal model (Chen et al., 2024).
Section 4: we compare the learning dynamics (Tian et al., 2021) of PhiNet and SimSiam. Consequently, it elucidates that PhiNet can avoid the complete collapse of representations (Liu et al., 2023; Bao, 2023) more easily than SimSiam with the aid of the additional predictor.
Section 5.1: we investigate the image classification performance of PhiNet using CIFAR and ImageNet datasets. We show that PhiNet performs comparably to SimSiam but is more robust against weight decay.
we further extend PhiNet by proposing X-PhiNet to integrate the neocortex model based on the Complementary Learning Systems (CLS) theory (McClelland et al., 1995). Experi- mentally, X- PhiNet works effectively online and continual learning.
Section 5.2:
Limitations One major limitation of our approach is the use of backpropagation, which differs from the mechanisms in biological neural networks. Our long-term goal is to eliminate backpropagation
2

---

Published as a conference paper at ICLR 2025
to better imitate brain function, but this work focuses structural aspects of network architectures. Currently, backpropagation free predictive coding mechanisms for complex architectures like ResNet are in the early stages of development, with most research being limited to simple CNNs. Future research should explore if the proposed structure can enable effective learning with backpropagation- free predictive coding. Another key difference between PhiNet and brains is the presence of recurrent structures, Making the data into time series data and adding a recurrent structure to the model remains as future work.
RELATED WORK
Brain-inspired methods. Predictive coding, initially introduced theory of the retina (Srinivasan et al., 1982), has gained attention as unifying theory of cortical functions (Mumford, 1992; Rao and Ballard, 1999; Friston, 2005). They suggest that brains operate by predicting sensory inputs at various levels of abstraction to minimise prediction errors. Recent studies have leveraged these ideas for contrastive learning (van den Oord al., 2018; Henaff, 2020). Chen et al. (2024) extended the predictive coding theory to the hippocampus with the temporal prediction hypothesis. Specifically, the temporal prediction hypothesis supposes that prediction errors are calculated with the CAI model and used to update the CA3 model. Some studies have attempted to apply the hippocampal model to representation learning (Pham et al., 2021; 2023). Among them, DualNet refines representation learning based CLS theory (McClelland et al., 1995; Kumaran et al., 2016), which supposes that the interplay between slow (self-supervieed) and fast (supervised) architectures is the basis of brain learning. Pham et al. (2021) examined supervised learning tasks alongside self-supervised training.
Self-supervised learning. Current mainstream approaches to self-supervised learning (SSL) often rely on cross-view prediction frameworks (Becker and Hinton, 1992), with contrastive learning emerging prominent SSL paradigm. In contrastive learning like SimCLR (Chen et al., 2020a), network contrasts positive (similar) and negative (dissimilar) samples to learn data representations. One limitation of SimCLR is its empirical reliance on gigantic negative samples. Theoretically, contrastive learning essentially requires huge number of negative samples (Bao et al., 2022; Awasthi al., 2022). To address this issue, recent research has focused on approaches free from negative sampling (Grill et al., 2020; Caron et al., 2020; 2021). For instance, BYOL (Grill et al., 2020) trains representations by aligning online and target networks, where the target network is created by maintaining moving average of the online network parameters. SimSiam (Chen and He, 2021) utilises a Siamese network to align two augmented views of an input by fixing one of the networks with StopGradient. While the lack of negative samples may easily yield collapsed representations, namely, constant representations, Tian et al. (2021) analysed the BYOL/SimSiam dynamics with a two-layer network and found that complete collapse is prevented unless weight decay is excessively strong. We partially leverage their analysis framework to explain the mechanism of our PhiNet. In recent years, many studies have leveraged SimSiam for continual learning (Smith et al., 2021; Madaan et al., 2022) and reinforcement learning (Tang et al., 2023). RM-SimSiam (Fu et al., 2024) and CaSSLe (Fini et al., 2022) enhance the performance of continual learning by incorporating memory block into SimSiam, while its architecture has not been neuroscientifically grounded. In video self-supervised learning, built on predictive coding principles, a common strategy is train a predictor that takes a frame or clip from one time-step to generate a distinct representation at different sime-step (Han al., 2019; 2020; Tan et al., 2023; Bardes et al., 2024).
Note that our aim to bridge the temporal prediction hypothesis and self-supervised learning. To this end, non-contrastive learning provides a better model because both hippocampus and neocortex do not have any mechanism corresponding to negative sample generation. Specifically, SimSiam is a simple yet powerful learning model, and we can benefit from its StopGradient to effectively draw connection to predictive coding. Thus, we focus on SimSiam as a backbone model in this work.
3 PHINETS (?-NETS)
In this paper, we propose PhiNets, which are non-contrastive methods based on CLS theory (Mc- Clelland et al., 1995) and the temporal prediction hypothesis (Chen et al., 2024). Chen al. (2024, Figure 1) provides a hippocampal model, where the entorhinal cortex (EC) serves as an imput signal layer, the CA3 region serves as the predictor, and the CA1 region measures the prediction emor.
3

---

Published as a conference paper at ICLR 2025
Sinn22
prodector
predictoc
Smm-1
Sta-1
Stnz
predicter A
predictor
poodictor
eucoder
/
encader
suooder f
sanddr fuoe
Imsgee
Image
langge
(a) SimSiam architecture.
(h) PhiNet architecture.
(c) X-PhiNet architecture.
The architecture of SimSiam (Chen and He, 2021) and PhiNets. EMA in the X-PhiNet model stands for the exponential moving average. The architecture originates from a single input, branches out into three paths, and then compares the similarity of all paths in Sim-2. Thus, call it PhiNet (f-Net) because the shape of the architecture resembles the Greek letter Phi ($).
Figure 1:
The CA3 region receives an input signal from the EC and recurrently forecasts future signals. The prediction output of CA3 is propagated the CAI region, which computes the discrepancy between the CA3 prediction and the EC input and refines the internal model stored in CA3. Compensating for the time differences between EC-CA3 and EC-CAI hypothesised to facilitate the learning and replay of time sequences in the hippocampus.
Whereas Chen et al. (2024) tested this model replicate recorded neural activities through simulation, we develop self-supervised learner PhiNet based on this hypothesis as follows:
We use deep encoders and/or Siong to represent cortex. See Section 3.2 for more details.
We model CA3 by a predictor network.
We model CA1 by combining loss function and another predictor.
We train the model by jointly minimizing the loss for the hippocampus and the neocortex models.
The long-term memory is implemented by an exponential moving average.
Figures 1a and 1 depict the architecture of Sim- Siam and PhiNet, respectively. Figure 2 illus- trates how PhiNet be interpreted pocampal model (Chen et al., 2024) under the temporal prediction hypothesis.
CA1
EC
Sigu2
hip-
V
NC
predictor g
Note that our approach diverges from the tem- poral prediction hypothesis method proposed by Chen al. (2024). Specifically, while they as- sume image sequence as input, we consider an original input image and two augmented im- ages input and feedback signals with time difference (thanks to the StopGradient opera- tion), expanding the applicability of hippocam- pal models to standard vision tasks.
predictor à CA3
XSG-1
SG-2
20))
EMA eacoder f encedee fuee
encoder
image
Figure 2: The interpretation hippocampal
model. NC stands for NeoCortex.
3.1 FAST LEARNING BASED ON TEMPORAL PREDICTION HYPOTHESIS
We provide detailed implementation of the hippocampal model, which serves as a fast learner. The model consists of EC, CA3, and CA1, and we describe each of them below.
Modelling of EC layer. The entorhinal cortex (EC) is the main input and output cortex of the hippocampus (Chen et al., 2024). Let us denote the original input as Rdd. We model that the hippocampal model has two augmented signals from the original input as x Rd and a Rd in addition the original input æ; Let IRd Rm denote the encoder. Then, the cortical representation in the EC is given as follows:
2 (1) f(a
(2)
f(x and 2, f(x.).
4

---

Published as a conference paper at ICLR 2025
We regard each corresponding to the layers II, III, and V of the EC in Chen et al. (2024, Figure 1).
For self-supervised training, we have the triplet dataset DD {(a: (1) z; 2;) )==))== The hippocampal learning can be characterized as a learning problem of the encoder f from the training dataset D.
Modelling of CA3 region.
The CA3 region is responsible for predicting future signals:
h ! h h(2(")), h; =((=;)))
where h IR - R is the predictor network. We implement the predictor with a two-layer neural network with the ReLU activation and batch normalization.
Modelling of CA1 region. CAI measures the difference between the predicted signal and its future signal. In this paper, we model CAI by a mixture a loss function and a predictor, while Chen et al. (2024) uses only the MSE loss for modelling CAT. For the Sim-1 of CA1 depicted in Figure
use the symmetric negative cosine loss function to measure the temporally distant signal z! (layer
III of EC) and the predicted representation from CA3 h (1)
h(2,()) (z=()))
)
( T
1
Lo (
2n
2n
12 2
where 0 represent the entire model parameter and x(() = sG(2 StopGradient, in which the gradient update shall not be executed.
RD is latent variable with
Remark that StopGradient yields "time difference" during backpropagation, for which we can interpret PhiNet as a hippocampal model. Let us look closely at Sim-1 in Figure 1b. We let fo denote the encoder f at the t-th gradient update. The left path of Sim-1 can then be expressed h)) h(f((=))). As the right path of Sim-1 is adopted with StopGradient, it can be written as (2) (G)) (z). Eventually, Sim-1 aligns f((z) and fi-1(x) by the predictor h. This Sim-1 interpretation indicates that PhiNet predicts past signals, which slightly deviates from the original temporal prediction hypothesis (Chen et al., 2024) supposing that CA3 is in charge of futture signals.
In addition to measuring the difference, the CAI region outputs the signal to the EC (V layer). Thus, we model the output of CA1 as follows:
(h ()), y. - (h
where g R + Rm is another predictor network. As we will see soon, CLS theory supposes that this feedback from CAI to the EC eventually propagates to the neocortex (NC), which is stored in long-term memory.
3.2 INCORPORATING SLOW LEARNING MECHANISM
The hippocampus and neocortex play crucial roles in brain cognition. For effective long-term memory storage, essential to transfer information from the hippocampus the NC. We first aim to formulate the joint learning of the hippocampus and NC models. Then, we propose using the exponential moving average (EMA) to transfer model parameters from short-term long-term memory, with the goal of compressing the original input signal.
In the EC layer, we model the update of the encoder function by using the output of CA1 and the representation z; = f(x;) (V layer of EC). Then, the loss function can be given as follows:
LNC (0)
1
lly
SG(z,) )
Σ lv 2n 1=1
SG(2;)
2n 1=1
This corresponds Sim-2 in Figure and regarded slow learning. Finally, the whole objective function of PhiNet is given as
T(O))
T-Cos
[NC(()
Hippocampus Sim-1
Sim-2
We then minimise L(Ø) to learn the hippocampus and the NC models. The optimisation can be efficiently performed using backpropagation. It is worth noting that utilise different loss functions for Sim-1 and/or Sim-2 in PhiNet. In this paper, we set Sim-1 to negative cosine similarity and Sim-2 to either MSE the negative cosine similarity.
5

---

Published as a conference paper at ICLR 2025
X-PhiNet: Slow learning via stable encoder. The original PhiNet formulation employs the same encoder for both the short-term and long-term memories for simplicity (Section 3.2). To further enhance slow learning, the imput representation in EC-V 2; should maintain long-term signals. Thus, we introduce the following stable encoder for long-term memory:
2,
fong (x;).
Then, solve the PhiNet optimisation problem by minimising both (xc and Coss using the exponential moving average (EMA) of the model parameters of f and ftang as
Etong +- /Etongg (1-B)E.
where & and Elong are the model parameters of f and flong. respectively, and 3 E [0, 1] is hyperpa- rameter. Model parameters persist in fiong more stably than the original encoder f. which facilitates slow learning. We call this method X-PhiNet.
4 WHAT WE BENEFIT FROM ADDITIONAL CA1 PREDICTOR: LEARNING DYNAMICS PERSPECTIVE
When PhiNet is compared with SimSiam, the additional predictor in CA1 is peculiar. We study the learning dynamics of PhiNet with a toy model. Despite its simplicity, dynamics analysis is beneficial in showcasing how the predictor g effectively prevents complete collapse.
Analysis model. Let specify the analysis model, following Tian al. (2021). The d-dimensional input is sampled from the isotropic normal æ ~ .N( I) and augmented by the isotropic normal (2) (2) ~ (Nx I²)), where o² indicates the strength of data augmentation. The encoder and predictors g and h modelled by linear networks without bias: f(xx) W Wa g(h) W,h, and h(z)
W, where W mx and Wg, W E m= The predictors h and g transform latents 2 (1), h() E T into h (1), (1) E Rm with the same dimension m predict the other noisy latent 22)) and the noise-free latent z, respectively (see Figure lb).
Unlike Loss introduced in Section 3, we focus on the (not symmetrised) MSE loss for measuring the discrepancy between (1) and (2)) for the transparency of analysis. Interested readers may refer to Halvagal et al. (2023) and Bao (2023) for further extension to incorporate the cosine loss into the SimSiam dynamics. Consequently, the expected loss function of PhiNet L(W,, W,, Wx) given as follows:
L = EgE 10 [W, SGGWS f 2)) + IW W. w -G(()=
We will analyse the gradient flow W (f.ghh) -V- pW (fggh (p 0: weight decay intensity) subsequently. The gradient flows are derived as follows (see Appendix B.
W, WT. ((+++))++ W, W, Wr,- (I+ W W)W, pW,,
W, - -((1+=2) -
- IJW, WT WW g
WA
W W Wr, - (I+ WDW,,, WT pWn.
Eigenvalue dynamics. The matrix dynamics we have derived are rigorous but not amenable further analysis. Here, we decouple the matrix dynamics into the eigenvalue dynamics. Let Φ = W E Rmxm Following Tian et al. (2021, Theorem 3) and Bao (2023, Proposition 1). we can show that the eigenspaces of &. W and Wn quickly align as t increases (see Appendix B.2). Therefore, assume the following conditions:
(A1) Wg and WA a are symmetric.
(A2) The eigenspaces of &, Wg. and WA, align for every time step t.
Under these assumptions, +, W g. and WA are simultaneously diagonalizable and can be written as Φ UA Wg g = UA UT. and W, = UA where the (time-dependent) common orthogonal eigenvectors. Here, As diag(ør,.... ... pm)) Ag diag(7,-- .. 7m], and

---

Published as a conference paper at ICLR 2025
STRONG (p- 0.12)
MEDIUM 0.03)
-2
-022
0.0
0.2
0.4
-0.2
0.0
U.2
0.4
LIGHT (p 0.003)
WEAK (p 0.0001)
-6
-0.2
0.0
0.2
0.4
002
0.0
0.2
0.4
Figure 3: State space diagrams of PhiNet dynamics with different levels of weight decay: STRONG (p = 0.12), MEDIUM (p = 0.03), LIGHT (p 0.003), and WEAK (p = 0.0001). The vector fields are numerically computed with o² 1.5. The state space bifurcates the boundary of each level. The nullclines shown with the green real (1) 0) and dotted ( 0) lines. The red dots are sinks.
A. diaglvi, pm are the corresponding eigenvalues. Noting that the dynamics quickly falls on to d(t) = (t)² we can decouple the matrix dynamics into the eigenvalue dynamics of (v,,)) only (shown in Appendix B.3 and B.4):
f ((1+)) (1+ -1 + 2 pab.
(PhiNet-dynamics)
1 (1- my.
From the (1b, (b,,)))dyyamll dynamiis,, easy to see that (1/,7) (0, 0) one of the equilibrium points. Can the cigenvalues escape from this collapsed solution?
Bifurcation of PhiNet dynamics. The state space diagrams of dynamics (I) shown in Figure 3. In this figure, the nullclines e and 0 shown in the green real and dotted lines, respectively, Noting that intersecting points of nullclines are equilibrium points (Hirsch et al., 2012), we observe saddle-node bifurcation of PhiNet dynamics parametrized by weight decay 0.
STRONG: Weight decay p is too strong that the collapsed point (t), 7) (0, 0) is unique sink.
MEDIUM: A new sink (v), 7) such that x/ > and R emerges. The number sinks is two.
LIGHT: Another non-trivial sink (4b, 7) such that v/ 7 emerges. There three sinks.
WEAK: The last sink emerges such that 16 < 0 and
0. The number of sinks is four.
Comparison with SimSiam dynamics. Tian et al. (2021) derived the SimSiam dynamics under the same setup as above. Specifically, they modelled the encoder f and the predictor with linear networks W y and W,z,, N, respectively, and defined the gradient flow dynamics with the MSE loss [[W W (1) x - SG W I1) (without the additional predictor g). By decoupling the matrix dynamics into the eigenvalues with the same adiabatic elimination Φ 2 we can derive the SimSiam dynamics solely with respect to vi-dynamics as follows:
(SimSiam-dye -isiamm-yymmics)
인 = (1 (1+ o )2 - pt:
(2)
We set (ablating the exponential moving average used in BYOL) in Tian et al. (2021, Eq. (16) to obtain this dynamics. SimSiam is free from the additional predictor g, so the dynamics (2) is univariate, unlike the bivariate system (1). Figure shows the dynamics (2).
7

---

Published as a conference paper at ICLR 2025
STRONG:
4(140])
MEDIUM: P < 4(140])
PhiNet-MEDIUN
Retraction basin to 1
Collagne
R
SimSiam-
Figure 4: Illustration of SimSiam dynamics (2). Un- like the bivariate PhiNet dynamics shown in Figure 3, SimSiam dynamics is univariate, shown in the vd-axis. The red dots are sinks.
The SimSiam-MEDDU flow is conjugate with the flow on the nullcline e 0 (green real line) in PhiNet-MEDIUM.
Figure 5:
Table I: PhiNet is comparable to SimSiam. We trained the models for 100 epochs and then validated them on the test sets using linear probing on the head. We trained with three seeds and calculated means and variances (subscripts). Both are unstable when the weight decay is small, but PhiNet still achieves high accuracy.
Accuracy by Linear Probing (w.r.. weight decay)
0.0
0.00001
0,00002
0.00005
0.0001
0.0002
SimSiam
25.41o.02
2.
60.82
44.5133.64
68.177.18
67. 67.12..13
PhiNet (MSE)
49.89 49.890.35
55.901.s7
33.927.42
66.73p.03
68.25g.21
67.83p.15
The SimSiam dynamics bifurcates into STRONG and MEDIUM p 1/4(1 + o²). These two modes correspond to STRONG and MEDIUM of PhiNet in that y-axis of Figure 4 and the nullcline & (green real line) in Figure 3 are topologically conjugate. The other LIGIIT and WEAK are peculiar to the PhiNet dynamics. By comparing Figures and 4, we have the following observations:
(C1) The retraction basin to non-coll non-collapsed solutions wider: Since SimSiam dynamics is univariate, y cannot avoid collapse once (0) is initialized outside the retraction basin to the non-collapse point vi* * 0 (namely, smaller than the source point Figure 5). By contrast, PhiNet avoids collapse even if (0) is close to zero, long (0) is sufficiently large (see the initial point * in Figure 5).
(C2) Even negative inirialization db can avoid collapse: In SimSiam-MEDIUS vp cannot be attracted to the non non-collappe col-coolapsed solution if (0) is initialized to negative. By contrast, PhiNet- WEAK has negative sink (at the bottom left Figure 3), which attracts negative initialization (0)
To sum it up, we have witnessed with a toy model that PhiNet is advantageous over SimSiam because the collapsed solution can be avoided more easily. This is why another predictor g is beneficial.
Remark The learning dynamics analysis in this section reveals that smaller weight decay brings us benefits only regarding the stability of non-collapsed solutions. Indeed, may benefit from larger to accelerate convergence to the invariant parabola and eigenspace alignment of (d, W W g)) (Appendices B.4 and B.2), euch of which corresponds to the positive effects #3 and #7 in Tian al. (2021), respectively. Moreover, moderately large p often yields good generalization in non-contrastive learning (Cabannes et al., 2023). Thus, smaller may not be a silver bullet.
5 EXPERIMENTS
We first the robustness of PhiNets against the design choice and weight decay hyperparameter. We then discuss the effectiveness of X-PhiNets in online and continual learning.
5.1 LINEAR PROBING ANALYSIS
Figure 6 and Table show the sensitivity analysis using CIFARIO (Krizhevsky, 2009) and Ima- geNet (Krizhevsky al., 2012), respectively, by changing the weight decay parameter. First,
8

---

Published as a conference paper at ICLR 2025
(al) Batch size-125 128
(a2) Batch size-102 1024
(b1) Batch size-
(b2) Batch size 1024
Eval Acc
90
80
70
2-14
2-11
2 2-14
2-11
2 2-14
2 2-11
2 weight decay
2
weight decay
weight decay
weight decay
- PhiNet(g=-I mse)
X-PhiNet(cos)
Barlow Twins
RM- Sim-Smm
PhiNet(mse
PhiNet(g=h, mse)
BYOL
SimSiam
X-PhiNet(mse)
SimSiam
PhiNet(mse)
Figure 6: PhiNet and X-PhiNet are robust against weight decay. We evaluated PhiNet variants in (al)-(a2) and compared the existing non-contrastive methods with PhiNet in (bl)-(b2) CIFARIO. We evaluated the performance using linear probing. The loss function in brackets represents neocortex loss (O). PhiNets perform particularly better than the baselines when weight decay is small.
Ace (knn)
80
Cosine Loss
-0.4
PhiNet
SimSiam
-0.6
PhiNet
SimSiam
-0.8
200
400
600
800
200
400
600
800
Epoch
Epuch
Figure 7: PhiNet is stable in the early stages of learning, We trained PhiNet and SimSiam with batch size of 1024 and the weight decay of 1e 4 on STL10. SimSiam unstable in the early stages of learning. This may be due to the cosine loss being too small in SimSiam.
emphasise the improvement of PhiNet over SimSiam for most of the setups, supporting the importance of the CAI predictor and Sim-2 loss. Subsequently, we closely look at the results.
PhiNet improves SimSiam. We observed weight decay significantly impacts the final model performance. When the MSE loss is used for Sim-2, PhiNet consistently outperforms the original SimSiam or other baselines (B YOL, RM-SimSiam) regardless of weight decay value, shown in Figure (right), Moreover, as shown in Figure 7, PhiNet have stabilizing effect during the early stages of training. This can likely be attributed to PhiNet's regularization effect, which prevents the cosine loss from becoming too small at the early phase of training.
Bless of additional CA1 predictor. To see whether the additional predictor g besides h is beneficial, we test variants of predictor g: 9 h (reminiscent the recurrent structure in CA3) and y (identity predictor). For CIFARIO with batch size 128, Figure 6 (left) indicates that the predictor choice slightly affects the final model performance we properly set the weight decay. However, we set the batch size as 1024, the separate predictor performs more stably over other choices.
Sim-2 loss should be MSE. Based on Figure 6 and Table 11 in the appendix, we found that the MSE loss used for Sim-2 generally improves model performance across most weight decay parameters, while the negative cosine loss performs comparably to the MSE loss with smaller weight decay but degrades it with larger weight decay.
Overall, sensitivity study CIFARIO revealed that PhiNets are robust the choice of the weight decay parameter, which supports the importance of the CAI predictor and Sim-2 loss. See Appendix E.1 for more detailed sensitivity studies. In addition, the results for different batch sizes and datasets (STL10 (Coates et al., 2011)) can be found in Appendix E.4.
5.2 ONLINE LEARNING AND CONTINUAL LEARNING
SimSiam and other con-oontraetive methods typically require up to 800 epochs of training on CIFAR10, 10, which is quite different from the online nature of brains. To address this, conducted

---

Published as a conference paper at ICLR 2025
Table X-PhiNet performs good results when memorization is important. We trained PhiNets on CIFAR-5m and Split CIFAR-5m. In Split CIFAR-5m, Ace is the average of the final accuracy (higher is better), and Fg is Forgetting (smaller is better). We present the results for two different weight decay (5e 4 and 2e 5) in Split CIFAR-5m.
Barlow Twins
PhiNet (MSE)
RM-
X-PhiNet (MSE)
X-PhiNet
BYOL
SimSiam
SimSiam
(Cos)
CIFAR-5m
81.05g.04
77.71
85.32o.
76.74182
82.09
87.30o.13
87.46o.14
Split C-5m
Acc
90.44..20
90.84
90.30o.17
90.69b.11
90.04a.11
91.02..36
92.83o.1
(wd=5e-4)
Fg
1.61o.c0
2.450.42
3.361.10
2.960.23
2.440.22
3.44g.36
1.95a.19
Split C-5m
Ace Fg
86.877.12
88, 20p.35
89. 39.86g.34
88 60..5
87. 87.070.36
90.900 90.90g.ar
90. 00.720.c
(wd=2c-5)
-0.16.223
0.36p..1
1. 1.299.94
0.05p.19
0.82g.14
- -1.03p.41
0.43g.17
BarlowTwins
Mean Ace (1 better)
92
Forget ting better)
BYOL
90
PhiNet(msee
X- X-PhiNet
X-PhiNee(cos)
RM-SimSiam
2-12
2-10
2-12
2
SimSiam
weight decay
weight decay
Figure 8: X-PhiNet is also robust to weight decay in continual learning, We measured the mean accuracy and forgetting at different weight decay on Split CIFAR-5m.
experiments using the CIFAR-5m dataset, which has six million synthetic CIFARIO-like images generated by the DDPM generative model (Nakkiran et al., 2021). Instead of training CIFAR1O with 50k samples for 800 epochs, we trained CIFAR-5m with 5m samples for 8 epochs. Although this is not exactly online learning, it seems closer to online learning compared to CIFARIO due the restriction on the training epochs. Table 2 shows that X-PhiNet has higher accuracy than SimSiam and PhiNet. The superior performance of X-PhiNet compared to PhiNet suggests that long-term memory with EMA is important in online learning. Sensitivity to weight decay and results for one-epoch online learning are given in Appendix D.I. X-PhiNet draws inspiration from CLS theory, which proposes a framework for understanding con- tinual learning processes in human brains. To evaluate the effectiveness of X-PhiNet in continual learning, we created a split CIFAR-5m dataset from CIFAR-5m, dividing it into five tasks, each with two classes. We trained on each task for one epoch and evaluated performance by the average accuracy across all tasks and the average forgetting, which is the difference between the peak ac- curacy and the final accuracy of each task. Table 2 shows that X-PhiNet has higher performance than SimSiam while maintaining minimal forgetting. Figure 8 further demonstrates that X-PhiNet consistently outperforms other methods like SimSiam in continual learning, regardless of weight decay. X-PhiNet also demonstrates high performance on Split CIFAR 10 and Split CIFAR 100, as well as when using replay methods (Appendix D.2).
6 CONCLUSION
In this paper, we proposed PhiNets based on non-contrastive learning with the temporal prediction hypothesis. Specifically, we leveraged StopGradient to artificially simulate the synaptic delay, and the prediction errors are modelled via Sim-1 and Sim-2 losses. Through theoretical analysis of learning dynamics, we showed that the proposed PhiNets have an advantage over SimSiam by more easily avoiding collapsed solutions. We empirically validated that the proposed PhiNets are robust with respect to weight decay and favorably comparable with SimSiam in terms of final classification performance. Experimental results also show that X-PhiNet performs better than SimSiam in online and continual learning, where memory function matters. These findings corroborate the effectiveness of the temporal prediction hypothesis when robustness and adaptivity important.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
The authors thank Kenji Doya and Tomoki Fukaki for their helpful comments. We also thank Rio Yokota for providing computational resources. M. and H.B. were supported by JSPS KAKENHI Grant Number 24K03004, H.B. was supported by JST PRESTO Grant Number JPMJPR24K6, and Y.T. was supported by JSPS KAKENHI Grant Number 23KJ1336.
REFERENCES
Awasthi, N. Dikkala, and P. Kamath. Do more negative samples necessarily hurt in contrastive learning? In ICML, 2022.
H. Bao. Feature normalization prevents collapse of non-contrastive learning dynamics. arXiv preprint MxXi:23399.6600 2023.
H. Bao, Nagano, and K. Nozawa. the surrogate gap between contrastive and supervised losses. In ICML, 2022.
A. Bardes, Q Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran, and N. Ballas. Revisiting feature prediction for learning visual representations from video. vrXiv:2404.08477, 2024.
S. Becker and G. E. Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms, Nature, 355(6356):661-663, 1992.
V. Cabannes, B. Kiani, R. Balestriero, Y. LeCun, and A. Bietti. The SSL interplay: Augmentations, inductive bias, and generalization. In /CML, 2023.
M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties self-supervised vision transformers. In ICCV, 2021.
Chen, Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020a.
X. Chen and K. He. Exploring simple Siamese representation learning. In CVPR, 2021.
X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 2020b.
Y. Chen, H. Zhang, M. Cameron, and T. Sejnowski. Predictive sequence learning in the hippocampal formation. Neuron, 11:22445-2658 2024.
A. Coates, A. Ng. and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In AISTATS, 2011.
E. Fini, V. G. T. Da Costa, X. Haumeda-Pinedas Ricci, K. Alahari, and J. Mairal. Self-supervised models continual learners. In CVPR, 2022.
K. Friston. A theory of cortical responses. Philosophical Transactions of the Royal Society B: Biological Sciences, 360(1456):815--44
2005.
Fu, Y. Gao, Z. Lu, H. Wu, and S. Zhao. Unsupervised continual learning of image representation via rememory-baseee simsiam. In ICASSP, 2024.
J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, B. Piot, k. kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent-a new approach self- sulf-supe learning. In NeurlPS, 2020.
M. S. Halvagal, A. Laborieux, and F. Zenke. Implicit variance regularization in non-contrastive SSL.. In NewrlPS, 2023.
Han, W. Xie, and A. Zisserman. Video representation learning by dense predictive coding. In CVPR Workshops, 2019.
11

---

Published as a conference paper at ICLR 2025
T. Han, W. Xie, and A. Zisserman. demory-augmentt dense predictive coding for video representa- tion learning. In ECCV, 2020.
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. CVPR, 2020.
O. Henaff. Data-efficient image recognition with contrastive predictive coding. In JCML, 2020.
M. W. Hirsch, S. Smale, and R. L.. Devaney. Differential Equations, Dynamical Systems, and an Introduction to Chaos. Academic Press, 2012.
Y.-C. Hsu, Y.-C. Liu, A. Ramasamy, and Z. Kira. Re-evaluating continual learning scenarios: A categorization and case for strong baselines. arXiv: 12488, 2018.
A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.
A. Krizhevsky. I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. NIPS, 2012.
D. Kumaran, D. Hassabis, and J. L. McClelland. What learning systems do intelligent agents need? complementary learning systems theory updated. Trends Cognitive Sciences, 20:512-534, 2016.
Z. Lin, Y. Wang, and H. Lin. Continual contrastive learning for image classification. In JCME, 2022.
Z. Liu, E. S. Lubana, M. Ueda, and H. Tanaka. What shapes the loss landscape of self supervised learning? In ICLR, 2023.
D. Madaan, Yoon, Li, Y. Liu, and S. J. Hwang. Representational continuity for unsupervised continual learning. In /CLR, 2022.
L.. McClelland, L.. McNaughton, and O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102(3):419, 1995.
D. Mumford. On the computational architecture the neocortex: the role of cortico-cortical loops.
Biological Cybernetics, 66(3):241-251 1992.
Nakkiran, B. Neyshabur, and H. Sedghi. The deep bootstrap framework: Good online learners are good offline generalizers. In ICLR, 2021.
Q. Pham, C. Liu, and S. Hoi. DualNet: Continual learning, fast and slow. In NeurlPS, 2021.
Q. Pham, C. Liu, and C. Hoi. Continual learning, fast and slow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
R. Rao and D. H. Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2(1):79-87, 1999.
O. Russakovsky, J. Deng, H. Su, Krause, Satheesh, S. Ma. Huang, A. Karpathy, A. Khosla,
M. Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 1:5:211-252, 2015.
Samtheen, G. Bachmann, S. Anagnostidis, and T. Hofmann. Random teachers are good teachers. In ICML, 2023.
Smith, C. Taylor, Baer, and C. Dovrolis. Unsupervised progressive learning and the STAM architecture. In IJCAI, 2021.
M. V. Srinivasan, S. B. Laughlin, and Dubs. Predictive coding: a fresh view inhibition in the retina. Proceedings of the Royal Society of London. Series B. Biological Sciences, 216(1205): 427-459. 1982.
R. Tan, M. De Lange, M. luzzolino, B. A. Plummer, K. Saenko, K. Ridgeway, and L. Torresani. Multiscale video pretraining for long-term activity forecasting. arXiv:2307. / /2854, 2023.
12

---

Published as a conference paper at ICLR 2025
Y. Tang, Z. D. Guo, P. H. Richemond, B. Avila Pires, Y. Chandak, R. Munos, M. Rowland, M. Ghesh- laghi Azar, C. Le Lan, C. Lyle, A. György, S. Thakoor, W. Dabney, B. Piot, D. Calandriello, and M. Valko. Understanding self pree-preeiitive learning for reinforcement learning. In ICML, 2023.
Y. Tian, X. Chen, and S. Ganguli. Understanding self-supervised learning dynamics without con- trastive pairs. In ICML, 2021.
G. M. Van de Ven, H. T. Siegelmann, and A. S. Tolias. Brain-inspired replay for continual learning with artificial neural networks. Nature Communications, 2020.
A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv: /807.03748 2018.
N. Vyas, A. Atanasov, B. Bordelon, D. Morwani, S. Sainathan, and C. Pehlevan. Feature-learning networks are consistent across widths at realistic scales, In NeurIPS, 2023.
J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny. Barlow Twins: Sepe-vupe learning via redundancy reduction. In ICML, 2021.
13

---

Published as a conference paper at ICLR 2025
A LIMITATIONS AND FUTURE WORK (EXTENDED VERSION)
One major limitation of our approach the use of backpropagation, which differs from the mech- anisms in biological neural networks. Our long-term goal is to eliminate backpropagation to better imitate brain function, but this work focuses on the model's structural aspects. Currently, backpropagation-fies predictive coding mechanisms for complex architectures like ResNet are in the early stages of development, with most research limited to simple CNNs. Conversely, non-contrastive methods like SimSiam require more advanced models than ResNet. Future research should explore the proposed structure can enable effective learning with backpropagation free predictive coding. Another key difference between PhiNet and brains the presence of recurrent structures. However, in this PhiNet, only one time step is considered, so it is possible that the recurrent structure required to predict time series data was not necessary. Making the data into time series data and adding recurrent structure to the model remains as future work.
It is also unclear whether cosine loss or MSE loss is more suitable for the Sim-2 in PhiNets. Cosine loss performs better when weight decay is small or online and continual learning where the additional predictor of PhiNets is important. However, MSE loss is preferable when weight decay is large on CIFARIO. I0. This is likely because using cosine loss in sim-2 has a stronger impact on learning dynamics compared to MSE loss. Analyzing gradient norms could be useful for this kind of evaluation, but is left for future work.
B DETAILS OF LEARNING DYNAMICS ANALYSIS
In this section, we complement the missing details of learning dynamics analysis provided in Section 4. In our analysis, we will use the gradient flow W (fg.h)} = VT. pW (p > 0), which is the continuous limit gradient descent. This corresponds to considering the following gradient descent in discrete updates and taking the limit + 0.
W (L + 7) W( V np WnpWssg
(3)
W Wlang) + -Wisan(()) = VT. VL- W (Sggh)
(4)
B.1 DERIVATION OF MATRIX DYNAMICS
Recall the PhiNet loss function:
T. = Ea [ W -Vaaa)sSS(WWaar oz + WWW,,, -S sz)
Let us derive its matrix gradient.
Vw, - 1 Vw, ( w/ WT SG w/ (z(
H(x WT SG(x W W. W (w SG(W
-(WT W. W E/
W W Elx (2) 1T
+(W, W. F|
W W W W,
=W, (II+ W. W/ E/ (TT W WF (1) W W ([ax
= =W
W, W,) - (I ( W
where the last line is derived from our assumption on the data distributions:
Eyg a lx (1 (1)T Eg [xx | + o (1+
E E ( a |x (z) (z) (1)T
= Ex [xx | I,
Ey (ca: (1TT
E (mar []
14

---

Published as a conference paper at ICLR 2025
Similarly, derive V Vw, L and Vw, I.
Vw, L= = W, W Bl
w WyE W
=
-
Vw, I = W. w W W,E [
+ W W, W. W El= W Wr (1) [[
{(1 + )T W WW,,,, (I w. W,wj.
From these, we obtain the matrix dynamics.
B.2 EIGENSPACE ALIGNMENT
Our aim is show that the three matrices + W and Wt & share common eigenspace, i.e., simultaneously diagonalizable, asymptotically in time Let
C₁ D We). C2 + Wol. and C3 [Wg W
where B| simultaneously diagonalizable, we show that the time-dependent commutators Ct),), C2((), and Ca(() asymptotically converges to O as + 00.
= AB BA is the commutator (matrix). By noting that commutative matrices are
Hereafter, we assume the symmetry assumption (A1) on W. and WA. and heavily use the following formulas on commutators implicitly:
AA O.
A - [B.
BC = [A, BJC + C(A
(AB, CJ A A C| CJB.
First, compute C based on the matrix dynamics of & (can be found in Appendix B.3) and W
C áW + oW
W $- w
(-(1+o) W W, & DW, W (W.
+ +WW.W, oW, W,) W, + + - -oo -WW]
+ +[[(1+ W - I,,,, + +WW, 1
+ +Wg((1++))W,,1+ W,,,, $ &W (I+ (I+ W 4 ΦW
W,) + bppb)
+3)0)+ 3. W W W 1 + + W wW (I W W.)
+ + W + W., W dw + $ WWW,
+ +[((1+o)), -I@ $
-3/C)+(1++
1 CW (W W.,
WCGW,WW.WWWW'CC, W, C
-(WCC,+C,W, B(C,W C.W)
+
DC, + +W,CC,)
w, +.C.W.W,++WW.C)-- -L C,#Ww) +C2.
15

---

Published as a conference paper at ICLR 2025
Similarly, C2 and C3 are computed:
C2 = -3pCp +
- - (4o + +C, BG
WW
- -(+ (1
+ (C,, W, -- (1 (WW C3
+ (I+
Next, we vectorize the commutator matrices-for C E Rhxh vec(C) E RR indicates a (column) vector stacking the columns C. For the commutators C1. C2, and C3. let write & vec(Cn) = 2)), and == vec(Cz). In what follows, we heavily leverage the vectorization formula:
vec(ABC) = (CT T & A vec = @I Ae) = (C B @ I)ve((A) vec(AB) = @I A) ve) = (B" & ())ec(A)
Here, A % B denotes the Kronecker product of two matrices. We write A + for notational convenience. We derive the ODE of & vec(C1) as follows:
A B
á = -3pIg + (1 ( + I) + ( @ W + (Ø @ ( W θΙІ
(W,W 2 I) (IO ( + (I 1 (W, W @ I) + (IO ( Wi)e
W, - W (I 4)n
(10L= W
- OI+ (1=
= -3pl Ki3n
where
K11 = I e ((1 (11 W II W W
K (1+0²) W. T) I +
K13 = (I+W, + (1 ( @I+ (IO (W, W I)).
Similarly, we derive the ODEs of r vec C2) and (= veo Ca).
n = K21 - (3pl + K2) - K23S.
Č K3ı K32 (3pl K33)
where
K21 = $ 8 I+ + @ I),
K = (1
(W, + Wa)),,
K -I Φ -1 ( (W, oW,)) (T W,).
K31 (1 + (L+ ( WL,,
Kg2 = -W, @ (I-
K3 = (1 (1+ ( @ (I- w)))
By combining all the above, obtain single ODE for (E. 17, C):
3pl + K11
K
K13
E
d
n
K21
3pI
K22
K2
17
dt
ç
K31
K
3pl + K
S
3
or alternatively. = -3pp + K)E. Note that K(t) is time-dependent. Finally, we can obtain the desired result by invoking Tian al. (2021, Lemma 2).
16

---

Published as a conference paper at ICLR 2025
Lemma 1 (Tian et al. (2021, Lemma 2)). Let H(t be time-varying positive semidefinite matrices whose minimal eigenvalues are bounded away from zero:
inf Min (H(t)) Ao > 0.
Then, the the following dynamics
dw(() H(()(())
dt
satisfies [|w(t)l2 I exp( exp(-Aot)] (oo))-Ao)]((00)) ((]] which means that w(t) 0.
When minimal cigenvalues of 3pl + K(t) are always bounded away from zero, we immediately see [t)) + O, namely, (t), (t),), C3(t)) + (0, O, O) as + co. The strict positive-definitnnss of 3pI + (t) would not be necessarily satisfied; however, larger weight decay > induces more easily. The convergence of the commutators is faster with larger p > 0 as well.
B.3 DECOUPLING INTO EIGENVALUE DYNAMICS
We have obtained the following matrix dynamics:
W, - ((+ W W W Wr - (I+ -W
W -(1 - WT WW
WA = -1) W W Wt - (I W W WW
Our aim is decouple the matrix dynamies into their eigenvalue counterparts. Beforehand, let us execute the change-of-variabe +-
W.
き- w,w/ + W,
- [ d²) (I W W 3) W. +I+ W) 4 {(1++0²) W (I+ W W,) - (I w v)) Wn, 2p
By the symmetry assumption (A1), (Ø, Wg. W. W dyyamiiss can be simplified as follows:
i = (1 + 2 [1+ 2 W.. 1 + [Wr. 2 (W, w +
- -2p4, (5)
W, , (11+0)) - @W W
W, ((1+ (I+
w g W, (I W w))) pW,. h
where (A, B) AB BA is the anticommutator for two symmetric matrices A and B with the same dimension.
Next, we decouple them into the corresponding cigenvalues. The parameter matrices are simultane- ously diagonalized by 4- U W, UA, U and W, UA U with the aid of the symmetry assumption (A1) and common eigenspace assumption (A2). Here, we can easily show that the eigenspace is -eneinndependent, namely. Ú = O, using the same argument of Tian et al. (2021, Appendix B. By multiplying U and U from left and right, respectively, -ddynaiics be written as follows:
A 2(1 + o0) A3)AAA Ao 2A,A,Ao 2pA..
where all matrices are diagonal, and thus, we can write down the dynamics in terms of j-th diagonal element (but the index is omitted for simplicity):
2 + + >") 2Φ6 + 200d 2pc.
We can decouple W gr and Wa wdydmmissss similarly:
t = - + 03) 1)64 MY:
1 = - } + (1 + 7) pbb
Note that s and y correspond to (one of the) eigenvalues of the linear networks h and g. respectively. Intuitively, we can regard y and "scalarization" of the predictor networks.
17

---

Published as a conference paper at ICLR 2025
To sum up, decouple the dynamics of (æ, Wh. Wg) into the following dynamics of (Ø, v):
( (e-dynamics)
2// (1 + (1 (1+ (1+++)] 2pΦ,
(6)
(W, --dynamics)
vb + - - -mb.
(7)
(W, -dynamics
v 0 p7,
(8)
B.4 ADIABATIC ELIMINATION
The eigenvalue dynamics obtained in Appendix 3. is jointly with respect to (Ø, v, 7). Here, climinate á by confirming that and & are asymptotically bound on an invariant parabola.
This can be integrated, and we obtain By combining (6) and (7), we have 2 á 22p(v x). the following solution:
(1) (()) C Cexp(-2pt) $220
where C' is constant of integration. Thus, (ø(t), (t)) converges this invariant parabola (9) expo- nentially quickly, which we suppose much faster than the dynamics stabilization. On this invariant parabola 2 the eigenvalue dynamics be further simplified as follows by eliminating øi
{x/ vb ((1 - o1) + + pbb,
= (1= p7:
Note that the convergence to the invariant parabola is faster when weight decay p is more intense.
C PSEUDOCODE FOR PHINET AND X-PHINET
The pseudo codes for PhiNet and X-PhiNet are shown in Listing. and Listing.2.
f: backbone + projection mlp
h: prediction mlp
g: prediction m1p
For x in Icader: # 1oad a minibatch x with n samples
x1, x2 (x), (vg & randem augmentation
z0, zl, z2 = (x), (x1), (x2) # projections, n-by-d
pl, p2 =b(21), h(22) predictions, n-by-d yl, y2 gipl), (p2) predictions, n-by-d
10
20 20. 20.detach()
Leos (pl, 22) /2 D (p2, /1) loss Lcor myc_loss 20) mse_loss (y2, 20)
12
13
Leos + Lcor L.backward # back-propagate
14
uf h) # SGD update
17
def D (p, 2): # negative cosine similarity
18
2 z.detach() stop gradient normali (ermalize(p, dim-1) # 12-nerma1 ize
19
normalize(2, dim-1) $ 12-normalize return (paz) sum (dim=1) .eea ()
21
Listing PhiNet Pseudocode (PyTorch-like)
backbone projection mlp
2 h: prediction mlp
# g: prediction mlp
for in Icader: # load minibatch with n samples
x1, x2 aug (x), aug (x) & randem augmentation
7
20, z1, z2 lonog (x), (xx1,, f(x2) projections, n-by-d
pl, p2 h(2l), h(2)) predictions, n-by-d
yl, y2 g(p)), (p2) predictions, n-by-d
10
z0 z0.detach()
18

---

Published as a conference paper at ICLR 2025
Table 3: X-PhiNet performs robustly for different weight decays on C5m.
Accuracy by Linear Probing (w.r.t.
weight decay)
0.0001
5e-05
2e-05
le-05
BYOL
67. 57.88
75.710.34
81.05p.04
80.70p.84
SimSiam
77. 77.69b.67
75.025.92
76.873.13
77.711.97
PhiNet
76.432.12
77.572.01
77.641.44
77.740.79
RM-SimSiam
74.24g.56
77.520.88
82.09p.22
79.38p.38
X-PhiNet with Aug (mse)
65.96 16.24
83.21o.15
86.45..25
85.170.54
X-PhiNet with Aug (cos)
84.31o.29
86.400.31
86.96p.17
84.85p.99
X-PhiNet (mse)
69.0214.25
84.240.37
87.30o.13
85.11
X-PhiNet (cos)
85.80
87.290.22
87.460.19
85.03p.19
X-PhiNet (cos, y 7)
84.72o.16
86.41o.08
86.71
83.83p.36
Table 4: X-PhiNet performs robustly well for one epoch training.
Accuracy by Linear Probing (w.r.t.
weight decay)
0.0001
5e-05
2e-05
le-05
BYOL
63.86p.77
59.80p.50
58.04g.52
57. 57.65g.38
SimSiam
68.50p.19
69.65p.29
69.411.06
69.60g.97
PhiNet
66.271.60
64.26 64.261.82
64.341.13
62.681.60
RM-SimSiam
62.901.11
63.301.s0
63.45p.92
63.051.78
X-PhiNet (mse)
74.250.s0
72.65g.85
71.20p.48
71.95g.65
X-PhiNet (cos)
74.760.62
72.89p.66
12.10p.15
71.93g.51
Loos (pl, 22) /2 (p2, / loss
12
Leor mse _as__loss(y)) (yl, 20) /2 mms__lss (y2, 20) /2
13
L = Lees + Lcor
14
L.backward() # back-propagate
update (f, h) # SGD update
16
f_long beta f_long (1-beta) f EMA for projection
17
18
def D (p, z): # negative cosine similarity
2.detach() () stop gradient
p normalize (p, dim=l) # 12-normalize
21
z normalize dim=1) # 12-normalize sum
22
return - (p+z) sum (dim=1) .ean ()
Listing 2: X-PhiNet Pseudocode (PyTorch-like)
D ADDITIONAL EXPERIMENTS ON ONLINE AND CONTINUAL LEARNING
D.1 CIFAR-5M (ONLINE LEARNING)
Table 3 shows the accuracy of linear probing for different weight decay values in CIFAR-5m. X- PhiNet consistently demonstrates high performance. In Table 4, we trained for only one epoch on CIFAR-5m. Also in this case, X-PhiNet performs better than SimSiam. Additionally, X-PhiNet with g achieves lower accuracy compared to the standard X-PhiNet (cos). This represents a major difference from the results shown in Figure 6, where both achieved similar accuracy with batch size 128.
In Table 3. "X-PhiNet with Aug (mse)" and "X-PhiNet with Aug (cos)" represent cases where data augmentation is also applied to the input of flong- In this scenario, all inputs to the model are augmented. While X-PhiNet with Aug outperforms other baselines such as SimSiam and RM- SimSiam, its performance is still inferior to standard X-PhiNet This discrepancy might be attributed to an imbalance in regularization strength, although the exact reason remains unclear and is left for future work.
19

---

Published as a conference paper at ICLR 2025
D.2 CONTINUAL LEARNING
Epochs per task. In Table 2, we trained on each task for one epoch. However, in Madaan et al. (2022), 200 epochs were trained for each task on Split CIFARIO, 10, and the number of iterations differs from this case. The effect of early stopping may be apparent when the number of iterations different. Thus, we trained on each task for two epochs to match the number of iterations. The result is shown in Table 5. The performance of X-PhiNet is still high even when the number of epochs per task is set to epochs.
Table 5: X-PhiNet shows higher accuracy when the number of epochs per task is increased. We trained X-PhiNet on Split CIFAR-5m. Unlike Table 2, this table presents results obtained from training 2 epochs for each task.
Barlow Twins
RM- SimSiam
X-PhiNet (MSF)
X-PhiNet (Cos)
BYOL
SimSiam
PhiNet
Split C-5m (2epoch)
Acc
91 23.36
02.250.10
90. 90.733.28
92.220.09
90. 10.31
92.33n.15
92.830.06
Fg
2.10
3.880.33
5.250.67
4.01o.26
6. 6.120.50
3.790.47
3.7
Replay with Mixup. Replay is one of the most promising methods for improving the performance of continual learning though it requires additional memory costs (Hsu et al., 2018; Van de Ven et al., 2020; Madaan et al., 2022: Lin et al., 2022). We thus examined the performance of our method in combination with the mixup- replay method proposed in (Madaan et al., 2022). Table 6 shows that when only one epoch is trained for each task, X-PhiNet shows considerably higher accuracy than the other methods. On the other hand, when we train two epochs for each task, the accuracy of other methods such YOL, Barlow and RM-SimSiam also increases, showing an accuracy comparable to that of X-PhiNet.
Table 6: X-PhiNet performs higher or comparable results for Split-CIFAR5m even with Mixup. We trained X-PhiNet on Split CIFAR-50 with replay methods.
Barlow Twins
RM-
X-PhiNet (MSE)
X-PhiNet (Cos)
BYOL
SimSiam
PhiNet
SimSiam
Split C-5m
Acc Fg
90.18a.m
91.51o.42
90.7 40.74
01.666.21
91.92o.17
91.78a.29
92 92.43p.14
(Jepoch)
0.36
-1.70g.09
00.97
2.211.54
-1.144.27
0..33..7
-0.60..5
Split C-5m
Acc
92.36g.03
91.770.01
92.36o.
91.00 91.001.7
92. 12.480.12
92. 92.12g 28
92. 92.26b.a5
(2epoch)
Fg
0.64p.02
2.240.01
-0.07p.64
1.971.41
1.050.a1
2.02a.71
1.82p.71
Split CIFAR10 and Split CIFAR100. Up to this point, we have experimented with continual learning using the CIFAR-5m-bused dataset. Now, we test on the standard benchmarks, Split CIFARIO and Split CIFAR100. Table shows that in both Split CIFARIO and Split CIFAR 100, X-PhiNet outperforms SimSiam. However, PhiNet sometimes shows higher accuracy than X-PhiNet. Note that PhiNet is a special case of X-PhiNet, and we have set the momentum of X-PhiNet to 0.99 in this study. If we carefully select the momentum value, X-PhiNet's performance might improve, surpassing PhiNet. When using mixup for replay, X-PhiNet shows significantly higher accuracy compared to other methods.
Effect of exponential moving average X-PhiNet has an additional hyperparameter, the exponential moving average. We set 8 0.99 in all the experiments in this paper. As shown in Table 8, in tasks such as continual learning, where it is important to apply a strong exponential moving average, accuracy increases 3 increases and then decreases again from certain point.
E ADDITIONAL EXPERIMENTS ON THE ROBUSTNESS OF PHINET
E.1 ADDITIONAL ABLATION STUDY WITH CIFARIO
Usage of original input: We first investigate what is the best way to input the original signal to the model. To this end, we first replace one of the augmented signals in SimSiam original
20

---

Published as a conference paper at ICLR 2025
Table 7: X-PhiNet produces good results when memorization is important. We trained X-PhiNet on Split CIFARIO. Acc is the average of the final Acc (higher is better), and Fg is Forgetting (smaller better).
PhiNet (MSE)
X-PhiNet T, MSE)
X-PhiNet (MSE)
X-PhiNet (Cos)
SimSiam
AM-SimSiam
Split C10
Acc
91.05
89.35
91.250.00
90.65p.4a
90.90..50
90.97o.4
(uineTune)
Fg
5.319.65
3.700.16
4.86p.50
1.02p.31
5.72g.82
3.95p.37
Split C100
Acc
77. 77.93u.64
78. 18
78. 78.50p.20
78.31o.18
77. 77.50p.04
77. 77.44b.28
(FineTune)
Fg
7.061.00 .00
0.570.98
6.51o.31
5.491.27
8. 8.460.21
1.46p.34
Split C10
Acc
90.68a.so
91. 91.14a.s4
89.89p.69
90.69p.21
90.490.32
91.566.11
(Mixup)
Fg
0.85g.16
1.080.60
1.040.22
-2.111.1
1.800.09
1.36p.15
Split C100
Acc
81.77g.14
82.47 82.477.70
10.76p.18
82. 22.160.76
83. $3.320.04
83 88p. an
(Mixup)
Fg
1.230.81
-1.35p.05
1.18
-1 -1.232.11
1.280.34
-0.07a.30
Table 8: X-PhiNet produces good results when memorization is important. We trained X-PhiNet on Split CIFAR 100 with mixup with different exponential moving average value.
EMA |
0.999
0.997
0.99
0.97
0.9
0.7
Split C100
Acc
83.19
82 22.62..06
83,88p.26
82.72o.96
81.766.64
81.73p.06
(Mixup)
Fg
-0.22g.34
0.33p.06
-0.07
0. 0.730.64
2.15p.11
1.82
input. Then, we found that comparing the augmented images and original input significantly degrades the model performance. This indicates that the original SimSiam performs pretty well even if we do not use the original inputs, and naively adding additional input hurts the model performance significantly. contrast, the PhiNet with MSE loss and StopGradient compares favorably with the original SimSiam model.
We analysed the impact of the StopGradient-2 technique, as shown in the table. The StopGradient operator effectively prevents mode collapse. Interestingly, while the StopGradient operator is not essential for avoiding mode collapse, models without it perform worse compared to those with it. Thus, the StopGradient operator contributes to improved stability when using the MSE loss. On the other hand, mode collapse still occurs with the negative cosine loss function.
StopGradient-2:
E.2 COMPARISON OF FAST LEARNER WITH SLOW LEARNER
We can observe that EMA plays as a slow learner through experiments. In Figure 9, we conducted continual learning experiments, where linear probing of the EMA encoder (dashed lines) performs consistently worse than the encoder without EMA (solid lines). This indicates that EMA does not quickly adapt the most recent samples and learns sense, we believe it is natural to think of slow learning as serving as a regularization for past samples, similar other continual learning techniques such as elastic weight consolidation.
stable features slow learner. In this
E.3 ADDITIONAL ABLATION STUDY WITH IMAGENET
Table 10 shows the ablation of additional predictors ImageNet. In this case, we used a higher weight decay of 1e-3. g h has a lower accuracy than other methods, which is consistent with the results in CIFAR10.
E.4 FOR DIFFERENT DATASETS AND EVALUATION METRICS WITH DIFFERENT BATCH SIZES
Table 11 and Table 12 present the CIFAR10 10 experiment results with varying batch sizes and weight decay. PhiNet shows equal or better performance than SimSiam across different batch sizes. The evaluation trends from KNN classification and linear probing also consistent. It is also consistent result that training on CIFAR 10 performs poorly when cosine loss used as the cortex loss function. Table 13 shows the results for STL.10, where PhiNet performs comparably SimSiam. However, as
21

---

Published as a conference paper at ICLR 2025
Table 9: Ablation study for PhiNet using CIFARIO data. We use SGD with momentum an optimiser and set the base learning rate as 0.03 and run 800 epochs. We evaluated the performance using KNN classification with K 200. See Table 19 for further details.
Ace (w.rt. weight decay)
Method
Sim-2
SG-2
Pred-2
0.0
0.0005
0.001
SimSiam
74.12p.39
90.39u.10
90.98p.uz
SimSiam (Orig-In)
72.82g.18
76.671.13
69.0311.37
MSE
g
77. 77.77
90.77g.22
91 91.38p.19
MSE
= I
77.634 77.63o.11
91.01g.12
91.50p.07
MSE
g
62.80o.47
91.40p.23
89.01
MSE
h
74.870.58
91.230.12
91.180.34
PhiNet
Cos
g
80.06g.47
87.73g.26
88.270.24
Cos
I
75.343.27
87.38 87.38o.17
87 87.90p.
Cos
g
27.57
9.98p.00
9.98g.00
Cos
75.990.28 h
85.970.23
85.04..11
CIFAR-5m
Spli--CIFAR1((dd=1e-)
80
Accuracy
80
80
75
70
70
70
60
2
6
0
10
10 task
Epoch
task
X-PhiNet (cos)
X-PhiNet (mse)
Figure 9: The accuracy for slow weight is lower than the accuracy for fast weight. The solid line represents the accuracy of the fast weights (the encoder without momentum), while the dashed line represents the accuracy of the slow weights (the encoder with momentum). Note that the accuracy for split-CIFAR represents the average accuracy.
Table 10: g h has low accuracy ImageNet. We trained the models for 100 epochs and then validated them on the test sets using linear probing on the head. Unlike Table 1, we train linear probing for 40 epochs to save computational costs.
Model
SimSiam
PhiNet (MSE)
PhiNet (g
PhiNet (g h)
Linear Probing Acc
66.35
66.64
66.47
$5.12
illustrated in Figure 7, PhiNet demonstrates better convergence in the early learning stages. In the early stages of training, when SimSiam is not stable, the cosine loss is smaller than that of the PhiNet, while the training continues, the cosine loss increases again, in agreement with the PhiNet. This suggests that something close to mode collapse occurs in the early stages of SimSiam training, while PhiNet may suppress this collapse.
E.5 PERFORMANCE TRANSFER LEARNING
We conducted experiments for transfer learning using object detection on VOC. In Table 14, following the original SimSiam paper, conducted pre-training experiments with two different settings for learning rate and weight decay. This table demonstrates that our X-PhiNet produces comparable performance MoCo across various tasks. Furthermore, shown in Table 15, we find that
22

---

Published as a conference paper at ICLR 2025
Table 11: PhiNet shows equal better performance than SimSiam. Sensitivity analysis for PhiNet using CIFARIO 10 data. We use SGD with momentum as an optimiser, set the base learning rate as 0.03, and run 800 epochs. We evaluated the performance using KNN classification with K 200.
Acc (w.rt. batch size)
weight decay
128
256
512
1024
SimSiam
86.352.28
88.05u.ei
88.34p.28
85.962.95
PhiNet
88.90
88.92o.11
88. $8.93p.33
88.910.07
0.0001
X-PhiNet (MSE)
88.67 08.67
88.6 0..27
88.71o.20
88.440.32
X-PhiNet (Cos)
82.575.67
79.3110.65
72.3519.83
84. 84.790.56
SimSiam
90.15o.15
90.360.15
90.39o.10
90.89p.0r
PhiNet
90. 90.40o.16
90. 90.48o.34
90.570.
91.150.00
0.0005
X-PhiNet (MSE)
90.05
90. 93.13..02
90.39p.12
90.70p.
X-PhiNet (Cos)
86.35u.23
86.36u.16
86.42p.11
86.65p.65
SimSiam
91.23o.11
91.30u..5
90.98p.uz
76.6812.83
PhiNet
91.23o.or
91.44o.08
91.50o.0s
73.977.u
0.001
X-PhiNet (MSE)
91.04p.13
91. 91.09p.14
91.11o.1a
90.08 90.08p.37
X-PhiNet (Cos)
86.54..18
86.331.00
86.79p.75
87.471.16
Table 12: Linear probing shows similar trends to knn classification. Sensitivity analysis for PhiNet using CIFARIO data. We use SGD with momentum as an optimiser, set the base learning rate as 0.03, and run 800 epochs. We evaluated the performance using linear probing on the head.
Acc (w.r.t. batch size)
weight decay
128
256
512
1024
SimSiam
88.641.73
89.44o.33
89.39o.49
88.01
PhiNet
90.27
89.830.35
89.79h.21
89.700.21
0.0001
X-PhiNet (MSE)
88.670.39
88.67p.23
88.71
88.440.32
X-PhiNet (Cos)
83.294.10
84.150.46
72.3519.84
83.232.52
SimSiam
90.39u.or
90.68p.u5
91.150.12
91.65g.06
PhiNet
90.68p.
90.870.34
91.11o.08
91.99 91.930.13
0.0005
X-PhiNet (MSE)
90.05p.29
93.13..02
90.39o.12
90.70o.
X-PhiNet (Cos)
86.52..12
86.47o.11
86.45o.16
87.00b.18
SimSiam
92.09
92.36p.11
92.460.29
77.71 12.73
PhiNet
92. 02.18p.00
92.440.21
92.630.0s
75.186.60
0.001
X-PhiNet (MSE)
91.04o.13
91.096.14
91.11o.13
90.08g.37
X-PhiNet (Cos)
87.04p.33
87.08p.20
87 87.24p.19
88. 18.115.08
Table 13: SimSham and PhiNet show comparable performance. Sensitivity analysis for PhiNet using STL10 data. We use SGD with momentum an optimiser, set the base learning rate as 0.03, and run 800 epochs. We evaluated the performance using linear probing on the head.
Acc (w.r.t. batch size)
weight decay
128
256
512
1024
SimSiam
85.972.48
87.26p.26
87.53p.20
87.170.05
0.0001
PhiNet
84.28g. 41
87.32o.
87.22..
87.01o.28
SimSiam
88.89
89. 23.23
89.39
88.57o.
0.0005
PhiNet
89.33g.1
89.26 89.26p.02
89.366.22
88. 08.62o.36
SimSiam
89.544.08
89.61o.11
89.37a.00
nannan
0.001
PhiNet
89.52..06
89.71
89.28g.23
10.00p.01
23

---

Published as a conference paper at ICLR 2025
VOC 07 detection
VOC 07+ 12 detection
Pretrained
30
AP
AP
APs0
AP
AP
MoCo v2
73.2
46.6
50.2
82.3
57.1
63.2
SimSiam (Ir=0.05, wd=le-4)
71.7
45.5
49.4
80.6
55.1
61.0
SimSiam (Ir=0.5, wd=le-5)
73.6
46.6
49.8
82.7
57.3
64.6
PhiNet (Ir=0.05, wd=le-4)
72.7
46.35
50.4
81.9
56.8
62.81
PhiNet (Ir=0.5, wd=le-5)
74.4
46.2
49.8
82.6
56.4
62.6
X-PhiNet (Ir=0.05, wd=le-4)
72.9
46.2
49.9
82.3
56.9
63.9
X-PhiNet (I==0.5, wd=le-5)
74.9
45.9
50.1
82.7
55.7
62.4
Table 14: In transfer learning for object detection, X-PhiNet is comparable to MoCo (He et al., 2020) and SimSiam. PhiNet is pre-trained by two training recipes similar to those in the SimSiam paper.
wd
2e-3
1e-3
5e-4
2.5e-4
1.25e-4
6.25e-5
3.125e-5
1.5625e-5
SimSiam
10.00
63.53
90.80
90.05
89.06
79.70
78.14
76.35
MoCo
87.47
88.11
87.76
87.01
85.79
83.45
81.34
80.44
PhiNet
10.00
78.33
91.19
90.35
89.34
86.25
83.21
81.60
Table 15: PhiNet is robust weight decay in transfer learning. Performance comparison of SimSiam, MoCo, and PhiNet at different weight decay values.
PhiNet demonstrates higher sensitivity to weight decay. Thus, seems that our PhiNet can be extended to object detection without any modifications.
E.6 ON THE AUGMENTATION FOR
We the unaugmented view for the Sim-2 loss to simulate "time difference" between different views, which is partially supported by the temporal prediction hypothesis. This architecture does slightly increase the performance. See Table 16 and Figure 10, where "with aug" performs slightly worse than our proposed architecture while robustness to weight decay is still higher than SimSiam.
E.7 COMPUTATIONAL COSTS
Table 17 shows the memory consumption when training CIFARIO. There is little overhead for PhiNet and X-PhiNet over SimSiam, as the maximum memory consumption during training not only related to weights, but also to gradients and activation state. In fact, the GPU consumption is highly dependent on batch size, indicating that the gradient and activation state, which are dependent on batch size, are dominant in this setting. Additionally, Table 18 includes comparison of training times, demonstrating that PhiNet can trained in time comparable to SimSiam.
Accuracy by Linear Probing (w.rt. weight decay)
0.0001
5e-05
2c-05
le-05
SimSiam
77.690.67
75.02s.92
76.873.13
77.711.97
X-PhiNet with Aug (mse)
65.9616.24
83.21o.15
86.450.2s
85.170.55
X-PhiNet with Aug (cos)
84.31g.29
86.40p.31
86.96a.17
84.85g.a 99
X-PhiNet (mse)
69.0214.25
84.24o.37
85.11o.17 87.30o.1s
X-PhiNet (cos)
85.80o.34
87.29o.22
87.460.19
85.03o.19
Table 16: Even when is augmented, X-PhiNet performs better than SimSiam (CIFAR-5m).
We used the same setting as in Table the original paper. "with aug" performs data augmentation for x, which is not augmented in Table 3.
24

---

Published as a conference paper at ICLR 2025
Batch Size
128
Batch Size 1024
95
BYOL
X-PhiNet (mse)
90
X-PhiNet (mse with aug)
Eval
X-PhiNet
(cus)
75
X-PhiNet
2 2-15
2
2 2-11
2
2
2
2
2
(cos with aug)
weight decay
weight decay
SimSiam
Figure 10: Even when a is augmented, PhiNet is more robust to weight decay than SimSiam (CIFAR10). We used the same setting in Figure 6 in the original paper. "with aug" performs data augmentation for which is not data augmented in Figure 6.
Table 17: Comparison of GPU memory costs. This is a comparison of memory consumption when training on CIFARIO. We report batch sizes of 128 and 1024.
Barlow
RM- SimSiam
X-PhiNet (MSE)
X-PhiNet (Cos)
Batch Size
BYOL
SimSiam
PhiNet
Twins
BS=128
4.3 (GB)
3.26 (GB)
2.79 (GB)
3.25 (GB)
4.06 (GB)
3.44 (GB)
3.44 (GB)
BS=1024
22.28 (GB)
17.10 (GB)
12.23 (GB)
17.18 (GB)
21.96 (GB)
17.11 (GB)
17.11 (GB)
E.8 STABLE RANK OF ADDITIONAL PREDICTOR LAYER
Figure 11 shows the rank for linear layers in additional predictor blocks of PhiNet. We used stable rank in this figure and it is defined as srank(M) [| || which is the lower rank of the standard rank and is more stable to the small eigenvalues of M. According to this figure, the rank of the additional layer remains large when the weight decay is small, suggesting that the additional layer may play a more important role in learning when the weight decay is small.
F EXPERIMENTAL SETTINGS
F.1 SETTINGS FOR TRAINING WITH CIFARIO, CIFARI00 AND STL10
Table 19 shows the model and experimental setup for Figure 6, Table 9, Table 11, Table 12 and Table 13. Note that in the graph of sensitivity with respect to weight decay, we explored wider range of values. For linear probing evaluation, both CIFAR10 10 and STL10, we used 50,000 samples for training and 10,000 samples for testing. We have implemented it based on code that is already publicly available'
trained the head layer by SGD for 100 epochs. For
F.2 SETTINGS FOR TRAINING ON IMAGENET
In our ImageNet (Russakovsky et al., 2015) experiments, we follow the formal implementation of SimSiam by Pytorch² (Chen and He, 2021). Table 20 shows the model and experimental setup for Table 1. For ImageNet, we used 1,281,167 samples for training and 100,000 samples for testing. We trained on the three seeds and obtained the mean and variance.
httpps://iithe com/ httm///acboookeeercchccoo simsiimmimmmmmmmmmmmmmmmmmmmmmam
Table 18: Comparison different models with varying batch sizes.
Batch Size
SimSiam
BYOL
Barlow-Twins
RM-SimSiam
PhiNet
X-PhiNet
BS=128
6.89 (h)
7.09 (h)
21.38 (h)
7.76 (h)
6.89 (h)
7.04 (h)
BS=1024
6.74 (h)
6.51 (h)
7.68 (h)
7.03 (h)
6.44 (h)
6.52 (h)
25

---

Published as a conference paper at ICLR 2025
Rank for h.i
Rank for h.out
Weight Decay
Stable rank
Stable rank
1c-05
2
3e-05
0.0001
0.0003
24
0.001
200
400
100
300
500
Epoch
Epoch
Figure 11: The smaller the weight decay, the larger the rank of the additional predictor. We trained PhiNet on CIFARIO with SGD and evaluate the rank for layers in additional predictor blocks.
Table 19; The experimental setups of Figure 6, Table 9, Table 11, Table 12 and Table 13.
Optimiser
SGD
Momentum
0.9
Learning
Learning Rate
0.03
Epochs
800
Backbone
ResNet18 _ait_/___________ __s_fffravvararnn variantl
Encoder
Projector output dimension
2048
Latent dimension
2048
Hidden dimension
512
Predictor h
Activation function
ReLU
Batch normalization
Yes
Latent dimension
2048
Hidden dimension g
512
Predictor g
Activation function (Hidden)
ReLU
Activation function (Output)
Tanh
Batch normalization
Yes
Computational resource
GPUs
V100 or A100
Table 20: The experimental setups of Table 1.
Optimiser
SGD
Momentum
0.9
Learning
Learning Rate
0.05
Epochs
100
Backbone
ResNet50
Encoder
Projector output dimension
2048
Latent dimension m
2048
Hidden dimension h
512
Predictor
Activation function
ReLU
Batch normalization
Yes
Latent dimension
2048
Hidden dimension g
512
Predictor
Activation function (Hidden)
ReLU
Activation function (Output)
Tanh
Batch normalization
Yes
Computational resource
GPUs
4x0000
26

---

Published as a conference paper at ICLR 2025
F.3 SETTINGS FOR TRAINING ON CIFAR-5M
CIFAR-5m (Nakkiran et al., 2021) is a dataset that is sometimes used vision dataset for online learning (Vyas et al., 2023; Sarnthein et al., 2023). We experimented with CIFAR-5h 55mm in a setting similar online learning. Note that CIFAR-5m has 5m samples, but we chose to train CIFAR-5m for 8 epochs, as most of the SimSiam training
CIFARIO involves training for 800 epochs. We experimented with three learning rates: (0.03, 0.01, 0.003), and selected the one that yielded the best results. For Barlow Twins, the learning rate of 0.03 does not converge, so 0.003 is chosen instead. For all other methods, a learning rate of 0.03 is selected. The model architecture is the same as in CIFARIO.
F.4 SETTINGS FOR TRAINING ON SPLIT CIFARIO, 10, SPLIT CIFARION 10 AND SPLIT-CIFAR-MM
As benchmark for evaluating continual learning, we used split CIFARIO and split CI- FAR100 (Krizhevsky, 2009). Additionally, we created split cifar-5ni 5m, which is inspired by split- CIFARIO but uses CIFAR Rm dataset. In split CIFARIO and split CIFAR5m,
split CIFARIO and CIFAR-50 into 5 tasks, each of which contains 2 classes. In split CIFARIO, we split CIFARIO0 into 10 tasks, each of which contains 2 classes. The model architecture is the same in CIFAR10. The implementation continual learning is based on the official implementation of Madaan et al. (2022).
We evaluated the results using Average Accuracy and Average Forgetting, The average accuracy after the model has trained for tasks is defined as:
T Σ AT T
OT:
(10)
where dr is the validation accuracy on task i after the model finished task The average forgetting defined the difference between the maximum accuracy and the final accuracy of each task. Therefore, average forgetting after the model has trained for T tasks can be defined as:
F
max (at,i - aT,)) (1..,-11)
(11)
T
27


Reviews:
[{"review_id": "oiv2efjYKH", "reviewer": "Reviewer_pS8F", "summary": "In this paper, the authors propose PhiNet, a self-supervised learning (SSL) method partly inspired by the temporal prediction hypothesis and a previous SSL method SimSiam. PhiNet can be seen as an extended version of SimSiam with an additional predictor and a stable encoder which is an exponential moving average of the main encoder. The authors analyzed the learning dynamics of PhiNet in a linear setting and found that the network is less prone to collapse compared to SimSiam. The method is then evaluated using the CIFAR benchmarks and is shown to outperform classical baselines in SSL, especially in online learning and continual learning settings.", "questions": "1. One major concern is that the proposed method is not really doing \"temporal predictive coding\", instead, the predictor is predicting the embedding of an augmented image. The way to introduce the model is thus somewhat misleading. It also seems more natural to use some video datasets if the goal is to build a model for temporal predictive coding. I wonder if the authors can explain more about the motivations here.\n2. When connecting X-PhiNet to the brain as in Fig 1(c), we are basically assuming that layers II and III in EC share the same encoder and the NC encoder is EMA of the one in EC. I wonder if the authors can provide more rationale for these assumptions.\n3. While there are some qualitative comparisons of the learning dynamics between PhiNet and SimSiam, I think the result would be stronger if there were also quantitative comparisons, e.g., something showing that more random seeds would result in collapse for SimSiam than for PhiNet.\n4. For empirical performance, the errorbar is missing from Fig 5 and Fig 7 so it's a bit hard to measure the margin of error.\n5. When comparing models on online learning and continual learning benchmarks, it seems that the EMA encoder is critical, but at least for continual learning, this is not very surprising since EMA essentially prevents the representations from drifting away too much. As baselines for comparison here (POYO, SimSiam, etc) are not designed for continual learning, it might make sense to include more baselines from the field of continual learning, otherwise, the claim that X-PhiNet particularly excels at online and continual learning seems a bit unfair.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The hypothesized correspondence of the algorithm to the learning and memory mechanisms in the brain is interesting to both ML and neuroscience communities.\n2. The authors conducted a detailed analysis of the learning dynamics which nicely explains the advantage over the baseline.\n3. There are extensive experiments that clearly illustrate the strengths and weaknesses of the models.\n4. The presentation is clear and a lot of details can be found in the appendix.\nOverall this is a solid and interesting contribution that bridges predictive coding and SSL algorithms known to the ML community. I thus support acceptance provided that the authors can reasonably address my concerns.", "weaknesses": "1. While this is understandable for a more brain-inspired algorithm, there is little performance gain on classical SSL benchmarks.\n2. The link from the model and task setting to the temporal predictive coding idea is not that strong, see below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e2Z3uY7KJf", "reviewer": "Reviewer_LRvN", "summary": "The manuscript links the temporal prediction hypothesis observed in the brain's hippocampus-cortex interactions to a novel self-supervised learning structure (PhiNets). PhiNets is largely based on SimSiam but incorporates one more predictor layer and StopGradient module inspired by neuroscience observations. This model demonstrates greater stability and resilience against representational collapse compared to SimSiam and shows superior performance in online and continual learning scenarios. Additionally, the study presents X-PhiNet, an extension incorporating an exponential moving average encoder for improved long-term memory retention in continual learning.", "questions": "- The weight decay term appeared out of sudden and definite needs more explanation about how it approximates modern optimizers and its relationship with learning rate etc.  because the robustness about decay is where PhiNet showed most improvements in the later numerical experiments. \n\n- The definition of time steps in the network and its link to temporal prediction is not well articulated. I roughy get the general idea how StopGradient could lead to synaptic delays but how are these two linked mathematically is still vague. Does one time step refers to one gradient update? Is gradient update performed in an online manner?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A clever observation between the temporal prediction hypothesis and the StopGradient module, making this architecture more brain-like. However, the definition of temporal steps was poorly articulated and needs more work. (See weakness)\n- Rigorously proved the benefits of adding one more predictor layer through the analysis of gradient flow in a linearized case \n- Demonstrated improved performance in terms of robustness to weight decay in terms of image classification", "weaknesses": "- Presentation of the paper needs more improvement: For example, it would be nice to add the diagram of SimSiam to Figure 1 and this is the main work the current manuscript was based on. \n- From the derivation, we know that adding one more layer of signal prediction could prevent from representation collapse. But is this going to slow down the convergence rate? What are the drawbacks of having one more layer of prediction. \n- In the performance evaluation plots: Phinet showed sufficient improvement compared to its original version SimSiam but not much improvement compared to other methods such as Barlow twins. And it's unclear why mse loss is better than cos loss in L_{NC}.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OKbAZu4Me1", "reviewer": "Reviewer_1cHn", "summary": "This papers takes another step in building non-contrastive self-supervised learning algorithms by taking inspiration from the architecture and supposed function of the hippocampal circuit. This PhiNets presented in this work are an extension of the SimSiam algorithm and the authors compare their method extensively to SimSiam, showing that it outperforms SimSiam in terms of robustness of low weight decay settings and continual learning. Their algorithm is competitive with many well-known contrastive and non-contrastive SSL methods (BYOL, Barlow Twins, etc.). Taken together, this is a nice paper showcasing the inspiration-from-neuroscience to usefully-deployed-ML-algorithm possibility.", "questions": "What is the physiological relevance of low weight decay - why is that the important analysis here? (the authors spend a lot of space on it)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality\n- As far as I could tell, their conversion of a hippocampal circuit process to a ML algorithm is novel. It does go beyond the closest match, SimSiam, in both architecture and abilities.\n\nQuality\n- In addition to empirical results, the analytical treatment is super useful to make us understand *why* their algorithm might outperform SimSiam in terms of robustness to low weight decay (among other things). \n\nSignificance\n- As I wrote in the summary, \"this is a nice paper showcasing the inspiration-from-neuroscience to usefully-deployed-ML-algorithm possibility.\"", "weaknesses": "I have two core concerns:\n1. The authors link their algorithm to the temporal prediction hypothesis. While I can see how in principle their setting (with image augmentations and the StopGradient) is similar to what temporal prediction would entail, results from this project cannot be used easily to validate their success as a success of temporal prediction SSL solely due to the reason that they do not use a temporal sequence. Yes, in a temporal sequence like a video subsequent frames could be thought of as augmentations but it is unclear if this algorithm will necessarily generalise to those settings because it is unclear if the class of augmentations used here resemble \"natural\" augmentations in videos.\n2. Keeping the hippocampal inspiration aside for a moment, the addition of g and Sim-2 is what distinguishes this algorithm from SimSiam. However, Fig.5. suggests that setting g=I doesn't influence the performance too much - it would be good to know if the continual learning results and other results do require that g is trainable and not I. Similar concern for Sim-2 - what happens when Sim-2 is left out? These ablation analysis are important to understand what makes PhiNet special. Also removing Sim-2 makes the algorithm similar in essence to SimSiam - I'd be curious if this version already outperforms the original SimSiam implementation - this will help us establish a stronger \"baseline\" for the full PhiNet.\n\nClarity suggestion:\n- In the analytical Section 4, it'd be great if you'd give us an intuition for what phi and gamma in Eq. 1 actually denote - how are they related to g and h, etc. This will truly help tie in the section to the rest of the paper (although it is fine as is too).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gq4pZZeo63", "reviewer": "Reviewer_QH5V", "summary": "This paper introduces PhiNet and X-PhiNet, where the input is processed in three parallel pathways with 0,1,2 predictors and with two pathways having the stop-gradient operation. The authors report favorable results over SimSiam and explore online/continual learning effectiveness in their models.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The idea is well motivated and the paper is thoroughly cited. The analysis is rigorous and convincing.", "weaknesses": "The majority of my weaknesses stem from my lack of understanding about certain claims, and I believe that this paper will be improved if the following points are made clearer:\n\n- Can the authors explicitly explain, with specific reference to any important terms, why stop-gradient leads to any temporal lagging? I understand how stop-gradient works, but I fail to see how it is related to time in any way.\n\n- I am not able to see any claims in section 5.1 in Figure 5, even though it is referenced in the text. What kind of improvement am I supposed to see and what effect am I looking for?\n\nI will update my review once I gain a better understanding of these results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)