Rethinking Hebbian Principle: Low-Dimensional Structural Projection for Unsupervised Learning
Authors: Shikuang Deng
Jiayuan Zhang
Yuhang Wu
Ting Chen
Shi Gu
Venue: NeurIPS 2025 poster ()
Keywords: Synaptic plasticity
Hebbian learning
Unsupervised learning
Local learning
Primary area: general_machine_learning
TL;DR: Our work effectively bridges Hebbian principles with explicit representation learning objectives, demonstrating considerable potential and biological plausibility.

Abstract: Hebbian learning is a biological principle that intuitively describes how neurons adapt their connections through repeated stimuli. However, when applied to machine learning, it suffers serious issues due to the unconstrained updates of the connections and the lack of accounting for feedback mediation. Such shortcomings limit its effective scaling to complex network architectures and tasks. To this end, here we introduce the Structural Projection Hebbian Representation (SPHeRe), a novel unsupervised learning method that integrates orthogonality and structural information preservation through a local auxiliary nonlinear block. The loss for structural information preservation backpropagates to the input through an auxiliary lightweight projection that conceptually serves as feedback mediation while the orthogonality constraints account for the boundedness of updating magnitude. Extensive experimental results show that SPHeRe achieves SOTA performance among unsupervised synaptic plasticity approaches on standard image classification benchmarks, including CIFAR-10, CIFAR-100, and Tiny-ImageNet. Furthermore, the method exhibits strong effectiveness in continual learning and transfer learning scenarios, and image reconstruction tasks show the robustness and generalizability of the extracted features. This work demonstrates the competitiveness and potential of Hebbian unsupervised learning rules within modern deep learning frameworks, demonstrating the possibility of efficient and biologically inspired learning algorithms without the strong dependence on strict backpropagation. Our code is available at https://github.com/brain-intelligence-lab/SPHeRe.

Rethinking Hebbian Principle: Low-Dimensional Structural Projection for Unsupervised Learning
Shikuang Deng ", Jiayuan Zhang"*, Yuhang Wu', Ting Chen', Shi Gu
Schooo of Computer Science and Engineering, UESTC Glasggow College, UESTC
School of Computer Science and Technology: Zhejiang University
State Key Lab of Brain-Machine Intelligence, Zhejiang University, China dengskQuestc. edu.cn, gus@zju. cerdzjju.du..com edu.cn
Abstract
Hebbian learning a biological principle that intuitively describes how neurons adapt their connections through repeated stimuli. However, when applied to ma- chine learning, suffers serious issues due to the unconstrained updates the connections and the lack of accounting for feedback mediation. Such shortcomings limit its effective scaling to complex network architectures and tasks. To this end, here introduce the Structural Projection Hebbian Representation (SPHeRe), novel unsupervised learning method that integrates orthogonality and structural information preservation through local auxiliary nonlinear block. The loss for structural information preservation backpropagates to the input through an auxil- iary lightweight projection that conceptually serves as feedback mediation while the orthogonality constraints account for the boundedness of updating magnitude. Extensive experimental results show that SPHeRe achieves SOTA performance among unsupervised synaptic plasticity approaches on standard image classification benchmarks, including CIFAR- CIFAR-100, and Tiny- ImageNet Furthermore, the method exhibits strong effectiveness in continual learning and transfer learning scenarios, and image reconstruction tasks show the robustness and generalizability of the extracted features. This work demonstrates the competitiveness and potential Hebbian unsupervised learning rules within modern deep learning frameworks, demonstrating the possibility of efficient and biologically inspired learning al- gorithms without the strong dependence on striet backpropagation. Our code is available https:/ / / github. com/ combbrainn-- intelligence lab/SPHeRe.
Introduction
In recent years, deep learning based on error backpropagation has achieved revolutionary devel- opments in the field of artificial intelligence [1, 2]. However, the biological plausibility of back- propagation remains subject of debate, particularly concerning its requirement for symmetric feedback connections and precise global error signal transmission, mechanisms not readily observed in biological neural circuits [3, 4]. At the same time, some neuroanatomical studies have found that the plasticity regulation of cortical synapses mainly follows local rules, with the strength changes dynamically dominated by the activities of pre-/postsynaptic neurons, occasionally under the syner- gistic regulation of higher-order signals such neuromodulators. Based on these neuroanatomical studies, researchers have proposed various biologically inspired learning algorithms to replace the backpropagation algorithm. The Hebbian rule, canonical algorithm of this type, was first pro-
Equal contribution Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

posed by Donald Hebb in 1949 [5]. Its core idea is that "neurons that fire together wire together," which means that learning and memory formation are achieved through the cooperative strengthening of synapses. As it does not rely on global signals, it belongs to unsupervised learning.
In order overcome the limitations of the original Hebbian rules in terms of stability, competitiveness, and feature extraction capabilities, researchers have developed various important variants based on wowpttooc--oooo--oiiio
them, such as threshold mechanisms, A significant theoretical leap was made by Pehlevan, Chklovskii, and collaborators, who established a principled connection between Hebbian learning and similarity- matching objectives [8. 9, 10]. Their foundational work demonstrated that optimizing objectives of the form | X - Y YI? 1--hich aim to preserve the geometric structure (Gram matrix) of the input-can can be achieved through fully local Hebbian and anti- Heti-Hebiian synaptic updates. However, extending these elegant theories to deep. nonlinear networks often theoretically relies on feedback connections for inter-layer credit assignment. More recent works, such as combining Oja's 's rule with Winner- Takee-- (WTA)) mechanisms [11, 12], have achieved impressive results on complex datasets, yet many Mebbian-inspired methods still lack clear, optimizable objective that seamlessly integrates with modern deep learning paradigms, hindering their scalability.
(BCM) rules [6], and Oja's rules [7].
In this paper, we revisit this problem from a different architectural perspective. We propose the Structural Projection Hebbian Representation (SPHeRe), a novel. beebiann-nsspitg greedy layer- wise unsupervised learning framework. We begin by simplifying the equivalent loss of Oja's 's rule, arriving at a stable objective that, coincidentally, aligns with the similarity matching goals pioneered by Pehlevan and Chklovskii. Then we provide theoretical proof, demonstrating that minimizing this simplified loss corresponds to finding an optimal low-dimensional projection of the input data (Lemma 4.2.1), akin to principal subspace analysis (PCA). Our core innovation lies in how we apply this objective. Instead of deriving synaptic-level rules or relying on feedback, we introduce a lightweight auxiliary projection module (ø), creating a purely feedforward, block-wise training architecture. This "main-patwway learning, side-pathway supervision" design decouples the main network's high- dimensional feature learning from the low-dimensional structural preservation objective. This allows the main block (f) to learn rich representations (Y') while the auxiliary block (ø) efficiently computes the structural loss (Lort)) to encourage feature decorrelation, SPHeRe provides a practical and effective bridge between Hebbian principles and modern deep learning. Our experiments show that SPHeRe achieves SOTA performance among Hebbian-inspired unsupervised methods and demonstrates strong generalization in continual and transfer learning, positioning it as competitive greedy layer-wise pre-training strategy. The following summarizes our main contributions:
similarity-matching
oow-dimesiional projection (Z). Complemented by an orthogonality constraint
We propose SPHeRe, a novel Hlebbian-inspired unsupervised learning framework, whose core innovation is purely feedforward, block-wise training architecture. By introducing lightweight auxiliary module (6). SPHeRe decouples high-dimensional feature learning in the main pathway from a low-dimensional structural preservation objective, enabling the effective application of Hebbian principles to deep, non-linear networks.
We provide a theoretical analysis (Lemma 4.2.1 and Appendix B) showing that optimiz- ing the core loss LSPUERE is equivalent to finding the optimal low-dimensional projection (principal subspace) of the input data. Through experiments on reconstruction and transfer learning, we further verify that SPHeRe extract robust and generalizable image features.
We demonstrate through extensive experiments that SPHeRe achieves state-of-the per- formance among Sebbiaan-inspied unsupervised methods on CIFAR-10/100 and Tiny- ImageNet. Furthermore, its strong performance in continual learning. transfer learning, and feature reconstruction validates its effectiveness as a greedy layer-wise pre- pre-training veeirrainng strategy that learns robust and generalizable representations.
2 Related Work
Synaptic Plasticity Rules. Synaptic plasticity, the modulation of connection strength between neurons, is recognized as fundamental biological substrate for learning, memory consolidation, and behavioral shaping [13]. The classical Hebbian principle, which posits that correlated activity strengthens synapses [5], laid the foundation for understanding activity-dependen modifications. But neuroscience experiments reveal that the rules of synaptic plasticity in the brain are much more

---

complex, such as the dependeenendd effects of STDP [14], network- balancing through homeostatic plasticity [15], influences between synapses through heterosynaptic plasticity [16], and even the control of plasticity itself, termed metaplasticity [17]. Recognizing the power of these biological adaptation strategies, researchers are increasingly working to integrate similar neuroplastic concepts into neural networks. Notable examples include investigations into the functional impact of homeostatic regulation in artificial systems [15], the synergistic use of Hebbian and non- oon-Hebbian rules within recurrent architectures [18], and the development of unsupervised learning paradigms based on complementary Hebbian and anti-Hebbial updates [12], demonstrating a growing trend towards biologically inspired learning mechanisms in modern artificial intelligence. A significant advancement in connecting Hebbian learning to principled optimization was made by [8, 9]. They demonstrated that optimizing similarity maaching objective, which minimizes the discrepancy between input and output Gram matrices, can be mathematically decomposed into a set of purely local, online synaptic update rules with Hebbian and anti- anti-Hebbian forms. Subsequent work extended this framework to deep networks [10], but these extensions often theoretically rely on feedhack connections to propagate error or target signals for credit assignment across layers. Starting from Oja's rule, obtain the same core mathematical objective as they did. But we propose fundamentally different architectural solution that is a purely feedforward, block-based approach applying this goal deep networks and avoiding the need for explicit feedback paths.
Unsupervised learning. Unsupervised learning (UL) aims to learn data representations from unlabeled data. Classical methods such as K-means [19] and Principal Component Analysis (PCA) [20] exploit the statistical structure of the data to perform dimensionality reduction or clustering. In the context of image classification, recent advances in UL for neural networks are typically categorized into two main approaches: self-supervised learning (SSL) and the reconstruction method [21]. SSL methods, including contrastive learning approaches such as SimCLR [22], BYOL [23], and SimSiam [24]. rely on pretext tasks and often employ pseudo-labels. The reconstruction method, on the other hand, trains the encoder and decoder simultaneously through image reconstruction loss to train the encoder to extract -imeennniional representations of the images, and the decoder can reconstruct the images from the oww-dimesiional representations. The autoencoder [25]. VAE [26], and MAE [27] are representative of this type of method. However, the inherent limitations of these mainstream SSL and reconstruction methods are that they fundamentally rely on end-to-end gradient computation through backpropagation, a process that lacks biological plausibility. In this paper, we attempt to establish a biologically inspired new path for unsupervised learning.
3 Preliminaries
3.1 Hebbian Rule
The Hebbian learning rule satisfies the locality property, where the synaptic connection is determined solely by the spike states of its pre- and post-synaptic neurons, without the influence of other neurons. In the classical Hebbian rule, the equation is Awtjj = r ny representing the product of the firing rates of the pre- and post-synaptic neurons. Then consider simple linear matrix multiplication process, we use X E Bx Bx to represent the input vector, Y E RBxM to represent the output vector, W E R.N X to represent the synaptic weights, and the relationship between them is: Y = X W. So. the original Hebbian rule for W is:
AW = = (XTx))
(I)
3.2 Oja's rule
The classical Hebbian rule has certain deficiencies, such as the potential for overtraining and instability due to consistently positive firing rates. The Oja's rule is well-known variant of the Hebbian rule, incorporating an additional decay term in its formula; the greater the post-synaptic activity, the more significant the decay. Under Oja's rule, the weight update equation is Awyy = nn( WiyU;)u And the Oja's 's rule formula for W can be expressed as follows:
AW- 7 Y W Y -Y.
(2)
To some extent, Oja's rule approximates principal component analysis (PCA) for dimensionality reduction and has the effect of extracting features. However, the current Oja 's rule does not consider
3

---

the complex nonlinear transformations and feature quantity adjustments in deep learning, making it difficult directly upply multilayer networks.
Methodology
This section introduces the Structural Projection Hebbian Representation (SPHcRe) method, an unsupervised learning approach inspired by Hebbian principles but adapted for modern deep learning architectures. SPHeRe aims learn meaningful representations by preserving the structural rela- tionships within the input data, utilizing a specific loss function, an orthogonality constraint, and a lightweight auxiliary block.
4.1 Analyze Hebbian rules from the perspective of loss functions
In this section, we will analyze how the Hebbian rule can extract information from the perspective of the loss function. Since the Hebbian rule does not care for its activation function, just start from the simplest linear layer, where we use X E BRxNN to represent the input vector, Y E BxM represent the output vector, W R to represent the synaptic weights, and the relationship between them is: W.
Original Hebbian rule. For the simple linear layer trained using the classical Hebbian rule, its equivalent loss function is Lheth = | Minimizing the loss Lhebs will increase the overall magnitude of the output matrix Y, which means that the values of the elements in Y will increase. This optimization might lead to a larger variance the distribution of Y, making easier to classify. However, without constraints, this could result excessively large output values, potentially causing numerical instability. In contrast, the equivalent loss function of anti-hebbian is Lanti-hebb = = l and optimizing Lanti-hebb equivalent driving all elements of the output as close zero as possible. This is a form of L2 regularization for the output.
Oja's rule. Oja's rule improves upon the classical Hebbian rule by introducing suppressive term that prevents the weight matrix from growing indefinitely. The equivalent loss function of Oja's 's rule is:
Loja Ir (LYY - -xx
-xx))
(3)
In which Tr denotes the trace of matrix. Optimizing this loss function is somewhat equivalent to minimizing the Gram matrix (sample relationship matrix) Kx = XXX of the input X and the gram matrix Ky YY of the output Y. When M 2 N, the weight W tends to WWT == IN; when M < N, the output Y becomes a representation of X, which will mimic the geometric structure of X in the N-dimensional space much as possible in the M-dimensional space, thus preserving the pairwise relationships between samples as much as possible. In the loss function, (XXT)-1 acts weighting mechanism, weighting the errors according to the characteristics of the input data. However, the existence of (XX*) ) requires (XXT) to be invertible, and it may lead to numerical instability.
4.2 Structural Projection Hebbian Representation (SPHeRe)
4.2.1
Preserving Data Structure
SPHeRe starts from the equivalent loss function of Oja's rule. First, we simplify the loss function by removing the inverse term (X (- and the constant term. This new objective directly seeks to learn an output representation (Y) that preserves the pairwise structural information inherent the input, as captured by their respective Gram matrices (XXX T and TYT). While derived from Hebbian learning rule (Oja's s). this simplified loss also connects to dimensionality reduction techniques aiming for optimal data projection. We thus define the core SPHeRe loss function as:
LSPHERe Tr ((YYT -
X JYY - = XXT 2
From the above equation, it can be intuitively seen that the goal of simplifying the loss function to make the output Gram matrix approach the input Gram matrix. Some works use this method to achieve multidimensional scaling (MDS) [28] or non-negative matrix factorization (NMF) [29].

---

Notably, while we derive this objective from a Hebbian perspective, it is mathematically identical to the umiiarity-matchhae objectives studied extensively by [8]. The theoretical justification for this loss in capturing essential data structure comes from its connection to principal subspace projection, as formalized in the following lemma.
Lemma 4.2.1, Let X E BR be the input data matrix and let Y XW E RB be the ouppu, W E R.N be the weight matrix, and M < N. Consider the loss function L ||Y XX II? Minimizing with respect to W yields an optimal output Y given by Y = XVM. where Va consists of the first M right singular vectors of X. This Y* represents the projection of the rows of X onto the M M-dimensional principal subspace, and Y*y*T is the best rank-M approximation of XX in the Frobenius norm.
The relevant proof is provided in Appendix A. From the lemma 4.2.1, can be concluded that the objective of the new equivalent loss function is to find the optimal projection of the low-dimensional space of the input X, and the minimum value of the equivalent loss function is Σ 2i= M--1 of where o; is the i-th largest singular value of the input X. addition, when the relationship between and X is nonlinear, such as Y = f(X), where is nonlinear neural network (universal approximator). Due to the powerful ability of f. minimizing the loss function will still lead Y to approach the optimal solution Y XVM to minimize L.
After obtaining the equivalent loss function, we can find the derivative of W as: Vw -YY XX TY. The outer term ( still reflects the fundamental Hebbian rule: the change in weights is related to the correlation between pre-synaptic activity (X) and post-synaptic activity (Y). (YY XX the weight term of the fundamental Hebbian rule. It wants to transform the statistical structure of the input space (XX))) into the statistical structure of the output space (YYT) and adjust the Hebbian rule through the higher-order relationships between the sample pairs in the input and output.
4.2.2 Enhancing Features: Orthogonality constraint
To enhance the quality and interpretability of the extracted features. we introduce an additional Orthogonality constraint term, called orthogonal loss. This loss encourages the different features represented by the columns of the output matrix Y to be mutually orthogonal. Decorrelated or orthogonal features often lead to less redundant representations and can improve the effectiveness of downstream tasks. The Orthogonal Loss function is defined as:
booth YTT TM 2
(5)
F
The matrix Y E IR X.M computes the product of dots between all pairs of columns (features) of Y. So, minimizing Louth pushes the columns of Y towards orthogonality. By promoting orthogonality among the features, we aim to reduce redundancy in the learned representation, ensuring that each feature captures distinct aspects of the input data. Under orthogonal loss, the gradient of W Vs: X Y IM). Interestingly, the gradient associated with this loss also exhibits Hebbian modulation structure, which adds a weight term Y Y IM to the standard Hebbian rule and drives the output features of Y be orthogonal as possible.
As a result, the total loss combines the new hebbian loss and the orthogonality loss and can be described as:
Loaal LSPHeRe + XCotth
(6)
where À is a hyperparameter that balances the contribution of the orthogonality constraint against the low-rank projection.
4.2.3 Integrating SPHeRe into Neural Networks: The Auxiliary block
The simplified Hebbian loss function LSPHORe and the orthogonal loss Lonh presented earlier are defined based on a linear transformation Y XW. However, deep learning heavily relies on non-linear layers to capture complex data patterns. Directly applying linear Hebbian loss to train deep neural networks is often insufficient. Therefore, we extend our Hebbian method to non-linear settings.
5

---

Linear to nonlinear. we replace the linear transformation with a nonlinear mapping represented by a neural network. Let Y = f.x, W) denote the output of neural network layer (or block) f with parameters W, given the input X. According to Lemma 1.2.1, minimizing the simplified Hebbian loss LSPIERE drives the output Y towards the optimal low-dimensional projection of the input X. Meanwhile, the neural network f is universal approximator, it has the capacity to learn complex functions, including the optimal linear projection identified by Lemma 4.2.1 if that minimizes the objective. Thus, even with non-linear function f. optimizing the new Hebbian loss that compares input and output statistical structures can guide f to learn meaningful representations. We provide the corresponding verification experiments in Appendix B.
Auxiliary block. To further enhance flexibility and computational efficiency, we introduce an additional lightweight non-linear projection layer with parameters A. This layer takes the interme- diate representation Y' as input and produces the auxiliary output Z = o(Y', 0) off(X, X W), !). We provide a comparison between SPHeRe with the auxiliary block and other similar methods in Appendix G. This two-stage non-linear mapping offers several advantages:
Computational Efficiency. The dimension My of the auxiliary output Z can be chosen to be small (e.g.. Mz < M'), significantly reducing the cost of computing ZZ and the LspHere loss, making it practical for large batches and deep networks.
Flexibility for Main Block (f). The main block f can learn a potentially nigh-dimensional (M') and complex intermediate representation Y' without being directly constrained by the low-rank structure imposed by LSPUERE The auxiliary network á handles the projection to the low-dimensional space Z where the structural comparison occurs.
Effective Nonlinear Learning. Even though f and e are nonlinear, minimizing LSPHERe on Z still effectively guides the network. As indicated by Lemma 1. the target for Z is the principal subspace projection XVM2.. Since f and Φ are universal approximators, they possess the capacity to learn the complex mapping from X to an approximation of XVMz through Y' and Z during optimization. Experimental validation (detailed in Appendix B) confirms this: using this non-linear auxiliary setup, the learned auxiliary features exhibit high structural similarity (measured by Centered Kernel Alignment (CKA) and alignment of Singular Value Decomposition (SVD) components) to features obtained from the ideal linear projection defined by Lemma 4.2.1. This demonstrates that LSPhERe effectively drives the non-linear system to capture the principal structural information of the input via the auxiliary branch.
Locality. The objective of W is determined locally by comparing its input X repre- sentation & derived (via Φ) from its output Y'. Although gradients still flow through f and 6 during optimization, the objective itself is defined relative to the block's input and (projected) output structure, differing from end-to-end supervision from a final task loss.
4.3 Distinguishing SPHeRe from PCA and Oja's Rule
While inspired by principles related to Oja's rule and PCA, SPHeRe differs significantly a practical deep learning method: (1) Nonlinear: SPHeRe is explicitly designed for training deep, nonlinear neural networks layer-by-layer
block-by-block, whereas standard PCA is a linear technique and Oja's rule is typically analyzed in linear or simple non-linear settings; (2) Auxiliary block: The use of the auxiliary block é is a core architectural innovation of SPHeRe. It decouples the dimensionality required for efficient structural comparison (M2) from the dimensionality of the main block feature representation (M'), allowing flexibility and scalability not present in direct PCA/Oja implementations: (3) Motivation The primary goal is unsupervised pre-training or layer- wise training of deep networks to learn features beneficial for downstream tasks (like classification, transfer learning), rather than solely dimensionality reduction as in standard PCA. We provide a comparison of SPHeRe with more unsupervised dimensionality reduction methods in Appendix G.
4.4 Overall Method
The core idea of SPHeRe is to leverage an auxiliary lightweight nonlinear projection & to compute low-dimensional representation Z, whose relationship structure (captured by a kernel matrix Kz) is driven to match the relationship structure of the input X (captured by Kx). This is achieved

---

by minimizing the simplified Hebbian loss LsPHeRe defined on these kernel matrices. Then, an orthogonality constraint Conh can be applied to the output Y' of the main block f to encourage the decorrelation of the features. The auxiliary branch allows the main block f to operate with potentially higher-dimensional intermediate features Y' while keeping the Hebbian loss computation efficient via the low-dimensional Z. The process can be summarized by the following equations and Fig. 1:
Layer
Layer
y (),
Z = 2(Y,0)
X 1/
0 = Z/11Z112
(Kx) = XX
(Kz) 7Z
ESPHERE 1| Kx 1?,, Lunth = |2 Z -11
Forward
Backward Detach Cradient
Diaal = LSPHORE + Aootth
SPHeRe
Figure 1: The concept of SPHeRe.
5 Experiments
5.1
Experimental Setup
Our Hebbian network architecture consists of three Hebbian-trained convolutional layers (384, 768, 1536 channels) with kernel size 3 and a fully connected output layer trained with backpropagation. Each convolutional layer is followed by max-pooling layer reduce the image resolution by half. A skip connection is employed in the final layer, in which an Avgpool(2.2)) is used to downsample, but gradients are not propagated backward through it. If not specifically stated, we use Leaky-ReLU as the activation function. A lightweight auxiliary network, é, is introduced for the downsampling of convolutional features. Let the input feature map of & have k channels. The network & comprises three sequential stages: (i) a 1 X 1 convolution halving the channel dimension to k/2; (ii) an Adaptive Pooling operation collapsing the spatial dimensions the intermediate k/ /2-diimenioonal features onto final 256 256-dimensional representation. We trained the network using Adam (learning rate: 0.001, weight decay: 0.05) with batch size of 128 and a learning rate scheduler; the hyperparameter X for Lunt is set to 0.8; no data augmentation was applied except for standard normalization. For fair comparison, the SoftHebb baseline adopted the same network structure but otherwise retained its standard optimal hyperparameters.
1: (iii) a fully connected layer projecting
5.2 Comparison to Existing Works
In this section, we compare our method with other existing unsupervised synaptic plasticity learning methods on the CIFAR-IO, CIFAR- 100 and Tiny-ImageNet datasets. For the SoftHebb method, we use open source code and verify it in our network architecture. It is possible that SoftHebb is sensitive to network structure, which has led to a little decrease in network performance than reported in the paper after using our structure. As shown in Table 1, our method consistently outperforms these existing methods on all three benchmark datasets. On CIFAR-10, our method achieves the test accuracy of 81.1 1..1% %c exceeding the best previous result (80.3%). This advantage is maintained on CIFAR-100 (0.79% higher than SoftHebb) and becomes even more pronounced on the more complex Imag-Ima dataset, where our method achieves 40.33% compared to 34. 12% for SoftHebb.
7

---

Table 1:
Compare with existing synaptic plasticity works. denotes results reported on the source paper. it denotes self- eeff-implemented result on our network structure with open source code. All accuracy values are presented in the format mean + standard deviation.
Approaches
CIFAR-10
CIFAR-100
Tiny-ImageNet
D-CSNN [30]
73.7
45.17
14.36
Hard WTA [31]
72.2
32.56
Hard WTA [1]]
74.6
SoftHebb [12]*
80.3
56.0
SoftHebb [12]1
78.86
54.18
34.12
SPHeRe
81.11
0.11
56.79
0.69
40.33 + 0.24
5.3 Analysis Experiments
In this section, we conduct series of analysis experiments to evaluate different aspects of our proposed method. Due to space limitations in the main text, we present selection of key results here. For a more comprehensive evaluation and additional analysis experiments, we refer the reader to Appendix K.H. which provides more analysis experiments and results.
5.3.1 Ablation Study
In this section, we conduct an ablation study on different components of our method to verify their effectiveness to the final performance. Specifically, we decompose our method into the original Oja's Hebbian loss Laja, the simplified Oja's Hebbian loss LSPheR, the orthogonality constraint loss of features Corth and the auxiliary block Φb Then, we verify the unsupervised classification performance on CIFAR-10 with different combinations. The results are summarized in Table 2. Using only Laja can achieve an accuracy of 73.8%, indicating that the original Oja's rule is still effective. worth noting that ablation experiments show that both the orthogonality constraint loss Lonhh and the auxiliary block & can effectively improve the algorithm's performance. However, Lorth requires significant amount of computation, leading to memory overflow when dimension reduction is not performed using é. Although CSPHeRe does not perform as well as the original Coja. its combination with other components (Cunth and σ) is superior to the original Oja loss Laja- Ultimately, with the combination of LSPHeRe. Lonh: and Φ the network achieved 81.18% accuracy on the test set.
Table 2: Accuracy comparison for different loss function combinations
Table 3: Continual Learning results. denotes self-implemented. All accuracy values are pre- sented in the format mean + standard deviation.
Loja
LaMERe
Loch
Accuracy (%)
73.8
Approaches
SpARRC00
Splii-TinylmageNet
76.2
EWC [32]
71.96 0.37
62,87 0,31
76.3
HAT [33]
75.70 0,50
54.971
65.4
OWM [34]
70.891 0.13
78.9
GPM [35]
74.99 + 0.12
66.00 + 0.24
80.7
SoftHebb [12]*
51.1
81.18
SPHeRe
72.724 1.14
63.46:1 0.79
SPHeRe-EWC
76.531 0.64
67.054 1.16
5.3.2 Performance on Continual Learning
Continual learning (CL) is significant challenge in machine learning, which requires the net- work to learn new knowledge sequentially without suffering catastrophic forgetting of previously acquired knowledge. We evaluated our proposed method, SPHeRe, on standard CL benchmarks, Split-CIFARION 100 and Split-Ting Siiii-Tinnlmmeeeet,,, with the results presented in Table 3. Our approach achieves competitive performance, demonstrating its potential in the CL setting. The reason for this phenomenon may be that our unsupervised learning method pays more attention to the general representations of input images, so is more robust to distribution changes encountered between tasks, thus mitigating the phenomenon of knowledge forgetting. Furthermore, when combined with a well-established CL. technique (EWC), the SPHeRe-FWC method significantly improves performance, achieving ofte-of-the-as art results both datasets.

---

5.4 Performance on Transfer Learning
Since our method belongs to unsupervised learning, it can capture some general representations of the underlying data of images, which may be applicable to different target tasks or datasets. In this section, we conduct cross-dataset transfer learning experiments, training the three convolutional layers
dataset (source) through SPHeRe and only training the classification head another dataset (target) to evaluate network performance. The results, detailed in Table 4, show promising transferability. When transferring knowledge from Tiny-ImageNet to CIFAR-10, 10, the model achieved a test accuracy of 80.03%, which is remarkably close to the 81.1 I % accuracy obtained when training SPHeRe directly on CIFAR-I0 (a performance gap of only -1.0%)). Similarly, transferring from CIFAR-10 to Tiny-ImageNet yielded a test accuracy of 37.7%, compared to the 40.33% baseline achieved with direct training on Tiny-ImageNet (a gap of -2633)). Although there is slight decrease in performance compared to training directly on the target dataset, these results indicate that the feature extraction rules learned by SPHeRe on the source dataset remain effective when applied to different target datasets, confirming the method's 's effective transfer learning capability.
Table 4: Cross- boos--Dttsett transfer learning performance of SPHeRe
Transfer Direction
Transfer Learning (%)
Non- Transfer (%)
Cap
(Train/Test)
(SPHeRe alone)
(vs. Non- Non-Transter)
Tiny- ImageNet + CIFAR-10 CIFAR-IÓ - Tiny-ImageNet
9 80.03
81.11
-1.08%
/3.7
40.33
-2.63%
5.4.1 Reconstruction Experiments
In continuous learning and transfer learning experiments, SPHeRe achieved good experimental results, indicating that SPHeRe tends to extract more general information from images. To verify this statement, we designed an experiment to prove whether the image information extracted by SPHeRe can better compress the complete information of the image. We first pretrained both the baseline model and our SPHeRe model on the CIFAR-10 10 dataset. These pretrained models were then used as encoders to transform input images into feature maps. These images could be added with Gaussian noise. Next, we send the feature maps into a decoder, which was trained via backpropagation to reconstruct the original images. More details on the structure, training settings, and more results can be found in the Appendix H. Reconstruction loss (pixel-wise mean square error), calculated as the difference between the original and decoded images, served our evaluation metrie. As shown in Fig. 2, for reconstructing original clean images, the decoder utilizing features from our SPHeRe encoder achieved a remarkably low loss of 3.37 10-3, This outperforms the reconstruction based on features of other methods. At the same time, visually, our method provides the best image restoration, indicating that our method can better compress and retain the complete image information. Furthermore, even under these noisy conditions, the decoder using SPHeRe features maintained an outstanding reconstruction performance, achieving an MSE of just 3.59 x 10-3 effectively reconstructing the original clean image from the noisy input's 's features. Additionally, we notice that the SoftHebb method tends to discard the color information of the image. This may lead to decreased performance in tasks where color information is crucial. These reconstruction results strongly support our hypothesis that SPHeRe's 's unsupervised Hebbian learning mechanism captures more fundamental, complete, and robust visual information compared to both standard backpropagation and the SoftHebb baseline.
Conclusion
This work revisits Hebbian learning's connection to optimal low-dimensional projection via equivalent loss functions. We introduce SPHeRe, a novel unsupervised loss function derived by simplifying the equivalent loss of Oja's rule, designed to minimize structural differences between input and output representations. To enhance feature extraction, SPHeRe incorporates lightweight nonlinear modules and feature orthogonality constraint. SPHeRe achieves state-of-the-an results among unsupervised plasticity methods on multiple classification datasets and demonstrates robustness and generalizability across continual learning, transfer learning, and image reconstruction tasks. Our findings highlight the fcasibility and potential of Hebbian learning in modern deep learning.

---

Softilebb (1 opoch)
SoftHlebb (100 opoch)
Qurs
Original Image
loss: 7.79
loss: 12.75
loss: 4.75
loss: 3.37
Noise Image
loss: 10,00
loss: 19.75
loss: 17.67
loss: 5.11
loss: 3.59
Figure 2: A sample of reconstruction results. The loss values are scaled by the factor of 10
However, certain limitations invite further investigation and delineate promising avenues for future work. One notable observation from our experiments that performance gains slightly diminish as the number of convolutional layers increases. This suggests that noise accumulates the layer increases and that purely local unsupervised SPHeRe learning lacks sufficient contextual signals to learn to eliminate noise. This challenge resonates with biological reality, where neural circuits employ sophisticated synergy between local synaptic plasticity and global neuromodulatory signals. SPHeRe currently emphasizes the former. Therefore, compelling next step involves exploring the integration of multiscale regulatory or contextual signals, perhaps inspired by biological reward/punishment pathways or attention mechanisms. We believe that pursuing such multiscale approaches will not only advance the capabilities and scalability of Hebbian and other biologically inspired algorithms within modern deep learning frameworks but also offer valuable computational perspectives that could deepen our understanding of biological neural learning processes.
7
Acknowledgment
This project is supported by NSFC Key Program (No. 62236009). Youth Program (No. 62506065). Postdoctoral Fellowship Program of CPSF under Grant Number GZC20251045.
References
[1] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[2] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[3] Francis Crick. The recent excitement about neural networks. Nature, 337(6203): 1989.
[4] Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synap- tic feedback weights support error backpropagation for deep learning. Nature communications, 7(1): 2016.
[5] Donald Olding Hebb. The organization of behavior: A neuropsychologica theory. Psychology press, 2005.
[6] Elie L Bienenstock, Leon N Cooper, and Paul Munro. Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. Journal of Neuroscience, 2(1):32-44) 4332-8,117 1982.
[7] Erkki Oja. Simplified neuron model principal component analyzer. Journal of mahematical biology, 15:267-273, 1982.
10

---

[8] Cengiz Pehlevan, Tao Hu, and Dmitri B Chklovskii. A baabiian/antihhbbbbee neural network for linear subspace learning: A derivation from multichmensional scaling of streaming data. Neural computation, 27(7):1461-1499 2015.
[9] Cengiz Pehlevan, Anirvan M Sengupta, and Dmitri B Chklovskii. Why do similarity matching objectives lead to hebbian/anti habbian/antihhebbian networks? Neural computation. 30(1):84-1) 124, 2017.
[10] Dina Obeid, Hugo Ramambason, and Cengiz Pehlevan. Structured and deep similarity matching via structured and deep hebbian networks. Advances in neural information processing systems, 32, 2019.
[11] Leopold Grinberg, John Hopfield, and Dmitry Krotov. Local unsupervised learning for image analysis. arXiv preprint arXiv: 090.:09990....93, 2019.
[12] Adrien Journé, Hector Garcia Rodrigue7, Qinghai Guo, and Timoleon Moraitis. Hebbian deep learning without feedback. arXiv preprint 2rXiv:2209.11883 2022.
[13] Sarah C. Simmons, Greg G. Grecco, Brady K. Atwood, and Fereshteh S. Nugent. Effects of prenatal opioid exposure on synaptic adaptations and behaviors across development. Neurophar- macology, 222:109312 109312, 2023.
[14] Guo-qiang Bi and Mu-ming Poo. Synaptic modifications in cultured hippocampal neurons: de- pendence on spike timing, synaptic strength, and postsynaptic cell type. Journal of neuroscience, 18644):1044410472, 1998.
[15] Johannes Zierenberg, Jens Wilting, and Viola Priesemann. Homeostatic plasticity and external input shape neural network dynamics. Physical Review X. 8(3):031018, 2018.
[16] Marina Chistiakova, Nicholas M Bannon, Maxim Bazhenov, and Maxim Volgushev. Heterosy- naptic plasticity: multiple mechanisms and multiple roles. The Neuroscientist, 20(5):483-49) 2014.
[17] Wickliffe C Abraham and Mark F Bear. Metaplasticity: the plasticity of synaptic plasticity. Trends in neurosciences, 126-113,,,330, 1996.
[18] Priyadarshini Panda and Kaushik Roy. Learning to generate sequences with combination of hebbian and non-hebbian plasticity in recurrent spiking neural networks. Frontiers in neuroscience, 1 11:693, 2017.
[19] Stuart Lloyd. Least squares quantization in pem. IEEE transuctions information theory, 28(2):129-133) 1982.
[20] Harold Hotelling. Analysis a complex of statistical variables into principal components.
Journal of educational psychology, 24(6):417, 1933.
[21] Yanbei Chen, Massimiliano Mancini, Xiatian Zhu, and Zeynep Akata. Semi-supervised and unsupervised deep visual learning: A survey. IEEE transactions on pattern analysis and machine intelligence, 46(3):1327-1347 2022.
[22] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PmLR, 2020.
[23] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21288 2020.
[24] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15750-15758, 2021.
[25] Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 507 2006.

---

[26] Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013.
[27] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[28] Arild Nøkland and Lars Hiller Eidnes. Training neural networks with local error signals. In International conference on machine learning, pages 1839-4850 PMLR, 2019.
[29] Da Kuang. Chris Ding, and Haesun Park. Symmetric nonnegative matrix factorization for graph clustering. In Proceedings of the 2012 SIAM international conference on data mining, pages 106-117. SIAM, 2012.
[30] Bonifaz Stuhr and Jürgen Brauer. Csnns: Unsupervised, backpropagation- free convolutional free neural networks for representation learning. In 2019 18th IEEE International Conference Machine Learning And Applications (ICMLA), pages 1613-1620. IEEE, 2019.
[31] Thomas Miconi. Hebbian learning with gradients: Hebbian convolutional neural networks with modern deep learning frameworks. arXiv preprint arXiv:2107.01729, 2021.
[32] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Gtrabska-Barwinskee et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 (1((3))352133356 2017.
[33] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International conference on machine learning. pages 4548 4557. PMLR, 2018.
[34] Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 1(8):364-37,, 2019.
[35] Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. arXiv preprint 203iv:203.0097 2021.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have included main claims in the abstract and introduction, and we have justified this claims in the experiment section, providing detailed results and comparison other methods.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed about limitations in the last paragraph the Conclusion
section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
13

---

Answer: [Yes]
Justification: We have provided complete and correct proof for the only Lemma 4.2. 1 in the Appendix. The Lemma is provided with a full set of assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided the full network structure in 5.1 Experimental Setup, and the dataset we used (CIFARIO 10 CIFAR 100 Tiny-Imagenet) is open sourced, so reproducibility is guaranteed. We also provide the detailed information of the reconstruction experiment in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We will provide code with full instruction to reproduce main experimen- tal results. The dataset used (CIFARIO CIFAR100 Tiny-ImageNet) is opensourced, so reproducibility is guaranteed.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https:/ htttps://nips.coo cc/ httpc//ggiide//ooeeeuee
wwwiiissssiiiiiiioooooii for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
cttppsscccpbbpccccccecccc wwwpe//////////eeeeee /CodeSubmieeeeee onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
We have specified these details in 5.1 Experimental Sctup. For data splits and Justification: hyperparameters, choose the universal standard setting.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have included statistical significance in our main experiments. We also have included how error bars are calculated, either standard deviation or stand error the mean.
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have specified compute resources details in the Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tsttpa:/nerrre http///eeuuup.peeeeeeeeeee hwtp//ubbl....iiccssiiiieeeeeeee
Answer: [Yes]
Justification: We have carefully checked the Ethics Guidelines and are sure that we have not violated any Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiciio))
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We believe our method, a unsupervised biologically inspired layer-wise training method, will not have any significant societal impact, neither positive nor negative.
16

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
These are safety concerns for this paper far know.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We only used CIFARIO, CIFARI00 and and we cite them properly. For models and codes provided by other papers, we also cite them properly.
dataset in our work,
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
17

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi whwoee com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not introduce any new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve any crowdsoureing experiments, nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The core method development in this research does not involve any LLM
usage.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: : //neruurpp.... www.///ooeeence///00000/ for what should or should not be described.
19

---

A Proof of Lemma 4.2.1
Lemma 4.2.1. Let X BRXN be the input data matrix and let XW Bx be the output, W E RNNM be the weight matrix, and M < N. Consider the loss function L = -|Y -XX 112 Minimizing with respect to W yields an optimal output Y given by Y XVM. where Vas consists of the first M right singular vectors of X. This Y* represents the projection of the rows of X onto the M M-dimensional principal subspace, and Y=y*T the best rank-M approximation of XX in the Frobenius norm.
The proof of the above theorem is similar to the proof of the Eckart- Bicart-Young--a theorem, and its proof is given below:
Proof. The objective is minimize the loss (unction:C(W) ||YYT -XX || Substituting
Y XW, we get:
((W) = [[(XW)(WW) - -x -XX
|XWW X -xx
Let P WWT. The matrix P N is semidefinite positive symmetric (SPD). The rank of P is bounded by the dimensions of W: vank(P) = rank(W)) S min(N, M) = M, since M < N. The optimization problem can be reformulated to find an SPD matrix that minimizes
((P) ||XXXT 1
First, we can perform a Singular Value Decomposition (SVD) of the input matrix X that let
X = =UUVV,
(8)
where:
U E ER Bx is an orthogonal matrix TU = IB).
E NRNN an orthogonal matrix (VTV = Ty). The columns 01 uN, of V are the right singular vectors of X.
Σ E R BxN is rectangular diagonal matrix containing the singular values 2 02 2 2 o. on main diagonal, where r = rank(X).
We can express the terms in Eqn. 7 using the Eqn. 8:
XXT = R Bx diagonal matrix with diagonal entries of of. 0f,00..,,
= UEVTVETUT = UUT Let A = ΣΣT
XPXT- = (UEVT)P(UEVII = UEV UT
Then Eqn. 7 is change to:
((P) JUEV PVV UT UAUT 12 1 MV | PVST - All2:
1
where the second equals sign due the unitary invariance property of the Frobenius norm (i.e.. [|QAZ|| || for orthogonal matrices Q, 2). Let Q == V PV. Since V is orthogonal and P symmetric SPD with rank(P) < M, Q is also symmetric PSD with rank(Q)) rank(P) S M. The optimization problem transforms into finding symmetric PSD matrix Q NxN with rank(Q) < M that minimizes:
((Q) = -IQE Alp..
(9)
Let Σ, diag(o1..... or). We can appropriately partition E and Q. Without loss of generality regarding the dimensions B, N. the product takes the form:
EQΣ
Σ,
Q
Er
0
E,
0
(10)
0
0
BxN
NxB
BxB
20

---

Σ
0
where Qn is the top-left r x principal submatrix of Q. The matrix A EET The Eqn. 9 can be simplified to:
0 BxB
Let An = =EQQuE Since o, > 0 for i 1...., r, E, is invertible. Thus, -1==; (11)
((Q) - - E
A
An must be symmetric SPD and rank(A11) rank(Q11)) I M. The problem is equivalent finding the best rank-M symmetric SPD approximation A1 to the diagonal matrix Σ = _diaggo....... af in the Frobenius norm. And the optimal rank-k approximation of diagonal matrix (in Frobenius norm) is obtained by keeping the k diagonal entries with the largest magnitudes and setting the others zero. Since of 2 ... of > 0 and An must be SPD, the best rank-M symmetric SPD approximation Aji to DT is obtained by keeping the first M largest diagonal entries (assuming 2 M, otherwise keep all r): Ai diag(o ,.,, )), The corresponding optimal Qi is: Qi = Σ, Aii - = diag(1,.... ................................ 1,0,... Q 0
O)rx,, where there are M ones. The optimal N x N
matrix Q' can be constructed as: Q" = ( 0 NxN
("0 0)
Q" is symmetric SPD
=
0 NxN
and rank(Q") = M. Then, we have
p v (for 0) N NVT
IM
T
0)
VI (V-M)
[VM, VN M]
-VMLMV M VM VM
Pa is the projection matrix onto the subspace spanned by the first M right singular vectors of X. We need to find W* E RNXM such that W "W"") p VM M- A valid solution is W VM--
(Other solutions V = Vsr for any M x M orthogonal matrix O exist, but they lead to the same TY As result, the optimal output Y*:
===XW
XVM UEV VM UEBxM UM
(12)
So the optimal output Y = XVM calculates the coordinates of the rows of X projected onto this principal subspace. Furthermore, the product Y " is: (+
(M)MMM Em)
o which is the best rank-M approximation of XXT = UAUT = -E= uu in the N Frobenius norm. And the minimal loss Lmin > of.
B SPHeRe Loss in Nonliner Condition
In practice, instead of directly using the linear projection, applied nonlinear transformation to X, we use ff to represent the nonlinear transformation. So the loss function is change to:
L ())
xx 1PE:
(13)
where the parameters O, of universal approximator / (neural network). When 2 = S(XW;0))
R BBX (M' < N). through Lemma 4.2.1, we can obtain the optimal Z V that minimizes the loss L. Since ff is universal approximator, there theoretically exists set of parameters 07 and W* such that / (XW*: 07) XVM Therefore, during the training process, the output Z of the auxiliary branch will gradually approach the dimensionality reduction projection of X: X XVM,, and the output Y of the main branch layer will gradually summarize and extract the features of the input X through this training process.
To verify our viewpoint, we conducted the corresponding experiments. We trained two different network branches: 1) linear branch, where f is a linear mapping with parameter W; 2) nonlinear branch, where is a neural network with 3 layers. Both branches receive the exact same input data X, and we train them through Eqn. 13. During the training process, we record the output features different training epochs and use the centered kernel alignment (CKA) similarity to quantify and compare the similarity between the output features between two different branches (Fig. 3 A). From
21

---

the CKA similarity curve in Fig. B, it can be seen that, after only 20 training epochs, the CKA similarity exceeded 0.90 and tended stabilize at a high level as the training progressed for all datasets. This indicates that regardless of whether an explicit nonlinear transformation f is used, the SPHeRe loss drives the network to learn very similar feature representations.
Secondly, to further explore the structure of the feature space, we performed a single value decompo- sition (SVD) on the feature output of the branches. We focus the 36 main components (that is, right singular vectors) and calculate the similarity between the feature vectors obtained by linear SPHeRe optimization (Linear f) and those obtained by nonlinear SPHeRe optimization (Nonlinear f). We visualized these similarities the similarity matrix shows very high similarity, especially for the first few principal components with lower indices (the first 20). This indicates that the most important principal directions (directions explaining the most variance) in the feature space learned by the two optimization methods are highly aligned and consistent. Principal components with higher indices (the last 10) do not strictly follow the maximum similarity on the diagonal, but most singular vectors have relatively similar corresponding vectors.
36 x 36 similarity matrix (Fig. 3 C). The diagonal region of
A
B
ef
0:3
Col Gealary
0.3
CEAR-10
Hot Dear Dhafict Optndts Strafa Ltaafa
130
Figure 3: (A) The experiment's network architecture; (B) CKA similarity comparison of the linear and nonlinear branches; (C) SVD component correlation between the linear and nonlinear branches.
C Ablation study Auxiliary Block Architecture
To further evaluate the function of auxiliary blocks, we conduct an ablation study to the auxiliary block. We systematically varied two key components of this design:
1. Depth: We changed the number of convolutional layers within o (0, 1, or 2 layers).
2. Projection Dimension: We changed the output dimension of the final FC projection layer (128, 256, or 512).
The results of this ablation study, measured by test accuracy on CIFAR-10, are summarized in Table and Table 6.
Table 5: Ablation on the number of convolutional layers in ø
Number of Cony Layers CIFAR-10 Test Accuracy (%)
78.65
(Default) 2
81.11
79.93
Table 6: Ablation on the projection dimension of the FC layer in d
Projection Dimension CIFAR-10 Test Accuracy (%)
128
80.50
256 (Default)
81.11
512
81.18
22

---

D Comparison on Transfer Learning Performances
To provide strong baseline for comparison, conducted identical experiments using features from a standard Backpropagation (BP) trained model and the current state-of-the-art Hebbian method, SoftHebb [12].
Table 7: Cross-Datasee transfer learning performance comparison. Non-Transfer refers to the accuracy achieved by each method when trained and tested on the target dataset (results from Table 1). Cap is the difference between Transfer Learning (Test) and Non- Transfer accuracies.
Method
Transfer Direction
Transfer Learning (%)
Non- Non-Transfer (%)
(Train)
(Test)
BP
Tiny-ImageNet + CIFAR-10 10
100.0
81.7
88.7
CIFAR-10 + Tmny-ImageNet
99.9
39.2
45.3
SoftHebb
Tiny-ImageNet CIFAR-10
78.0
74.84
78,86
CIFAR-10 + Tiny-ImageNet
60.0
33.26
34.12
SPHeRe (Ours)
Tiny-ImageNet - CIFAR-10
97.3
80.03
81.11
CIFAR-I0 + Tiny-ImageNet
99.9
37.7
40.33
E
Comparison with Backpropagation-Basee Unsupervised//Sefssupervsss Learning
To further contextualize the performance of SPHeRe within the broader landscape of unsupervised learning, compare against modern superviu learning (SSL) method, SimCLR [? J. This comparison is insightful yet requires careful interpretation due fundamental differences in learning philosophy and optimal network architecture.
Modern SSL methods like SimCLR leverage strong, explicit pseudo-supervisory signals derived from sophisticated data augmentations to learn invariant and discriminative features. In contrast, SPHeRe operates on different principle: it uses a simple, unsupervised objective (LSPheRe) that aims to preserve the intrinsic structural information of the input without relying on such explicit invariance-inducing signals.
An empirical investigation was conducted on CIFAR-10 using an official SimCLR implementation. Both methods were evaluated under their own optimal hyperparameter settings. The results are summarized in Table 8.
Table 8: Comparison between SPHeRe and SimCLR under different training configurations CIFAR-10. Best Settings refers to the optimal hyperparameters for each respective method. The model structure follows SPHeRe optimal setting.
Method
Configuration
Train Acc. (%)
Test Acc. (%)
SimCLR
SimCLR Best Settings
87.83
71.94
SPHeRe
SimCLR Best Settings
75.03
70.61
SimCLR SPHeRe Best Settings
56.36
49.01
SPHeRe
SPHeRe Best Settings
97.91
81.13
F
Application to Deeper Architectures
key question for any unsupervised learning method is its scalability modern, deep architectures. As noted in the conclusion, our initial observation indicated that the performance gains of purely local SPHeRe learning diminish as network depth increases, due the accumulation representation noise across layers. To investigate this quantitatively and explore practical application, we designed an experiment where a SPHeRe-trained module serves as biologically-inspinn "stem" for standard deep ResNet.
23

---

To evaluate the potential of SPHeRe initial feature extractor, we conducted series of ex- periments integrating it with a standard ResNet- architecture on the CIFAR 10 dataset. We first established baseline by training standard ResNet- 88 model end-to-end with backpropagation. We then pre-trained the first convolutional layer of the ResNet using the SPHeRe objective. This pre-trained stem was integrated into the ResNet-18 88 in two ways: 1) by freezing the uPHeRe-trained stem parameters and only training the subsequent ResNet layers, and 2) by using the SPHeRe-trained stem as an initialization and fine-tuning the entire network. The results of these experiments are summarized below.
Table 9: Performance of ResNet-18 on CIFAR-10 with different training strategies for the initial stem layers.
Model & Training Strategy
Test Accuracy (%)
ResNet-18 (Standard Backpropagation)
79.46
SPHeRe Stem (frozen) + ResNet-18
78.78
SPHeRe Stem (fine-tuned) + ResNet-18
79.93
G
Comparison to related unsupervised methods
This section provides comparative overview SPHeRe and several related unsupervised learning and matrix factorization methods. Table 10 summarizes the typical optimization objectives and primary goals for SPHeRe along with typical unsupervised methods such as PCA, Kernel PCA, MDS, SymNMF, and Laplacian eigenmaps. This comparison helps to situate SPHeRe within the broader landscape of dimensionality reduction and structure preservation algorithms, highlighting the similarities and distinctions in their underlying mathematical formulations and objectives.
H Supplementary Information on the Reconstruction Autoencoder
Fig. 4 shows the framework of the image reconstruction experiments. The encoder shares the same structure as the unsupervised learning setting. The decoder has upsampling Convolution layers ([512, 256, 3} channels) with kernel size stride 2, padding I and out padding 1. To better visualize the result of the reconstruction, the decoder is trained on the specific image for 100 iterations. We use pixel-wise mean square error quantify the difference between the reconstructed image and the original image.
To guarantee generalization, also tested the whole CIFAR10 testing dataset, for which our method also shows superiority. In this experiment, we trained the decoder on the training dataset for 10 epochs, then tested on the testing dataset. The result is shown in Table. 11.
Noise
Stage
Pretrain with SPHeRe on CIFAR Freeze Parameters
Stage
Train with
Reconstructed Image
Original
Loss
Image
Encoder
Decoder
Element-wise Addition
Feature Map
Forward
Original Image
Detach Gradient
Gradient Backward
Figure Auto-Encoder framework.

---

Table 10: Comparison of SPHeRe with Related Unsupervised Learning Matrix Factorization
Methods
Method
Typical Optimization Objective
Primary Goal
Equation
SPHeRe
minw; | Kx11
Preserve input data structure (pair- wise relationships defined by xx) in the output representation 2. Find optimal low-dimensional projection.
= 6(Y,0) = W)
Kx XXT
Kz = 22T
PCA
maxw Tr(WTX XTXW)
Find orthogonal directions (princi- pal components) capturing maxi- mum variance. Dimensionality re- duction preserving global covari- ance structure.
S.L. W WI
Kernel PCA
maxy Tr(VTCoV)
Find principal components in a non- linear feature space defined by the kernel k. Preserve kernel-based structure.
S.t. V II
C 1 LT
)(2,) o(z;) 1
Eラ= 6(2,)
N
MDS
miny Ei, Wey (Y) )zz)2
Find a lww-dimensional embedding Y that preserves the given pairwise distances dissimilarities d.j.
(Y) is distance in embedding Y
6, is original distance.
SymNMF
min A H
Find non-negative low-rank matrix H such that H H approximates Often used for clustering.
a given symmetric matrix H must be non-negative
Laplacian Eigenmaps
miny Tr(Y LY) s.t. Y DY =I
Preserve local neighborhood struc- ture in low-dimensions embed- ding. Dimensionality reduction for manifold data.
(miny Eig Wiglly: ,3 D W
Table 11: Reconstruction result on CIFAR10
SoftHebb(100 epoch)
SoftHebb
BP
SPHeRe
Loss
1.81
2.39
1.68
1.62
And we provide more results of the image reconstruction tasks in Fig. 5 to support the experimental results and the analysis in the main text.
Note that the reconstruction performance varies across different random initializations. To account for this variability, we repeated the experiment with 10 different random seeds and report the mean reconstruction loss along with the standard error of the mean (SEM) in Fig. 6.
I t-SNE Visualization
To qualitatively assess the feature representations learned by the three convolutional layers trained by our method, we visualized the output features for the CIFAR-10 dataset using t-SNE. Fig. 7 presents the (-SNE embeddings for both the training and test set samples, comparing the feature space before training (random weights) and after training. As expected, after training, distinct clusters begin to emerge in the feature space for both training and test sets. This demonstrates that the proposed unsupervised Hebbian learning process successfully organizes the feature representations based on the underlying structure within the CIFAR-I0 data, and these learned structures generalize well from the training set to the unseen test set. Consistent with observations made for SoftHebb [12], our SPHeRe approach shows a clear propensity to separate classes characterized by relatively well-defined shapes and edges, such airplane, automobile, ship and truck. However, it is more challenging to separate
25

---

SoftHebb (1 epoch)
SoftHebb (100 epoch)
Ours
Original Image
loss: 16.75
loss: 9.37
loss: 2.68
loss: 1.99
Noise Image
loss:
loss: 44.81
loss: 9.43
loss: 3.60
loss: 2.37
Original Image
loss: 58.87
loss: 9.03
loss: 2.56
loss: 1.93
Noise Image
loss: 10
loss: 64.06
loss: 12.39
loss: 4,29
loss: 2.34
Figure 5: Reconstruction experiment results. The loss values are scaled by factor of 103.
Reconstruction loss
100
Test Conditions
Reconstruction Loss (MSE) x10*
2
SoftHebb
SoftHebby 100 epochs)
BP
SPHeRe
Models
Figure 6: Reconstruction loss across 10 random initializations (mean + SEM), The y-axis is log- scaled, and the loss values are scaled by factor of 103.
26

---

classes representing animals that often feature fur, feathers, and more variable textures, such as birds, cats, deer, and horses. These classes exhibit considerable overlap in the t-SNE embedding, suggesting that the features captured by SPHeRe, while effective for edge-based discrimination, may be less sensitive finer textural details or complex shape variations.
Trainset
Testset
automeblle
deer
dog frog
Random weights
Trained weights
Random weights
Trained weights
SPHeRo
Figure 7: The t-SNE results CIFAR-10 dataset.
Experiments with Different Activation Functions
To validate the impact of the selection of activation functions on the performance of our model, we conduct experiments considering several common activation functions: ReLU, Leaky-ReLU, tanh, sigmoid, and binary step function. The results are summarized in Table. 12. The performance differences achieved by different activation functions vary significantly, indicating that our approach still influenced by the choice of activation function. The training set accuracy of ReLU, Leaky- ReL.U, and tanh is above 95%, indicating that they are all good choices for the activation function. Among them, Leaky-Rel.U performs the best, possibly because retains the most information before activation while introducing nonlinearity.
Table 12: Comparison for different activation functions
ReLU
Leaky-ReLU
tanh
sigmoid
binary
Train accuracy 96.9
97.7
96.6
74.3
83.9
Test accuracy
79.9
81.2
78.5
72.7
71.2
K KNN Experiment
To better monitor the training process, we replace the final classifier with K-Nearest Neighbors (KNN). As shown in Figure 8, the clustering process converges quickly during training. However, after convergence, slight decrease in accuracy is observed. We assume that this drop is mainly due to overfitting on the training dataset.
L Spiking Neural Network Adaption
To further demonstrate the biologically plausible nature of SPHeRe, demonstrate that method could work decently on Spiking Neural Networks. The overall network structure is the same with maaky-Integraae-and-Fiin (LIFF) dynamic (with timesteps). We
CNN setting, activation function use a surrogate gradient, as spikes are non-differentiable The surrogate function we choose the Piecewise- Exxprietaixxo surrogate function, which can be derived as: g'(x) 3 -
We verify our method and SoftHebb, comparing with random initial weight. The results in Table 13 show that our method can reach decent accuracy, while SoftHebb training has a negative effect on the final result. This may be because softhebb directly implements the Hebbian rule operation on the gradient, without considering the impact different activation functions on the result distribution, which is also mentioned in the Appendix of the Softhebb paper.
27

---

Accuracy
0.60
0.62
0.60
0.5
0.6
Figure 8: KNN validation accuracy
Table 13: Spiking Neural Network comparison
Approach Random weight
SoftHebb
SPHeRe
Accuracy
60.8
55.8
75.28
M Experiments compute resources
All the experiments are carried out on one Nvidia RTX 3090 Graphic Card with 24GB VRAM. The detailed information in below:
Table 14: Computational resources used for experiments. Training was conducted on an NVIDIA
RTX 3090 GPU (24GB VRAM) with batch size of 128 and dataloader workers set to 2.
Dataset
Training Time GPU VRAM Usage
CPU Workers
Total Epochs
CIFAR-10
30 min
2,5 GB
2
100
CIFAR- 100
~00 min
~25 GB
2
100
Imageem
~3. hours
8.2 GB
100
28


Reviews:
[{"review_id": "av5pETJIKD", "reviewer": "Reviewer_b3Ah", "summary": "This paper provides a similarity-based representation learning approach. This similarity measure allows the 'inputs' of a module to be nonlinearly transformed before applying the similarity measure. The authors also add some orthogonality constraints on intermediate-layer neurons. This approach with multiple similarity blocks/layers generates representations that are good for supervised learning, but also do well for continual learning and transfer learning. The representations learned also could be used for an auto-encoder.", "questions": "Clarification of how the labels were used in their evaluation would help a broader set of readers, some of whom may come from computational neuroscience.\n\nContrasting with previous work would place the work in the proper context. In that vein, could the authors comment on how their training relates to Hebbian learning as understood in neurobiology?", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Finding a similarity-based multilayer network where different layers learn more abstract features has been a goal for many researchers, especially those interested in biologically plausible networks. It seems that the authors have made some progress there. Compared to some other unsupervised methods inspired by Hebbian learning, the authors claim that they have better performance.\n\nOne major complaint I have is that the paper is written as if there has been not much work previously done on similarity-based objectives and Hebbian learning. It completely ignores works like Pehlevan and Chklovskii, Asilomar, 2014, Pehlevan, Hu, and Chklovskii, Neural Computation, 2015; Pehlevan and Chklovskii, NeurIPS, 2015; Pehlevan, Sengupta and Chklovskii, Neural Computation, 2017; Obeid, Ramambason, and Pehlevan, NeurIPS, 2019. Some of these previous works were mentioned in the Nokland and Eidnes paper the authors cite. Contrasting with these works gives a better appreciation of authors' innovations. Also, some of the references to arXiv papers should be changed to their published version. For example, Saha et al is in ICLR 2021 and the Journe et al paper is in ICLR 2023.\n\nIt might also be helpful to be explicit about how the labels are being used. My impression from reading the paper is that representation learning does not use labels at all, and I imagine only later some shallow supervised learning is done using the learned representation. Some similarity-based methods like HSIC Bottleneck (Ma et al, AAAI, 2020; Pogodin and Latham, NeurIPS, 2020), use both inputs and labels while training networks. Hence a clear description of the method may help.", "was_revised": true, "final_justification": "I believe the authors have tried to change their manuscript and responded to the reviews constructively. I think this work is an interesting starting point for multilayer similarity-based representation learning, even if it is not yet fully Hebbian. I don't know whether some day we will get deep models with hierarchical feature learning, with Hebbian rules, or not. However, it as a serious effort in that direction. So, I am upgrading my score.", "rebuttal": "**Comment 1:**\n\nOne major complaint I have is that the paper is written as if there has been not much work previously done on similarity-based objectives and Hebbian learning. It completely ignores works like Pehlevan and Chklovskii, Asilomar, 2014, Pehlevan, Hu, and Chklovskii, Neural Computation, 2015; Pehlevan and Chklovskii, NeurIPS, 2015; Pehlevan, Sengupta and Chklovskii, Neural Computation, 2017; Obeid, Ramambason, and Pehlevan, NeurIPS, 2019. Some of these previous works were mentioned in the Nokland and Eidnes paper the authors cite. Contrasting with these works gives a better appreciation of authors' innovations. Also, some of the references to arXiv papers should be changed to their published version. For example, Saha et al is in ICLR 2021 and the Journe et al paper is in ICLR 2023.\n\n**Response to comment 1:**\n\nWe are very grateful to the reviewer for pointing out the series of foundational works by Pehlevan, Chklovskii, and their collaborators. We fully agree that placing our work in this important academic context for discussion is the key to enhancing the quality of the paper. Our work shares a similar mathematical starting point with these foundational works: optimizing a similarity matching objective of the form $||X^\\top X - Y^\\top Y||^2_F$. However, the path we take to solve the core challenge of \"how to effectively apply this objective to deep nonlinear networks\" is distinctly different, which precisely highlights the unique contribution of our SPHeRe method. \n\nPehlevan, Chklovskii, Obeid (2019), et al.'s path: They elegantly derived local learning rules with Hebbian/anti-Hebbian forms starting from similarity matching objectives through methods such as min-max dual optimization and extended it to deep networks. As described by Obeid et al., their deep model theoretically relies on feedback connections to achieve credit assignment across layers. \n\nOur path: We start from the equivalent loss function of the Hebbian rule (specifically Oja's rule) and simplify it to achieve the same objective. However, to apply it to deep networks, we propose a different, purely feedforward architecture. We introduce a decoupled lightweight auxiliary module ($\\phi$) that projects the high-dimensional features of the backbone network into a low-dimensional space and calculates similarity loss in this low-dimensional space. This design of \"main-pathway learning, side-pathway supervision\" structurally avoids the need for explicit feedback paths, making the model more modular and seamlessly integrable into mainstream deep learning frameworks. \n\nIn the revised manuscript, we will clearly articulate this \"different paths lead to the same goal\" derivation process in the methodology section and compare in depth the differences between our feedforward block-level training scheme and feedback-dependent schemes to more accurately position our work. Meanwhile, we will re-examine the issues regarding the version of the cited papers you mentioned in the revised manuscript.\n\n\n**Comment 2:**\n\nClarification of how the labels were used in their evaluation would help a broader set of readers, some of whom may come from computational neuroscience.\n\n**Response to comment 2:**\n\nThank you for your insightful feedback, and we agree that clarifying our evaluation methodology is crucial for a broad readership, particularly those with a background in computational neuroscience. We apologize if our initial description was not sufficiently clear.  To clarify, the core of our method, SPHeRe, is entirely unsupervised. The feature learning process in the convolutional layers does not use any ground-truth labels. The training of these layers relies solely on the SPHeRe loss, which aims to preserve the structural information from the input data itself, aligning with biologically plausible, local learning rules. We added a fully connected layer as a classifier after the convolutional layers, and only this last linear layer is trained in a supervised manner, using the standard cross-entropy loss function and image labels. We will revise the experimental setup in our manuscript to explicitly and clearly describe this two-stage training and evaluation process to prevent any ambiguity.\n\n**Comment 3:**\n\nContrasting with previous work would place the work in the proper context. In that vein, could the authors comment on how their training relates to Hebbian learning as understood in neurobiology?\n\n\n**Response to comment 3:**\n\nThank you for this insightful question. We agree that this is a crucial point. We position SPHeRe not as a literal, synaptic-level copy of neural circuits, but as a functional and computational model that abstracts and implements key principles of biological learning. Instead of viewing plausibility as a binary \"yes/no\" state, we see our work as a step forward from end-to-end backpropagation. At its core, our learning rule retains the fundamental associativity of Hebbian learning (\"fire together, wire together\"). However, our main contribution lies in addressing the inherent instability of classical Hebbian learning by incorporating computational analogs of more advanced neural regulation mechanisms:\n\nOur core loss, $L_{SPHeRe}$​, is an implementation of the Efficient Coding Hypothesis. As our lemma shows, in the linear case, this objective is equivalent to PCA, a classic redundancy reduction strategy. Furthermore, based on the pioneering work of Pehlevan and Chklovskii et al., this objective can also be understood as a predictive similarity matching, and the entire optimization process can be decomposed into fully local Hebbian and anti-Hebbian update rules at each synapse. \n\nThe orthogonality constraint, $L_{orth}$​, computationally mirrors the function of lateral inhibition, promoting competition among neurons to learn efficient and decorrelated features.\n\nThe auxiliary module ($\\phi$) serves as an abstraction of local error circuits, such as dendritic compartments, which are theorized to guide synaptic plasticity locally without requiring the symmetric weights or global error signals that challenge standard backpropagation.\n\nWe acknowledge that batch-level computations are a computational abstraction. We view a \"batch\" as a set of stimuli arriving in a short time window, where the computation estimates the input statistics that a neural circuit might capture through biophysical means. \n\nIn summary, our goal is to model the functional objectives of biological learning, and we believe SPHeRe makes a significant step towards biological plausibility by modeling the functional goals—associativity, homeostasis, and competition—of neural learning systems."}, {"review_id": "eod2S8Q5HC", "reviewer": "Reviewer_RoRs", "summary": "This paper rethinks the Hebbian learning rule and proposes an improved unsupervised learning method called SPHeRe. SPHeRe simplifies the equivalent loss function of the Hebbian OJa’s rule and introduces auxiliary blocks and orthogonality constraints. The paper has proved that optimizing the proposed loss function is equivalent to finding the best low-dimensional projection of the input data. SPHeRe achieves the current best classification accuracy among unsupervised synaptic plasticity methods and performs well in continual learning and transfer learning.", "questions": "1.The target of the orthogonal loss in the paper is applied to the auxiliary feature Z. Is this effective for the high-dimensional feature Y?\n2.Can the SPHeRe method be combined with self-supervised learning to achieve better results?", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strength\nThe paper build a bridge between the Hebbian rule and a clear and optimizable loss function. The proposed SPHeRe loss function is simple but has explicit mathematical significance (seeking the optimal low-dimensional projection). This provides a solid theoretical foundation and development perspective for Hebbian-like rules that originally lacked clear optimization objectives. \n\nWeakness\n1.The method proposed in the paper reduces the locality constraints of Hebbian-like rules, where its learning rule depends on the introduced auxiliary blocks for gradient backpropagation to update the backbone network layers. Although SPHeRe is trained layer by layer on the backbone network, it actually uses backpropagation at the block level.\n2.The core theory of the paper is derived from linear transformations, but it has been extended to nonlinear networks. The authors argue that due to the universal approximation ability of neural networks, they have the capacity to learn the theoretically optimal solution. However, even though the authors have empirical validated this through experiments in the appendix, it does not constitute a strict theoretical guarantee. Moreover, the introduction of the auxiliary network has improved final performance, possibly allowing the network to learn more powerful features. \n3.Table 2 shows that using L_{SPHeRe} alone is less effective than the original L_{oja} loss, and the paper does not adequately discuss and explain this phenomenon. \n4.The ablation experiments in the paper demonstrate the effectiveness of the auxiliary block, but the authors have not analyze in depth how its structure affects the results.  \n5.In the caption of Figure 2, “10^3” should be “10^{-3}”.", "was_revised": true, "final_justification": "Thanks for response.", "rebuttal": "**Comment 1:**\n\nThe method proposed in the paper reduces the locality constraints of Hebbian-like rules, where its learning rule depends on the introduced auxiliary blocks for gradient backpropagation to update the backbone network layers. Although SPHeRe is trained layer by layer on the backbone network, it actually uses backpropagation at the block level.\n\n**Response to comment 1:**\n\nWe thank the reviewer for this precise observation, which is entirely correct. SPHeRe indeed uses backpropagation for credit assignment within each block, and its claim to locality is at the block level, not the individual synaptic level. This design is a deliberate and pragmatic choice. Our primary goal is to apply Hebbian principles—specifically, preserving input-output structural similarity—to train modern, deep, non-linear networks effectively. While classical Hebbian rules are synaptically local, they lack a clear mechanism for credit assignment within a complex, multi-layer computational block (like a ResNet block). Our solution is to define a Hebbian-inspired local objective for the block (comparing its input X to its projected output Z) and then use backpropagation as an efficient and well-established computational tool to optimize the block's parameters to meet that objective. The auxiliary block and intra-block backpropagation function together as a computational abstraction for a local error circuit (e.g., dendritic computation), which guides plasticity based on information available locally to the module.\n\n**Comment 2:**\n\nThe core theory of the paper is derived from linear transformations, but it has been extended to nonlinear networks. The authors argue that due to the universal approximation ability of neural networks, they have the capacity to learn the theoretically optimal solution. However, even though the authors have empirical validated this through experiments in the appendix, it does not constitute a strict theoretical guarantee. Moreover, the introduction of the auxiliary network has improved final performance, possibly allowing the network to learn more powerful features.\n\n**Response to comment 2:**\n\nWe fully acknowledge that our theoretical analysis is grounded in a linear setting and that extending it to deep, nonlinear networks relies on the universal approximation capacity, which, as the reviewer correctly states, is not a formal guarantee of convergence to the global optimum. Our justification is two-fold. First, as we argue in our response, the purpose of Lemma 4.1 is to provide a strong theoretical motivation for the $L_{SPHeRe}$ loss. It demonstrates that the objective $||X^\\top X - Y^\\top Y||^2_F$ is principled and has a clear, desirable target in an analyzable setting: finding the optimal principal subspace. This provides a solid foundation for its use, moving it beyond a mere heuristic. Second, the key to applying this objective to deep networks is our proposed purely feedforward, decoupled architecture. We introduce a lightweight auxiliary module ($\\phi$) that operates alongside the main backbone block ($f$). This \"main-pathway learning, side-pathway supervision\" design creates a crucial separation of concerns: 1) The main block (f) is not forced to learn a direct, low-dimensional projection of the input. Instead, it can learn richer, higher-dimensional features (Y') that are more powerful for downstream tasks; 2) The auxiliary network $\\phi$ then bears the responsibility of projecting these rich features into a low-dimensional space Z where the structural preservation loss is computed. The empirical results in Appendix B, showing high CKA similarity, serve as strong evidence that this decoupling strategy successfully guides the nonlinear system to learn representations that are functionally equivalent to the theoretical linear optimum, even if a formal proof remains an open challenge.\n\n**Comment 3:**\n\nTable 2 shows that using L_{SPHeRe} alone is less effective than the original L_{oja} loss, and the paper does not adequately discuss and explain this phenomenon.\n\n**Response to comment 3:**\n\nWe thank the reviewer for this keen observation. It is a crucial point that highlights the core design trade-off behind SPHeRe. While $L_{oja}$ appears superior in isolation, its strength comes from the $(XX^\\top)^{⁻¹}$ term, which acts as an adaptive weighting mechanism. This can be powerful, but it also introduces significant numerical instability and high computational cost, especially in deep networks where the input feature maps can be high-dimensional and ill-conditioned, making the matrix inversion impractical or unreliable. By removing this inverse term, we consciously traded some of $L_{oja}$'s isolated adaptive power for a stable foundation. The true value of$ L_{SPHeRe}$ is revealed not in isolation, but in its synergy with the other components of our method. As Table 2 shows, the simplified and stable nature of$ L_{SPHeRe}$ allows it to be effectively combined with the orthogonality constraint ($L_{orth}$) and the auxiliary block ($\\phi$), leading to the overall best performance. The brittleness of $L_{oja}$ makes such stable integration more challenging. We will explicitly add this discussion to the revised manuscript to clarify this important design choice.\n\n**Comment 4:**\n\nThe ablation experiments in the paper demonstrate the effectiveness of the auxiliary block, but the authors have not analyze in depth how its structure affects the results.\n\n**Response to comment 4:**\n\nIn the manuscript, the auxiliary block $\\phi$ adopts the simplest nonlinear network, which consists of only one convolutional layer with a kernel size of 1, an average pooling layer that downsamples the feature map to 1, and a fully connected projection layer. Here we provide the results after changing the convolutional layer and the dimension of the projection layer.\n ```\n| convolutional layers      | 0     | 1     | 2     |\n| ------------------------- | ----- | ----- | ----- |\n| CIFAR10 Test accuracy (%) | 78.65 | 81.11 | 79.93 |\n ```\n\n ```\n| Project dimension         | 128  | 256   | 512   |\n| ------------------------- | ---- | ----- | ----- |\n| CIFAR10 Test accuracy (%) | 80.5 | 81.11 | 81.18 |\n ```\n\n**Comment 5:**\n\nIn the caption of Figure 2, “10^3” should be “10^{-3}”.\n\n**Response to comment 5:**\n\nWe sincerely thank the reviewer for their meticulous reading and for catching this typo, we will correct it in the revised manuscript.\n\n**Comment 6:**\n\nThe target of the orthogonal loss in the paper is applied to the auxiliary feature Z. Is this effective for the high-dimensional feature Y?\n\n**Response to comment 6:**\n\nThank you for this excellent question, which points to a crucial implementation detail that we admit was not perfectly clear in the manuscript.  To clarify, in our final implementation, the orthogonality loss $L_{orth}$ is indeed applied to the low-dimensional auxiliary feature Z for computational efficiency, as this avoids the high cost of computing $Y^\\top Y$ for a high-dimensional feature Y. The mechanism is one of indirect regularization. By optimizing the parameters of both the main block ($f$) and the auxiliary block ($\\phi$) to make $Z=\\phi(Y)$ orthogonal, the learning process implicitly forces the main block f to produce features Y that are more easily separable and projectable into an orthogonal basis by the simple $\\phi$ network. In essence, this encourages Y to become more decorrelated and less redundant, acting as a computationally tractable proxy for promoting good feature separation in the high-dimensional space. We will revise the manuscript to resolve any inconsistency and explicitly describe this mechanism.\n\n**Comment 7:**\n\nCan the SPHeRe method be combined with self-supervised learning to achieve better results?\n\n**Response to comment 7:**\n\nWe thank the reviewer for this highly insightful question. The combination of SPHeRe and self-supervised learning is indeed an interesting direction. We guess that combining our local, block-level self-supervision with a global, cross-sample SSL objective (e.g., SimCLR or BYOL) could create a powerful, multi-scale representation learning method, but this represents an exciting direction for future work."}, {"review_id": "K2lQrEFqdh", "reviewer": "Reviewer_dLRw", "summary": "This paper introduces an unsupervised learning framework called Structural Projection Hebbian Representation (SPHeRe), built on three key components:\n\n1. A loss function that maximises the similarity in sample structure of the output of a function and its input. This loss is given by\n\n$$L=|| YY^\\top -  XX^\\top ||^2_F$$\n\n2. A loss that encourages orthogonal outputs given by\n\n$$L = ||Y^\\top Y - I||^2$$\n\n3. A lightweight projection mapping of hidden activations to a lower-dimensional space before the losses are applied, reducing the computational cost of calculating $Y^\\top Y$ and easing the low-rank constraint of the similarity loss.\n\nTogether, these elements yield an efficient unsupervised algorithm suited to representation learning, continual learning, and transfer learning in deep networks. Experiments show that SPHeRe performs on par with, or better than, comparable local-learning methods on CIFAR-10, CIFAR-100, and Tiny-ImageNet.", "questions": "### Major\n\n- The paper would benefit from a more explicit argument for why the proposed update rule is genuinely Hebbian. How, in concrete neural terms, could a circuit compute the off-diagonal elements of $ XX^\\top$, $ YY^\\top$, and $Y^\\top Y$ using only pre- and post-synaptic signals and perhaps a local inhibition path? If some steps still rely on non-local information, please identify them and outline what additional mechanisms would be required to make the scheme fully Hebbian. This also includes a local implementation of the auxiliary projections.\nA clear, mechanistic explanation of these points would strengthen the biological-plausibility claim and would prompt me to raise my overall score to Accept.\n\n### Minor\n\n- Please explain why you expect the model not to suffer from catastrophic forgetting. An ablation that isolates each loss term’s contribution could help identify the mechanism.\n- Lemma 4.1 assumes that $M<N$. This is not true in many model architectures researchers might consider (and in the brain, eg generative mdoels). How does the model performance change if $M\\approx N$ or $M> N$?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\nQuality\n\n- Principled formulation: Lemma 4.1 grounds the method on solid theory.\n- Clear ablations: Table 2 carefully teases apart the contribution of each component.\n- Discussion of depth limits: The authors acknowledge the difficulty of scaling to very deep networks.\n\nClarity\n\n- Well-stated model and contributions: The architecture and its novelty are clearly laid out and illustrated.\n- Sensible use of the appendix: Technical details and proofs are placed where readers can find them without interrupting the main text.\n\nSignificance\n\n- State-of-the-art among local rules: The model outperforms previous local-learning methods on unsupervised representations.\n- Continual-learning results: It surpasses competing approaches on Split-CIFAR/Tiny-ImageNet.\n- learns representation that are transferable across image datasets\n- Impressive reconstructions: The decoder shows that most image features are preserved in the latent space.\n\nOriginality\n\n- Novel local learning rule: The proposal is clearly distinct from earlier Hebbian or STDP variants.\n- Auxiliary projection + similarity loss: This combination extends local unsupervised learning by mapping the latent state back to its input without enforcing constraints that might be too prohibitive (L2 loss) and allowing for different projection dimensions.\n\n## Weaknesses\n\nQuality\n\n- Not fully Hebbian: The porposed learning method is not fully hebbian for three reasons. 1) Non-local in time: The off-diagonal component of $ XX^\\top$ and $ YY^\\top$ are not hebbian because they couple activities from different time steps (batch samples). 2) Non-local in space: the off-diagonal components of $ Y^\\top Y$ require access to all neurons in $Y$. 3) Weight updates in the main network depend on gradients flowing through the auxiliary projection, violating strict locality. While items 1 and 2 might be approximated with online updates and/or lateral inhibition, point 3 is harder to reconcile with Hebbian principles.\n- Transfer-learning baseline missing: Without a reference model it is hard to gauge the significance of the reported gains.\n\nClarity\n\n- Evaluation method not present in the main text: Section 5.1 should clearly state that classification accuracy is obtained by training the fully connected output layer to classify image labels from the representations learned by the model.\n- No algorithm box: A concise pseudo-code block that shows gradient detachment to prevent full back-prop would improve reproducibility.\n\nSignificance\n\n- Speculative explanation of continual-learning performance: The paper offers a hypothesis but no empirical evidence.\n- Back-prop comparisons lacking: Since the method partly relies on back-propagation, it should also be compared with back-prop-based unsupervised learners in §5.2 because fully Hebian methods are at a disadvantage compared to the proposed method\n- Opaque reconstruction evidence: Table 6 in the supplementary should appear in the main text to support the reconstruction claims.\n\nOriginality\n\n- Orthogonal-representation losses are not new: Several prior works have already explored orthogonality constraints.", "was_revised": true, "final_justification": "The manuscript explores an interesting Hebbian-inspired learning approach. While the authors effectively addressed reviewer concerns with additional experiments (transfer learning) and explanations (continual learning), extensive work on positioning the manuscript as a biologically plausible learning algorithm remains necessary. Therefore, I will maintain my score.\n\nThe authors should explicitly choose either a biologically plausible or high-level of abstraction machine learning focus:\n\nTo ensure biological plausibility, the authors should clearly articulate the non-local properties of their model, present an explicit neural implementation that relies solely on local mechanisms, and empirically compare the performance of their high-level non-local model with its fully local counterpart to assess if the proposed learning principle could be used by the brain or if the learning degradation is too significant from local learning. Working with smaller models, as currently done in the paper would be acceptable in this case due to the strong biological constraints.\n\nAlternatively, the authors could stick to the current light-level abstraction of the Hebbian principle and continue relying on some non-biologically plausible computation. However, they should then demonstrate scalability using deeper models and rigorously compare performance against established unsupervised methods. Following this approach, the authors could also consider having more than one non-linear transformation within an individual Hebbian block to ease the training of deep models. A Resnet could, for instance, be split up into 6 subparts each trained with the proposed Hebbian-like loss function. \n\nClarifying the manuscript's position will significantly improve its impact. I appreciate the authors' responsiveness and encourage them to focus the manuscript accordingly.", "rebuttal": "**Comment 1:**\n\nNot fully Hebbian: ...\n\n**Response to comment 1:**\n\nWe are grateful to the reviewer for this insightful critique. We agree that SPHeRe is not a literal, synaptic-level implementation of a classical Hebbian rule, but rather a computational model that abstracts its core functional principles. Regarding non-locality in time and space (points 1 & 2), we view batch-level computations ($XX^\\top, YY^\\top$) as a computational abstraction for estimating input statistics over a short time window, and the orthogonality constraint ($L_{orth}$) as a functional analog of lateral inhibition, which promotes competition and decorrelation. We acknowledge that point 3—gradients flowing through the auxiliary module $\\phi$—is the most significant departure. However, we interpret $\\phi$ not as a non-local feedback path like in backpropagation, but as a functional abstraction of local error-generating circuits, such as dendritic compartments. These biological microcircuits can compute local prediction errors to guide plasticity within the same block, without requiring symmetric weights or global signals. This feedforward \"main pathway learning, side-pathway supervision\" architecture is a deliberate design choice that allows us to apply Hebbian-inspired principles to complex, non-linear deep networks. We will clarify this \"functional abstraction\" perspective in the revised manuscript.\n\n\n**Comment 2:**\n\nTransfer-learning baseline missing: Without a reference model it is hard to gauge the significance of the reported gains.\n\n**Response to comment 2:**\n\nWe thank the reviewer for this valuable suggestion. We agree that a reference model is crucial for contextualizing our transfer learning results. To address this, we will conduct additional experiments using SoftHebb as a direct baseline, as it is the state-of-the-art method we compare against in our main classification tasks. We will follow the identical protocol: pre-training SoftHebb on the source dataset (e.g., Tiny-ImageNet) and then evaluating the frozen features on the target dataset (e.g., CIFAR-10). The results will be added to Table 4 in the revised manuscript, providing a direct comparison that will better highlight the significance of SPHeRe's transferability. For convenience, we also display the complete table below.\n\n ```\n| Method   | Transfer Direction         | Transfer Learning (%) (Train/Test) | Non-Transfer (%) | Gap (%)   |\n| -------- | -------------------------- | ---------------------------------- | ---------------- | --------- |\n| BP       | Tiny-ImageNet → CIFAR-10   | 100 / 81.7                         | 88.7             | -7.0      |\n| BP       | CIFAR-10 → Tiny-ImageNet   | 99.9 / 39.2                        | 45.3             | -6.1      |\n| SoftHebb | Tiny-ImageNet → CIFAR-10   | 78 / 74.84                         | 78.86            | −4.02     |\n| SoftHebb | CIFAR-10 → Tiny-ImageNet   | 60 /33.26                          | 34.12            | -0.86     |\n| SPHeRe   | Tiny-ImageNet → CIFAR-10   | 97.3 / 80.03                       | 81.11            | −1.08     |\n| SPHeRe   | CIFAR-10 → Tiny-ImageNet   | 99.9 / 37.7                        | 40.33            | −2.63     |\n ```\n\n**Comment 3:**\n\nEvaluation method not present in the main text: Section 5.1 should clearly state that classification accuracy is obtained by training the fully connected output layer to classify image labels from the representations learned by the model.\n\n**Response to comment 3:**\n\nThe reviewer is correct, this information is crucial for reproducibility. Although we mentioned that the model includes \"a fully connected output layer trained with backpropagation\" in Section 5.1, we agree that this was not explicitly framed as the evaluation method. In the revised manuscript, we will clarify the training process more explicitly.\n\n**Comment 4:**\n\nNo algorithm box: A concise pseudo-code block that shows gradient detachment to prevent full back-prop would improve reproducibility.\n\n**Response to comment 4:**\n\nThanks for this suggestion. We agree that a pseudo-code algorithm box is essential for clarity and reproducibility. In the revised manuscript, we will add an algorithm box detailing the training process.\n\n**Comment 5:**\n\nSpeculative explanation of continual-learning performance: The paper offers a hypothesis but no empirical evidence.\nPlease explain why you expect the model not to suffer from catastrophic forgetting. An ablation that isolates each loss term’s contribution could help identify the mechanism.\n\n**Response to comment 5:**\n\nWe thank the reviewer for this crucial point. We speculate that the robustness of SPHeRe in continual learning mainly stems from our loss function, which encourages task-agnostic representations. $L_{SPHeRe}$ focuses on preserving the fundamental structure of the input data itself, rather than task-specific discriminative features, while $L_{orth}$ promotes decorrelation in the feature space, which may reduce interference between features learned for different tasks. We conduct ablation experiments on the Split-CIFAR100 benchmark, comparing the continual learning performance of the complete SPHeRe method (73.95%) with a version that uses only the $L_{SPHeRe}$ loss (73.8%).\n\n**Comment 6:**\n\nBack-prop comparisons lacking: Since the method partly relies on back-propagation, it should also be compared with back-prop-based unsupervised learners in §5.2 because fully Hebian methods are at a disadvantage compared to the proposed method\n\n**Response to comment 6:**\n\nThis is a very insightful point, and we agree that comparison with the backpropagation-based method is important. However, we think that a direct performance comparison with mature unsupervised learning (e.g., a contrastive learning method like SimCLR), even in a block-wise setting, requires careful contextualization, as it highlights a fundamental difference in learning philosophy. Current SOTA unsupervised methods rely on a strong, explicit pseudo-supervisory signal derived from sophisticated data augmentations. This is a powerful, human-designed heuristic for learning discriminative features. In contrast, SPHeRe runs on a different principle, it uses a simple and unsupervised objective that aims to preserve the local information structure of the input, without relying on such explicit supervisory signals. We provided a comparison with a self-supervised learning method, but for the reasons mentioned above, we do not believe that the wins or losses in comparison to it will diminish the contribution of our method. We use SimCLR to train the same network for 100 epochs, and achieve the test accuracy 71.94%. Although the accuracy of SimCLR is lower than that of SPHeRe, this does not mean that the SimCLR method is inferior; rather, it indicates that the two methods are suitable for different model structures. SPHeRe is suitable for shallower but wider networks, while SimCLR is suited for narrower and deeper networks.\n\n**Comment 7:**\n\nOpaque reconstruction evidence: Table 6 in the supplementary should appear in the main text to support the reconstruction claims.\n\n**Response to comment 7:**\n\nWe thank the reviewer for this valuable comment. In the revised manuscript, we will move the quantitative results from Table 6 into the main experimental section.\n\n**Comment 8:**\n\nOrthogonal-representation losses are not new: Several prior works have already explored orthogonality constraints.\n\n**Response to comment 8:**\n\nWe agree that orthogonality constraints are an established technique, and our novelty does not lie in the invention of the loss itself. Rather, our contribution is the specific application and motivation. In this paper, it works with $L_{SPHeRe}$ to limit the output features of each dimension to be as orthogonal as possible. In practical use, L_orth is applied to the low-dimensional auxiliary features Z, rather than the main high-dimensional output Y, which avoids the high computational cost of high-dimensional features $Y^\\top Y$ and improves computational efficiency. At the same time, the learning process implicitly encourages the main block f to generate features Y that are easier to be separated and projected onto an orthogonal basis by a simple $\\phi$ network, encouraging Y to become more decorrelated and less redundant. In the revised manuscript, we will add corresponding explanations to clarify our application of $L_{orth}$.\n\n**Comment 9:**\n\nLemma 4.1 assumes that $M < N$ . This is not true in many model architectures researchers might consider (and in the brain, eg generative mdoels). How does the model performance change if $M\\approx N$   or $M > N$ ?\n\n**Response to comment 9:**\n\nThank you to the reviewers for pointing out the limitations of the lemma. We acknowledge that the restriction $M < N$ is intended to demonstrate that our objective function can effectively preserve the main information when compressing data. When $M\\approx N$ and $M > N$, $L_{SPHeRe}$ does not have a unique solution, so at this point, the orthogonal loss$ L_{orth}$ becomes particularly important, as it prevents the model from learning simple identity transformations or redundant feature spaces. Once we introduce the auxiliary mapping $\\phi$, the relationship constraints between M and N will be unlocked, and the output Y will always be able to effectively encode the main features of X."}, {"review_id": "ZJDBxho13r", "reviewer": "Reviewer_ozBv", "summary": "The paper proposes SPHeRe, a Hebbian unsupervised representation learning method. It works by minimizing the L2 distance between the layer input and output Gram or Kernel matrices, in addition to using a regularizer to softly enforce orthogonality in output representations. The theoretical result provided shows that minimizing the SPHeRe objective essentially performs PCA on the input data matrix. Experiments on standard datasets show improved performance as compared to existing Hebbian representation learning methods. Additional experiments indicate that the method may be useful for continual learning and transfer learning use cases.", "questions": "0. Why is this paper of interest to the general Deep Learning community, not just specific groups working on Hebbian learning?\n1. Based on weakness 2, is the claim that SPHeRe is biologically plausible still valid? Why?\n2. What is the key novelty of SPHeRe compared to the Principal Subspace Whitening (PSW) method from Pehlivan et al.? Why hasn't this work been discussed in the paper?\n3. Have attempts been made to apply SPHeRe to modern deep architectures (e.g., ResNets, ViTs)? If so, what were the results? A simple statement in the conclusion stating \"performance gains slightly diminish as the number of convolutional layers increases\" does not provide any quantitative answers or observations with popular standard architectures.\n4. In the image reconstruction experiments, what was the objective used to train the encoder in the backpropagation baseline? From the code, it appears that the encoder was trained separately, even in the case of backpropagation, which is not the standard way autoencoders are trained.\n5. Have any experiments been performed for unsupervised reconstruction using only Hebbian-like mechanisms (without backdrop)?\n\nAddressing the first four concerns sufficiently would make me lean towards acceptance, while successfully addressing the next two would make me more confident in said acceptance.\n\n\nEDIT\n--------------------------\n\nThe authors have addressed my concerns to a good extent in the rebuttal.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The results are impressive when compared to existing Hebbian methods.\n2. The paper is well written and easy to follow.\n\nWeaknesses:\n1. While the paper addresses an interesting area of brain-inspired learning, it does not explain why this is an important problem in the demonstrated use cases. Such learning rules may be useful in cases when backpropagation is infeasible or extremely computationally expensive, for instance, in spiking networks or highly recurrent architectures. A brief experiment on SNNs has been included in the appendix only as an afterthought without significant analysis. However, it is unclear why the proposed method is relevant in the experimental setups used for its primary evaluation, where backpropagation is generally better by a significant margin. In fact, the paper does not even show a comparison with backpropagation as the learning algorithm in the classification experiments.\n\n2. The paper claims to be completely Hebbian, yet from the provided code it appears that it still requires backpropagation at a module level to obtain the gradients of $\\textbf{W}$ with respect to the Hebbian loss function. This part is not clearly explained in the paper.\n\n3. I have serious concerns regarding the novelty of the work: it appears extremely close to \"Why do similarity matching objectives lead to Hebbian/anti-Hebbian networks?\" by Pehlvan et al., specifically to their Principal Subspace Whitening (PSW) method. In fact, it reads quite as a validation of this work in multilayer architectures rather than a novel method in and of itself.\n\n4. The tested network is shallow (3 conv layers + FC), which makes it hard to assess scalability to modern deep architectures (e.g., ResNets or ViTs). Claims about scalability are therefore speculative.\n\n5. The reconstruction decoder is trained using backpropagation, making it unclear whether the gains are due to SPHeRe’s representations or simply to the power of the decoder. Additional details about the backpropagation baseline that allow for a fair comparison are missing.", "was_revised": true, "final_justification": "I thank the authors for their response.\n\nI appreciate their ideas about the potential of the work in mainline Deep Learning, and would encourage them to include preliminary results in the stated directions as proof-of-concept. However, regarding the point relating to biological plausibility, I would suggest that the main text loosen the claim that the method is biologically plausible, since the authors' comment that \" SPHeRe represents a significant and principled step towards biological plausibility\" is a more accurate description of the paper's contributions. \n\nI also appreciate and agree with the comments about how the work is distinct from Pehlevan et al, especially when taking into account the comments of other reviewers and their rebuttal texts.\n\nThe authors' responses to comments 5 and 6 also clarify some details for me, and I strongly encourage them to include the explanations in the main paper.\n\nHowever, I still retain my criticisms about the scalability of the work, given that the approach does not scale well to larger networks. This makes the impact of the paper quite limited for the broader community unless proof-of-concepts involving large networks are included (see first para).\n\nGiven that the rebuttal addresses some of my concerns, I shall increase my score. However, I still feel that the paper's text needs refinement and several additions, and experiments need to include larger, realistic models to make the paper ready for NeurIPS. I therefore still lean towards a reject, and recommend a resubmission after addressing the highlighted issues.\n\nEDIT\n------------------------\nThe authors have addressed by concerns to a good extent in the discussions. I therefore raise my score to a weak accept.", "rebuttal": "**Comment 1:**\n\nWhy is this paper of interest to the general Deep Learning community, not just specific groups working on Hebbian learning?\n\n**Response to comment 1:**\n\nThanks for this question. SPHeRe's value extends beyond the Hebbian learning niche, it contributes to the broader deep learning community by proposing a novel, concrete, and effective greedy unsupervised pre-training paradigm. It tries to solve a universal challenge: how to achieve effective layer-wise feature learning without relying on global backpropagation or external supervision. Our key innovation lies in a purely feedforward architecture that uses a decoupled, lightweight auxiliary module to compute a structural similarity loss in a low-dimensional space, thereby guiding local updates in the main network. This design is not just theoretically sound, its efficacy is directly proven by its strong generalization capabilities in complex scenarios like continual and transfer learning, which validates the robustness of the learned features. Therefore, SPHeRe is more than an algorithm, it is also a practical technical solution designed for seamless integration into mainstream frameworks, offering a concrete and effective path for exploring local, unsupervised learning in modern deep networks.\n\n**Comment 2:**\n\nBased on weakness 2, is the claim that SPHeRe is biologically plausible still valid? Why?\n\n**Response to comment 2:**\n\nWe appreciate you highlighting the use of intra-module backpropagation, as it allows us to clarify our position on biological plausibility. We view this as a pragmatic and effective design choice that moves our model significantly closer to biological systems compared to standard end-to-end backpropagation.  SPHeRe is designed to specifically eliminate this global dependency. We achieve this by confining gradient calculations to a small, local, and decoupled auxiliary module ($\\phi$). We view the backpropagation within this module not as a model of how individual synapses update, but as a computationally powerful tool to solve a local credit assignment problem. This is functionally analogous to how local circuits, like dendritic compartments, are theorized to generate local teaching signals to guide plasticity without needing symmetric weights. Some works [1, 2, 3] are also dedicated to addressing the issue of biological plausibility in backpropagation. This \"main-pathway learning, side-pathway supervision\" design is a core contribution. It structurally avoids the need for the explicit, long-range feedback connections that other theoretical models rely on for credit assignment. Therefore, by ensuring all learning is driven by local signals within a purely feedforward architecture, SPHeRe represents a significant and principled step towards biological plausibility.\n\n[1] Bengio Y, Lee D H, Bornschein J, et al. Towards biologically plausible deep learning[J]. arXiv preprint arXiv:1502.04156, 2015.\n\n[2] Xiao W, Chen H, Liao Q, et al. Biologically-Plausible Learning Algorithms Can Scale to Large Datasets[C]//International Conference on Learning Representations.\n\n[3] Lv C, Xu J, Lu Y, et al. Dendritic Localized Learning: Toward Biologically Plausible Algorithm[C]//Forty-second International Conference on Machine Learning.\n\n**Comment 3:**\n\nWhat is the key novelty of SPHeRe compared to the Principal Subspace Whitening (PSW) method from Pehlivan et al.? Why hasn't this work been discussed in the paper?\n\n**Response to comment 3:**\n\nWe sincerely thank you for highlighting the foundational work of Pehlevan, Chklovskii, and their collaborators. This was a significant oversight in our initial submission, and we are grateful for the opportunity to place our work in this crucial context. We will add a thorough discussion and comparison in the revised manuscript. You are correct that our work shares a similar mathematical starting point: optimizing a similarity matching objective like $||X^\\top X - Y^\\top Y||^2_F$​. However, the path we take to solve the core challenge of how to effectively apply this objective to deep nonlinear networks is fundamentally different, which precisely highlights the unique contribution of our SPHeRe method. We see this as \"different paths to the same goal\":\n\n1) Pehlevan et al.'s Path (Feedback-Dependent): Their work elegantly derives local, Hebbian/anti-Hebbian synaptic update rules from the objective using methods like min-max dual optimization. To enable credit assignment in deep networks, their models theoretically rely on feedback connections to propagate information across layers.\n\n2) Our Path (Feedforward-Only): SPHeRe proposes a distinctly different, purely feedforward architectural solution. Our core innovation is the introduction of a decoupled, lightweight auxiliary module ($\\phi$). Instead of relying on feedback, this module projects the main pathway's high-dimensional features into a low-dimensional space where the similarity loss is calculated.\n\nThis design, which we term \"main-pathway learning, side-pathway supervision,\" structurally avoids the need for any explicit feedback paths. This makes our block-wise training scheme highly modular, scalable, and seamlessly integrable into standard deep learning frameworks, representing a novel and practical approach to implementing Hebbian-inspired principles in modern deep networks.\n\n\n**Comment 4:**\n\nHave attempts been made to apply SPHeRe to modern deep architectures (e.g., ResNets, ViTs)? If so, what were the results? A simple statement in the conclusion stating \"performance gains slightly diminish as the number of convolutional layers increases\" does not provide any quantitative answers or observations with popular standard architectures.\n\n**Response to comment 4:**\n\nThank you for this critical question on scalability. Our current focus on shallower (e.g., 3-layer), wider networks is a deliberate choice informed by both biological fidelity and a deep understanding of our method's inherent properties. On one hand, this architecture better reflects cortical microcircuits, a common strategy for Hebbian-inspired methods, including the current SOTA (SoftHebb). More critically, we observed that SPHeRe's performance saturates beyond 3-4 layers. We attribute this to the intrinsic nature of its greedy, local learning mechanism: each layer's low-dimensional projection is inherently lossy, inevitably discarding some information and introducing representation noise. While this effectively extracts robust low-level features in shallow networks, the accumulated noise progressively overwhelms higher-level semantic signals as depth increases, thus limiting performance. This insight directly explains why applying it to deep architectures like ResNets or ViTs is challenging. A practical path forward is to use SPHeRe as a powerful \"stem\" to provide high-quality initial representations for subsequent residual blocks or attention modules to improve the adaptation performance [1]. To train an entire deep network with SPHeRe, however, would likely require integrating larger-scale feedback or contextual signals to correct this error accumulation, which is a core direction for our future work.\n\n[1] Tang Y, Zhang C, Xu H, et al. Neuro-modulated hebbian learning for fully test-time adaptation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 3728-3738.\n\n**Comment 5:**\n\nIn the image reconstruction experiments, what was the objective used to train the encoder in the backpropagation baseline? From the code, it appears that the encoder was trained separately, even in the case of backpropagation, which is not the standard way autoencoders are trained.\n\n**Response to comment 5:**\n\nYes, our backpropagation  baseline encoder is trained on classification tasks, while the parameters are frozen during the reconstruction task, which is indeed different from the standard end-to-end autoencoder training. We apologize for not making the rationale for this design clearer. The purpose of the experiment is to compare the quality of features (retention of input image information) extracted by different pretrained encoders. In the reconstruction experiment, we freeze the weights of all pretrained encoders (trained on classification tasks through SPHeRe, SoftHebb, and supervised BP) and then fully train an independent decoder with the same structure for each encoder. This design allows us to directly assess how much image information is retained in the features generated by each method, with lower reconstruction loss indicating more complete features. We will add a clear explanation of the pre-training encoders and the design of this comparative experiment in the revised manuscript to eliminate ambiguity.\n\n**Comment 6:**\n\nHave any experiments been performed for unsupervised reconstruction using only Hebbian-like mechanisms (without backdrop)?\n\n**Response to comment 6:**\n\nIn the current work, we did not perform experiments on a fully Hebbian-based autoencoder where the decoder is also trained with backpropagation. Our primary goal for the reconstruction experiment was to use it as a diagnostic tool to quantitatively compare the information content and robustness of features learned by different encoders. Due to the task independence of Hebbian, unsupervised reconstruction using only Hebbian-like mechanisms is very difficult. We believe this is a very exciting direction for future work, and it will represent a major step towards a complete, end-to-end biologically plausible generative model, but currently, there is a lack of corresponding implementation pathways."}, {"review_id": "gQgH9LXawP", "reviewer": "Reviewer_uqXK", "summary": "Summary: In this paper, the authors present a greedy blockwise training mechanism for unsupervised learning on image data, and connect their objective function to Hebbian learning objectives. Relative to local learning algorithms like SoftHebb, the authors demonstrate an improvement in performance and representational quality. The authors also show improvements in continual learning relative to a diversity of baseline algorithms.", "questions": "The authors repeatedly refer to the optimum identified by Lemma 4.1 (Line 167) as the natural target for their nonlinear network optimization. However, this optimum is derived from a single-layer linear weight matrix, and it is not clear to me why a nonlinear network would achieve the same optimum. Furthermore, if this were the optimum achieved by a nonlinear network, why bother with a nonlinear transformation? And why bother with a multilayer network at all, when a single linear transformation would suffice?\nAre there any testable predictions that could be made for experimentalists looking to demonstrate that this learning algorithm is used in the brain? If not, it is unclear to me what the purpose of establishing a connection to biology is.\n\nLine 100-102: there are many theories for how gradient computations could be implemented in the brain—it is not fair to claim that end-to-end gradient computation is implausible (at least with some approximations required); see Lillicrap & Hinton 2020 for a review.\n\nLine 135: Is the ‘Hebbian loss’ actually a loss function? It is not bounded from below as far as I can tell.\n\nLine 173: Lemma 4.2.1 is referenced, but the text refers to the lemma as 4.1—I believe this happens in a few places in the text.\n Recommendations: Personally, I think that this paper has the potential to be high quality, but references to ‘biological plausibility’ should be removed—in my opinion this paper would be better written as an algorithm for greedy block-wise training, and the benchmarks should reflect this (comparing to greedy blockwise contrastive algorithms, autoencoders, or denoising objectives), instead of comparing to biological learning algorithms. Unfortunately, I suspect that this algorithm’s performance will not be as competitive when compared to these alternative benchmarks.", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "In terms of strengths, the paper’s experiments are very thorough and well controlled as far as I can tell. Benchmarks are provided against a variety of different methods, and the authors provide a lesion study of their objective function, removing different components to show which components of the loss variously affect the network’s performance.\n\nIn terms of weaknesses, unfortunately the authors advertise their algorithm as ‘biologically plausible,’ and yet incorporate numerous elements that rely heavily on backpropagation and architectural violations of biological locality principles. The authors use weight-shared convolutional layers and use multiple network layers per block in their network architecture: this means that their network does not resolve the weight transport problem, and it is consequently little surprise that their method is able to outperform local learning algorithms with a more direct connection to biology.\n\nEven in the linear network derivation, the SPHeRe update rule does not appear to be local (Line 180-181), though the authors explicitly advertise it as such. While I could be convinced otherwise, I don’t think there is local learning occurring in this model, even in its simplest form. I would recommend that the authors write out the parameter update for a single synaptic parameter, and demonstrate that it depends exclusively on pre- and post-synaptic activation terms as in traditional Hebbian learning; if it is not, the framing of much of the paper is very misleading.\n\nLastly, in terms of originality, the Gram matrix loss adopted by the authors is very similar to the similarity matching objective employed by Obeid et al. 2019, which was also applied to deep networks successfully—a comparison to this method would be very helpful.\n\nRecommendations: Personally, I think that this paper has the potential to be high quality, but references to ‘biological plausibility’ should be removed—in my opinion this paper would be better written as an algorithm for greedy block-wise training, and the benchmarks should reflect this (comparing to greedy blockwise contrastive algorithms, autoencoders, or denoising objectives), instead of comparing to biological learning algorithms. Unfortunately, I have know way of knowing whether this algorithm will be competitive when compared to these alternative benchmarks.", "was_revised": true, "final_justification": "The authors have addressed the most of my concerns, but substantial revision of the manuscript will be required to implement my suggested changes. I still think that the paper would be best resubmitted, but I have increased my score and decreased my confidence in order to recognize the substantial improvements the authors have demonstrated during the revision process.", "rebuttal": "**Comment 1:**\n\nThe authors advertise their algorithm as ‘biologically plausible,’ and yet incorporate numerous elements that rely heavily on backpropagation and architectural violations of biological locality principles.\n\n**Response to comment 1:**\n\nWe sincerely thank the reviewer for this sharp and important comment. We fully agree that our claim of \"biological plausibility\" requires careful framing. Our goal is not to create a literal simulation of synaptic dynamics, but rather a computational and functional model that abstracts key biological principles for effective application in modern, complex, nonlinear networks. Regarding the specific points raised: 1) Intra-block backpropagation: We view this as a functional abstraction of a local error circuit (e.g., dendritic computation). It is an efficient computational tool for local credit assignment that avoids the most biologically problematic requirements of a global error signal and symmetric weights. 2) Weight-shared convolutional layers: We see this as a computational abstraction for the replication of feature detectors observed in the brain. While updating a shared weight is computationally non-local, the function it performs—applying the same feature extractor across the receptive field—is functionally analogous to spatially repeated structures in the visual cortex. Therefore, we believe SPHeRe makes a pragmatic trade-off: it leverages modern computational tools to efficiently implement high-level, biologically-inspired functional goals, allowing us to scale Hebbian principles to complex deep models while respecting block-level locality.\n\n**Comment 2:**\n\n...I don’t think there is local learning occurring in this model, even in its simplest form...\n\n**Response to comment 2:**\n\nWe agree that the gradient equation $\\Delta W = 4X^\\top(YY^\\top - XX^\\top)Y$, when viewed in isolation, does not have the form of a classical Hebbian rule, which depends exclusively on the activity of a single pre- and post-synaptic pair. However,  this objective shares the exact same mathematical starting point with the foundational work of Pehlevan, Chklovskii, et al. Their work has rigorously proven that this very objective can be decomposed into fully local, Hebbian/anti-Hebbian update rules at each individual synapse through methods like dual optimization. Instead of deriving these synaptic-level rules, we leverage modern deep learning frameworks to directly optimize this principled, locally-grounded objective within a local block. Meanwhile, we design a purely feedforward, modular architecture that allows this powerful, locally-motivated objective to be efficiently scaled to complex, nonlinear deep networks. We hope that this method can become a small step in the combination of biological update rules and modern deep neural network learning.\n\n**Comment 3:**\n\nLastly, in terms of originality, the Gram matrix loss adopted by the authors is very similar to the similarity matching objective employed by Obeid et al. 2019...\n\n**Response to comment 3:**\n\nThank you for pointing out this crucial connection. Please Please refer to our response to reviewer b3Ah's comment 1. We will add a detailed comparison to Obeid et al. (2019) in our revision. While we share the similarity matching objective, our novelty lies in the method of its application in deep networks. Where Obeid et al. derived local rules requiring feedback for credit assignment, we introduce a decoupled, lightweight auxiliary block ($\\phi$) to compute the loss. This frees the main block ($f$) to learn richer, higher-dimensional features optimal for downstream tasks, rather than being constrained to a low-dimensional projection. Our approach is thus a modular, scalable block-wise scheme that offers a new, practical bridge between these biological principles and modern architectures.\n\n\n**Comment 4:**\n\nThe authors repeatedly refer to the optimum identified by Lemma 4.1 (Line 167) as the natural target for their nonlinear network optimization...\n\n**Response to comment 4:**\n\nThese are very good questions that get to the heart of our work. First, regarding why a nonlinear network aims for a linear optimal solution and why nonlinear multi-layer layers are needed, the key lies in the decoupled architecture we designed. The purpose of Lamma 4.1 is not to require the entire nonlinear main module f to learn a simple dimensionality reduction projection transformation, but rather to provide a principled motivation for the loss function $L_{SPHeRe}$, demonstrating that it can guide the system to find the optimal subspace for information retention under an analytically linear setting. However, the learning objective of the main module f is to learn powerful, high-dimensional, nonlinear transformations to extract complex features beneficial for downstream tasks, which cannot be achieved by a single-layer linear transformation. The auxiliary module ($\\phi$) takes on the task of projecting these rich features Y into a low-dimensional space Z. By optimizing $L_{SPHeRe}(Z, X)$, Y is more inclined to learn effective feature representations, thereby facilitating $\\phi$'s dimensionality reduction projection work. Our model proposes a specific, testable computational hypothesis: it predicts that in cortical areas, there exist local microcircuits (whose function is abstracted by our $\\phi$ module) whose activity encodes the \"mismatch\" signal between the input and output statistical structures of that region, and this activity modulates local synaptic plasticity without the need for a global error signal. Experimentally, neuroscientists can selectively inhibit these proposed locally modulatory interneurons (such as inhibitory interneurons) and observe whether it specifically impairs the experience-dependent plasticity of that region (such as feature learning in visual tasks) without affecting the main feedforward information flow. Our work originates from a simplification and development of Hebbian theory, and integrating it with deep learning frameworks, rather than designing rules for neuron-level synaptic updates.\n\n**Comment 5:**\n\nLine 100-102: there are many theories for how gradient computations could be implemented in the brain...\n\n**Response to comment 5:**\n\nThanks for your suggestion. We agree that our phrasing needs to be more precise. Labeling end-to-end computation as implausible may be too absolute, and there is a wealth of outstanding research exploring its biological possibilities. However, we believe that these theories of approximate backpropagation (such as feedback alignment) also introduce unverified biological hypotheses, like symmetric feedback neural pathways, when addressing the weight symmetry issue. In contrast, the architecture of SPHeRe provides a method that relies on local auxiliary blocks to generate learning signals within blocks, avoiding dependence on top-down end-to-end error signals. Therefore, our work does not deny the research on BP approximations but attempts to explore an independent path of biologically plausible learning.\n\n**Comment 6:**\n\nLine 135: Is the ‘Hebbian loss’ actually a loss function? It is not bounded from below as far as I can tell.\n\n**Response to comment 6:**\n\nYou are absolutely correct, $L_{hebb}$ is not, by itself, a bounded-below, stably optimizable loss function. Our purpose in introducing it is precisely to pedagogically demonstrate the inherent instability of the classical Hebbian rule within a modern optimization framework. We use the term \"equivalent loss function\" to intuitively illustrate the direction in which the classical update rule $\\Delta W \\propto X^\\top Y$ pushes the network—namely, towards an unconstrained maximization of the output's energy.  This instability is the explicit motivation for introducing Oja's rule and, ultimately, our final, well-behaved, non-negative $L_{SPHeRe}$ objective. The narrative builds from this \"ill-posed\" objective to our stable, grounded solution.\n\n**Comment 7:**\n\nLine 173: Lemma 4.2.1 is referenced, but the text refers to the lemma as 4.1—I believe this happens in a few places in the text. Recommendations: Personally, I think that this paper has the potential to be high quality, but references to ‘biological plausibility’ should be removed...\n\n**Response to comment 7:**\n\nWe appreciate the reviewer for pointing out the refer typo error in the manuscript, and will correct them in the revised manuscript. We will adopt your suggestion in the revised manuscript and refine the positioning of SPHeRe from a general \"bio-plausible\" algorithm to a more precise \"Hebbian-inspired, greedy layer-wise unsupervised pre-training method.\"  However, we believe it is vital to retain its \"Hebbian-inspired\" roots, as this is the core design philosophy and motivation that distinguishes our method from other purely engineered approaches. Our loss function, $L_{SPHeRe}$, is not an arbitrary heuristic, it is derived from simplifying and generalizing the objective behind the Hebbian principle. It aims to preserve the intrinsic structure of the input, which provides a distinct and complementary perspective to some mature unsupervised learning methods like contrastive learning that rely on strong priors like data augmentation. Therefore, our statement of \"biological plausibility\" does not refer to a literal simulation of neurons, but to an exploration of more macro (or mesoscopic) learning principles. This biologically-inspired constraint makes our approach an exploratory method towards biological plausibility learning, similar to algorithms (FA or DFA) that also attempt to solve the \"biologically plausible credit assignment\" problem. And we achieved a test set accuracy of only 71.94% using SimCLR under the same network structure and training epochs on CIFAR10, which indicates that our method is not inferior to some self-supervised learning methods with strong supervisory signals on shallow and wide network structures."}]

Author rebuttal:

Meta-review: Inspired by Oja's Hebbian learning rule, this work proposes a similarity-based objective for unsupervised representation learning. They optimize their objective in a deep neural network using the backpropagation algorithm and show that it performs favorable when compared with other unsupervised learning objectives/algorithms. Multiple reviewers took issue with two points: (a) the paper was misleading and led the reader to believe that the authors were proposing "biologically plausible" learning algorithm; (b) that the work did not situate itself within a rather extensive existing literature on similarity-based objectives that have been used to derive "biologically plausible" learning algorithms. There were also some concerns about the benchmarks for comparison with existing unsupervised models. In the rebuttal and during the discussion period, the authors clarified that their work is "biologically inspired" rather than "biologically plausible", promised to properly place their work within the existing literature on similarity-preserving objectives and performed additional experiments. After the discussion period, several reviewers raised their scores. In the end, some of the reviewers still thought the paper could use another round of revisions, while others thought that the authors had adequately addressed their concerns. Ultimately, I think that the main objections from the reviewers can be addressed by the authors in their revision and so I recommend acceptance.
Decision: Accept (poster)