Title: Rethinking Hebbian Principle: Low-Dimensional Structural Projection for Unsupervised Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Synaptic plasticity

Hebbian learning

Unsupervised learning

Local learning
Abstract: Hebbian learning is a biological principle that intuitively describes how neurons adapt their connections through repeated stimuli. However, when applied to machine learning, it suffers serious issues due to the unconstrained updates of the connections and the lack of accounting for feedback mediation. Such shortcomings limit its effective scaling to complex network architectures and tasks. To this end, here we introduce the Structural Projection Hebbian Representation (SPHeRe), a novel unsupervised learning method that integrates orthogonality and structural information preservation through a local auxiliary nonlinear block. The loss for structural information preservation backpropagates to the input through an auxiliary lightweight projection that conceptually serves as feedback mediation while the orthogonality constraints account for the boundedness of updating magnitude. Extensive experimental results show that SPHeRe achieves SOTA performance among unsupervised synaptic plasticity approaches on standard image classification benchmarks, including CIFAR-10, CIFAR-100, and Tiny-ImageNet. Furthermore, the method exhibits strong effectiveness in continual learning and transfer learning scenarios, and image reconstruction tasks show the robustness and generalizability of the extracted features. This work demonstrates the competitiveness and potential of Hebbian unsupervised learning rules within modern deep learning frameworks, demonstrating the possibility of efficient and biologically inspired learning algorithms without the strong dependence on strict backpropagation. Our code is available at https://github.com/brain-intelligence-lab/SPHeRe.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary: In this paper, the authors present a greedy blockwise training mechanism for unsupervised learning on image data, and connect their objective function to Hebbian learning objectives. Relative to local learning algorithms like SoftHebb, the authors demonstrate an improvement in performance and representational quality. The authors also show improvements in continual learning relative to a diversity of baseline algorithms.\n\nSTRENGTHS AND WEAKNESSES:\nIn terms of strengths, the paper’s experiments are very thorough and well controlled as far as I can tell. Benchmarks are provided against a variety of different methods, and the authors provide a lesion study of their objective function, removing different components to show which components of the loss variously affect the network’s performance.\n\nIn terms of weaknesses, unfortunately the authors advertise their algorithm as ‘biologically plausible,’ and yet incorporate numerous elements that rely heavily on backpropagation and architectural violations of biological locality principles. The authors use weight-shared convolutional layers and use multiple network layers per block in their network architecture: this means that their network does not resolve the weight transport problem, and it is consequently little surprise that their method is able to outperform local learning algorithms with a more direct connection to biology.\n\nEven in the linear network derivation, the SPHeRe update rule does not appear to be local (Line 180-181), though the authors explicitly advertise it as such. While I could be convinced otherwise, I don’t think there is local learning occurring in this model, even in its simplest form. I would recommend that the authors write out the parameter update for a single synaptic parameter, and demonstrate that it depends exclusively on pre- and post-synaptic activation terms as in traditional Hebbian learning; if it is not, the framing of much of the paper is very misleading.\n\nLastly, in terms of originality, the Gram matrix loss adopted by the authors is very similar to the similarity matching objective employed by Obeid et al. 2019, which was also applied to deep networks successfully—a comparison to this method would be very helpful.\n\nRecommendations: Personally, I think that this paper has the potential to be high quality, but references to ‘biological plausibility’ should be removed—in my opinion this paper would be better written as an algorithm for greedy block-wise training, and the benchmarks should reflect this (comparing to greedy blockwise contrastive algorithms, autoencoders, or denoising objectives), instead of comparing to biological learning algorithms. Unfortunately, I have know way of knowing whether this algorithm will be competitive when compared to these alternative benchmarks.\n\nQUESTIONS:\nThe authors repeatedly refer to the optimum identified by Lemma 4.1 (Line 167) as the natural target for their nonlinear network optimization. However, this optimum is derived from a single-layer linear weight matrix, and it is not clear to me why a nonlinear network would achieve the same optimum. Furthermore, if this were the optimum achieved by a nonlinear network, why bother with a nonlinear transformation? And why bother with a multilayer network at all, when a single linear transformation would suffice?\nAre there any testable predictions that could be made for experimentalists looking to demonstrate that this learning algorithm is used in the brain? If not, it is unclear to me what the purpose of establishing a connection to biology is.\n\nLine 100-102: there are many theories for how gradient computations could be implemented in the brain—it is not fair to claim that end-to-end gradient computation is implausible (at least with some approximations required); see Lillicrap & Hinton 2020 for a review.\n\nLine 135: Is the ‘Hebbian loss’ actually a loss function? It is not bounded from below as far as I can tell.\n\nLine 173: Lemma 4.2.1 is referenced, but the text refers to the lemma as 4.1—I believe this happens in a few places in the text.\n Recommendations: Personally, I think that this paper has the potential to be high quality, but references to ‘biological plausibility’ should be removed—in my opinion this paper would be better written as an algorithm for greedy block-wise training, and the benchmarks should reflect this (comparing to greedy blockwise contrastive algorithms, autoencoders, or denoising objectives), instead of comparing to biological learning algorithms. Unfortunately, I suspect that this algorithm’s performance will not be as competitive when compared to these alternative benchmarks.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes SPHeRe, a Hebbian unsupervised representation learning method. It works by minimizing the L2 distance between the layer input and output Gram or Kernel matrices, in addition to using a regularizer to softly enforce orthogonality in output representations. The theoretical result provided shows that minimizing the SPHeRe objective essentially performs PCA on the input data matrix. Experiments on standard datasets show improved performance as compared to existing Hebbian representation learning methods. Additional experiments indicate that the method may be useful for continual learning and transfer learning use cases.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The results are impressive when compared to existing Hebbian methods.\n2. The paper is well written and easy to follow.\n\nWeaknesses:\n1. While the paper addresses an interesting area of brain-inspired learning, it does not explain why this is an important problem in the demonstrated use cases. Such learning rules may be useful in cases when backpropagation is infeasible or extremely computationally expensive, for instance, in spiking networks or highly recurrent architectures. A brief experiment on SNNs has been included in the appendix only as an afterthought without significant analysis. However, it is unclear why the proposed method is relevant in the experimental setups used for its primary evaluation, where backpropagation is generally better by a significant margin. In fact, the paper does not even show a comparison with backpropagation as the learning algorithm in the classification experiments.\n\n2. The paper claims to be completely Hebbian, yet from the provided code it appears that it still requires backpropagation at a module level to obtain the gradients of $\\textbf{W}$ with respect to the Hebbian loss function. This part is not clearly explained in the paper.\n\n3. I have serious concerns regarding the novelty of the work: it appears extremely close to \"Why do similarity matching objectives lead to Hebbian/anti-Hebbian networks?\" by Pehlvan et al., specifically to their Principal Subspace Whitening (PSW) method. In fact, it reads quite as a validation of this work in multilayer architectures rather than a novel method in and of itself.\n\n4. The tested network is shallow (3 conv layers + FC), which makes it hard to assess scalability to modern deep architectures (e.g., ResNets or ViTs). Claims about scalability are therefore speculative.\n\n5. The reconstruction decoder is trained using backpropagation, making it unclear whether the gains are due to SPHeRe’s representations or simply to the power of the decoder. Additional details about the backpropagation baseline that allow for a fair comparison are missing.\n\nQUESTIONS:\n0. Why is this paper of interest to the general Deep Learning community, not just specific groups working on Hebbian learning?\n1. Based on weakness 2, is the claim that SPHeRe is biologically plausible still valid? Why?\n2. What is the key novelty of SPHeRe compared to the Principal Subspace Whitening (PSW) method from Pehlivan et al.? Why hasn't this work been discussed in the paper?\n3. Have attempts been made to apply SPHeRe to modern deep architectures (e.g., ResNets, ViTs)? If so, what were the results? A simple statement in the conclusion stating \"performance gains slightly diminish as the number of convolutional layers increases\" does not provide any quantitative answers or observations with popular standard architectures.\n4. In the image reconstruction experiments, what was the objective used to train the encoder in the backpropagation baseline? From the code, it appears that the encoder was trained separately, even in the case of backpropagation, which is not the standard way autoencoders are trained.\n5. Have any experiments been performed for unsupervised reconstruction using only Hebbian-like mechanisms (without backdrop)?\n\nAddressing the first four concerns sufficiently would make me lean towards acceptance, while successfully addressing the next two would make me more confident in said acceptance.\n\n\nEDIT\n--------------------------\n\nThe authors have addressed my concerns to a good extent in the rebuttal.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an unsupervised learning framework called Structural Projection Hebbian Representation (SPHeRe), built on three key components:\n\n1. A loss function that maximises the similarity in sample structure of the output of a function and its input. This loss is given by\n\n$$L=|| YY^\\top -  XX^\\top ||^2_F$$\n\n2. A loss that encourages orthogonal outputs given by\n\n$$L = ||Y^\\top Y - I||^2$$\n\n3. A lightweight projection mapping of hidden activations to a lower-dimensional space before the losses are applied, reducing the computational cost of calculating $Y^\\top Y$ and easing the low-rank constraint of the similarity loss.\n\nTogether, these elements yield an efficient unsupervised algorithm suited to representation learning, continual learning, and transfer learning in deep networks. Experiments show that SPHeRe performs on par with, or better than, comparable local-learning methods on CIFAR-10, CIFAR-100, and Tiny-ImageNet.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nQuality\n\n- Principled formulation: Lemma 4.1 grounds the method on solid theory.\n- Clear ablations: Table 2 carefully teases apart the contribution of each component.\n- Discussion of depth limits: The authors acknowledge the difficulty of scaling to very deep networks.\n\nClarity\n\n- Well-stated model and contributions: The architecture and its novelty are clearly laid out and illustrated.\n- Sensible use of the appendix: Technical details and proofs are placed where readers can find them without interrupting the main text.\n\nSignificance\n\n- State-of-the-art among local rules: The model outperforms previous local-learning methods on unsupervised representations.\n- Continual-learning results: It surpasses competing approaches on Split-CIFAR/Tiny-ImageNet.\n- learns representation that are transferable across image datasets\n- Impressive reconstructions: The decoder shows that most image features are preserved in the latent space.\n\nOriginality\n\n- Novel local learning rule: The proposal is clearly distinct from earlier Hebbian or STDP variants.\n- Auxiliary projection + similarity loss: This combination extends local unsupervised learning by mapping the latent state back to its input without enforcing constraints that might be too prohibitive (L2 loss) and allowing for different projection dimensions.\n\n## Weaknesses\n\nQuality\n\n- Not fully Hebbian: The porposed learning method is not fully hebbian for three reasons. 1) Non-local in time: The off-diagonal component of $ XX^\\top$ and $ YY^\\top$ are not hebbian because they couple activities from different time steps (batch samples). 2) Non-local in space: the off-diagonal components of $ Y^\\top Y$ require access to all neurons in $Y$. 3) Weight updates in the main network depend on gradients flowing through the auxiliary projection, violating strict locality. While items 1 and 2 might be approximated with online updates and/or lateral inhibition, point 3 is harder to reconcile with Hebbian principles.\n- Transfer-learning baseline missing: Without a reference model it is hard to gauge the significance of the reported gains.\n\nClarity\n\n- Evaluation method not present in the main text: Section 5.1 should clearly state that classification accuracy is obtained by training the fully connected output layer to classify image labels from the representations learned by the model.\n- No algorithm box: A concise pseudo-code block that shows gradient detachment to prevent full back-prop would improve reproducibility.\n\nSignificance\n\n- Speculative explanation of continual-learning performance: The paper offers a hypothesis but no empirical evidence.\n- Back-prop comparisons lacking: Since the method partly relies on back-propagation, it should also be compared with back-prop-based unsupervised learners in §5.2 because fully Hebian methods are at a disadvantage compared to the proposed method\n- Opaque reconstruction evidence: Table 6 in the supplementary should appear in the main text to support the reconstruction claims.\n\nOriginality\n\n- Orthogonal-representation losses are not new: Several prior works have already explored orthogonality constraints.\n\nQUESTIONS:\n### Major\n\n- The paper would benefit from a more explicit argument for why the proposed update rule is genuinely Hebbian. How, in concrete neural terms, could a circuit compute the off-diagonal elements of $ XX^\\top$, $ YY^\\top$, and $Y^\\top Y$ using only pre- and post-synaptic signals and perhaps a local inhibition path? If some steps still rely on non-local information, please identify them and outline what additional mechanisms would be required to make the scheme fully Hebbian. This also includes a local implementation of the auxiliary projections.\nA clear, mechanistic explanation of these points would strengthen the biological-plausibility claim and would prompt me to raise my overall score to Accept.\n\n### Minor\n\n- Please explain why you expect the model not to suffer from catastrophic forgetting. An ablation that isolates each loss term’s contribution could help identify the mechanism.\n- Lemma 4.1 assumes that $M<N$. This is not true in many model architectures researchers might consider (and in the brain, eg generative mdoels). How does the model performance change if $M\\approx N$ or $M> N$?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper rethinks the Hebbian learning rule and proposes an improved unsupervised learning method called SPHeRe. SPHeRe simplifies the equivalent loss function of the Hebbian OJa’s rule and introduces auxiliary blocks and orthogonality constraints. The paper has proved that optimizing the proposed loss function is equivalent to finding the best low-dimensional projection of the input data. SPHeRe achieves the current best classification accuracy among unsupervised synaptic plasticity methods and performs well in continual learning and transfer learning.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\nThe paper build a bridge between the Hebbian rule and a clear and optimizable loss function. The proposed SPHeRe loss function is simple but has explicit mathematical significance (seeking the optimal low-dimensional projection). This provides a solid theoretical foundation and development perspective for Hebbian-like rules that originally lacked clear optimization objectives. \n\nWeakness\n1.The method proposed in the paper reduces the locality constraints of Hebbian-like rules, where its learning rule depends on the introduced auxiliary blocks for gradient backpropagation to update the backbone network layers. Although SPHeRe is trained layer by layer on the backbone network, it actually uses backpropagation at the block level.\n2.The core theory of the paper is derived from linear transformations, but it has been extended to nonlinear networks. The authors argue that due to the universal approximation ability of neural networks, they have the capacity to learn the theoretically optimal solution. However, even though the authors have empirical validated this through experiments in the appendix, it does not constitute a strict theoretical guarantee. Moreover, the introduction of the auxiliary network has improved final performance, possibly allowing the network to learn more powerful features. \n3.Table 2 shows that using L_{SPHeRe} alone is less effective than the original L_{oja} loss, and the paper does not adequately discuss and explain this phenomenon. \n4.The ablation experiments in the paper demonstrate the effectiveness of the auxiliary block, but the authors have not analyze in depth how its structure affects the results.  \n5.In the caption of Figure 2, “10^3” should be “10^{-3}”.\n\nQUESTIONS:\n1.The target of the orthogonal loss in the paper is applied to the auxiliary feature Z. Is this effective for the high-dimensional feature Y?\n2.Can the SPHeRe method be combined with self-supervised learning to achieve better results?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a similarity-based representation learning approach. This similarity measure allows the 'inputs' of a module to be nonlinearly transformed before applying the similarity measure. The authors also add some orthogonality constraints on intermediate-layer neurons. This approach with multiple similarity blocks/layers generates representations that are good for supervised learning, but also do well for continual learning and transfer learning. The representations learned also could be used for an auto-encoder.\n\nSTRENGTHS AND WEAKNESSES:\nFinding a similarity-based multilayer network where different layers learn more abstract features has been a goal for many researchers, especially those interested in biologically plausible networks. It seems that the authors have made some progress there. Compared to some other unsupervised methods inspired by Hebbian learning, the authors claim that they have better performance.\n\nOne major complaint I have is that the paper is written as if there has been not much work previously done on similarity-based objectives and Hebbian learning. It completely ignores works like Pehlevan and Chklovskii, Asilomar, 2014, Pehlevan, Hu, and Chklovskii, Neural Computation, 2015; Pehlevan and Chklovskii, NeurIPS, 2015; Pehlevan, Sengupta and Chklovskii, Neural Computation, 2017; Obeid, Ramambason, and Pehlevan, NeurIPS, 2019. Some of these previous works were mentioned in the Nokland and Eidnes paper the authors cite. Contrasting with these works gives a better appreciation of authors' innovations. Also, some of the references to arXiv papers should be changed to their published version. For example, Saha et al is in ICLR 2021 and the Journe et al paper is in ICLR 2023.\n\nIt might also be helpful to be explicit about how the labels are being used. My impression from reading the paper is that representation learning does not use labels at all, and I imagine only later some shallow supervised learning is done using the learned representation. Some similarity-based methods like HSIC Bottleneck (Ma et al, AAAI, 2020; Pogodin and Latham, NeurIPS, 2020), use both inputs and labels while training networks. Hence a clear description of the method may help.\n\nQUESTIONS:\nClarification of how the labels were used in their evaluation would help a broader set of readers, some of whom may come from computational neuroscience.\n\nContrasting with previous work would place the work in the proper context. In that vein, could the authors comment on how their training relates to Hebbian learning as understood in neurobiology?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}