Title: Stochastic Forward-Forward Learning through Representational Dimensionality Compression
Venue: NeurIPS 2025
Accepted: yes
Keywords: Forward-Forward Algorithm

Effective Dimensionality

Energy-Based Learning

Neural Coding

Representational Geometry
Abstract: The Forward-Forward (FF) learning algorithm provides a bottom-up alternative to backpropagation (BP) for training neural networks, relying on a layer-wise "goodness" function with well-designed negative samples for contrastive learning. 
Existing goodness functions are typically defined as the sum of squared postsynaptic activations, neglecting correlated variability between neurons. In this work, we propose a novel goodness function termed dimensionality compression that uses the effective dimensionality (ED) of fluctuating neural responses to incorporate second-order statistical structure. Our objective minimizes ED for noisy copies of individual inputs while maximizing it across the sample distribution, promoting structured representations without the need to prepare negative samples. We demonstrate that this formulation achieves competitive performance compared to other non-BP methods.  Moreover, we show that noise plays a constructive role that can enhance generalization and improve inference when predictions are derived from the mean of squared output, which is equivalent to making predictions based on an energy term.  Our findings contribute to the development of more biologically plausible learning algorithms and suggest a natural fit for neuromorphic computing, where stochasticity is a computational resource rather than a nuisance. The code is available at https://github.com/ZhichaoZhu/StochasticForwardForward.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors develop a technique to enhance non backpropagation based learning, through the use of contrastive samples that are generated without supervision. Essentially through noise, regularization losses and dimensionality reduction.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\nThe paper presents a well-motivated and biologically plausible alternative to backpropagation by introducing a novel goodness function based on effective dimensionality (ED). This function captures second-order statistics of neural responses, promoting compact, class-consistent representations and diverse, globally discriminative features. They achieve this without requiring labels or manually crafted negative samples. The use of noise as a constructive mechanism to generate contrastive samples aligns well with principles from neuroscience and neuromorphic computing, providing a practical and theoretically grounded approach for unsupervised learning. Empirical results across MNIST, CIFAR-10, and CIFAR-100 show that the method achieves competitive performance compared to other non-backpropagation algorithms. Furthermore, the inclusion of ablation studies and orthogonality analyses helps support the framework's core claim, that optimizing representational structure via ED leads to more robust and linearly separable features.\n\n**Weaknesses:**\nDespite its conceptual elegance, the paper's empirical validation remains relatively limited. The model underperforms compared to equilibrium-based learning and backpropagation, particularly on more complex datasets, and it is tested only on shallow architectures. The reliance on manually chosen hyperparameters (e.g., the projection dimensionality and trade-off weight alpha) and the lack of large-scale or real-world benchmarks weaken the claim of generality. Additionally, while the theoretical motivation is sound, the exposition of experimental results lacks a clear narrative, and some of the key concepts, like the compression ratio and its link to performance, are underexplored in the broader context of learning dynamics. A more direct comparison to EBL and a clearer explanation of what is gained relative to prior methods would further strengthen the contribution.\n\nQUESTIONS:\nn/a\n\nLIMITATIONS:\nDiscussed in the conclusions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes a local self-supervised learning method. The learning minimizes a loss applied layer-wise, and gradients are blocked across layers. The loss measures the dimensionality of the data using the effective dimension, which is the ratio between the square of the sum of eigenvalues of the covariance matrix, divided by the sum of the squared eigenvalues. The loss is the difference between two forms of ED, one applied to a noisy sample from the same input, and the second is applied across the batch dimension. The dimensionality is therefore maximal for different images in a batch, but minimal with a noisy version of the same image. The method is tested on image classification datasets (with a post-training linear classification accuracy). It performs relatively well in comparison with other layer-wise learning rules. It is reported that the kernel weights become orthogonal through the learning process\n\nSTRENGTHS AND WEAKNESSES:\nWhile there have been many different approaches to layer-wise learning rules, I have seen ED being used to analyze data but not yet as a loss function. It is elegant and mathematically sound. \nThe learning algorithm is mathematically sound and well-explained, and it is great that it uses a simple loss function. The performance is tested on reasonable benchmarks, and the reported performance seems logical and reproducible. \n\nIn terms of weaknesses:\n\n- It is unfortunate that the loss function is not tested (with identical architecture and with similar analysis) against alternative loss functions. For instance, I wonder how the result compares to the same architecture but compared with the (more common) InfoNCE loss function. Methods like InfoNCE, which are usually applied end-to-end, can also be applied block-wise [1] and layer-wise [2,3], and vice versa, ED could be applied ETE. So it would be useful to know how ED compares to these methods.\n\n- One of the motivations is to make a \"non-contrastive\" algorithm. But I suspect that it is secretly a contrastive algorithm: the term ED_d applied on the batch dimension performs comparison on the batch dimension. In fact, it seems that the algorithm does not work for alpha=1, where this term is dropped. So I suspect that the contrastive component is needed. I do not think it is crucial for the message of the paper to claim that the method is non-contrastive, though.\n\n[1] https://arxiv.org/abs/1905.11786\n[2] https://arxiv.org/abs/2010.08262\n[3] https://www.nature.com/articles/s41593-023-01460-y\n\nQUESTIONS:\nI wonder what the specificity of the chosen loss function is in comparison with other learning algorithms. The algorithm is prominently compared with EBL, but EBL provides an alternative to gradient descent as a whole and is a supervised algorithm. What is the high-level intention of doing this comparison ? References like [1,2,3] are in the sense more similar to the proposed algorithm. If the key question is to provide a bio-plausible learning algorithm, how would he gradient descent step of the ED look in a simple case, would this indeed be apparent to a bio-plausible learning mechanism?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article proposes a variation of the forward-forward approach. In the original method, the variance of layer activations is used as a goodness-of-fit measure, with unbounded growth controlled by subtracting the objective evaluated on negative samples. To address the challenge of generating such negative samples, the current article introduces an alternative goodness-of-fit function based on effective dimensionality, defined as the ratio of the trace to the Frobenius norm of class-conditioned correlation matrices of hidden layer activations. The article reports empirical results on MNIST, CIFAR-10, and CIFAR-100 datasets.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- Addressing the negative sample issue of the original forward-forward algorithm is valuable.\n- The performance on the benchmark datasets provide improvement over the original forward-forward algorithm.\n\n## Weaknesses\n\n- The article lacks clarity in the description, explanation, and motivation of the proposed approach. The mathematical notation—particularly concerning class conditioning—is imprecise and difficult to follow, which obscures the formulation of the proposed objective. The authors should first clearly define the class-conditioned objective and then demonstrate how it is transformed into its final unsupervised form, explicitly presenting the full loss function across all layers. Additionally, the article should provide a clear rationale for why this objective is expected to be effective.\n\n\n- The reason why forward-forward algorithm introduced is to provide a biologically plausible explanation for biological networks. It is not clear whether the proposed effective dimensionality loss leads to a biologically plausible structure and local learning rule.\n\n- Although the performance is better than the original forward-forward algorithm it is not better than other biologically plausible algorithms.\n\nThe article's presentation requires major improvement. Some examples are:\n\n* Section 2: Background work on forward-forward based and other backprop alternatives are not properly explained.\n\n* Page 2, Line 78-79: DFA stands for  Direct Feedback Alignment. Besides the statement \" ...  each block to align its output directly with the final target\" for DFA is not very accurate.\n\n* Page 3, Figure 1 caption (b): Incorrect sentence \"Within-class responses that colored blue and orange resepectively are tightly clustered, leads to a low ED\"\n\n* Page 3, Figure 1 caption (b): There is no gray ellipse visible? is it the  dashed circle?\n\n* Page 3, Figure 1 caption (d): Incorrect sentence \"The statistic factors that affecting ED\"\n\n\n* Page 3, Section 3:  The tuning curve has not been properly defined before its use. The authors could use conditional distribution based mathematical notation here.\n\n* Page 3: The authors could provide more motivation about the loss in (1). They can describe why it favors lower dimensions through the norm inequality on the eigenvalues.\n\n* Page 3, Section 3: Equation (1) should be first defined based on label's as it measures the effective dimensionality of the based on the conditional (conditioned on the labels) distributions. This needs to be mathematically precise, so that unsupervised version later defined could be better explained in reference to this definition.\n\n* Page 4: the loss functions in (2)-(4) are not clearly described, and why they should work is not clearly explained.\n\n* Page 14, Figure S1 caption: incorrect sentence \" randomly drop its pixes\".\n\nQUESTIONS:\n* Section 3, Line 99: \" a population of neurons where each\n100 neuron is selectively responsive to a particular type of stimuli\" is this linear separability condition envisioned for neurrons at all layers, or they are mostly for the final layer neurons?\n\n* Figure 1 (c): Is ED in these figures for the unconditional distributions, or class condition distributions? why should ED of class conditioned distributions depend on the center locations of class clusters?\n\n* What does ED_c in (2) measeure and what does ED_d in (3) measure? why should they be a goodness of fit functions? how do they guarantee that the class conditioned  distribution domains have low effective dimension?\n\n* Does the proposed loss lead to local learning rule? if not what is its advantage over the backpropagation algorithm?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce, in the paper \"Stochastic Forward-Forward Learning through Representational Dimensionality Compression\", a novel variation of the forward-forward learning algorithm that expands upon the formulation of the layer-wise goodness. The new goodness function is proposed as a measurement of effective dimensionality which goes beyond the mere magnitude goodness that is used typically. This choice is well motivated and allows learning without the need for negative samples and instead cleverly apply noise. Their derivations are supported by experimental evidence on MNIST, CIFAR-10, and CIFAR-100.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* As far as I know the literature, this dimensionality compression goodness function is novel and original in this setting. Hence, an insightful extension that leverages neuron interactions.\n* Clever introduction of noise to enable learning and circumvent the need for negative samples.\n* The authors provide sufficient insight in the form of analysis and ablations.\n\nWeaknesses:\n\n* The method effectiveness is demonstrated only on shallow architectures, especially considering that the non-BP methods from the literature consider much deeper networks.\n* Equilibrium Propagation: The authors note their performance is behind to that of equilibrium propagation, commenting that it might be a result of deeper networks and increased computation. Based on this the comparison to equilibrium propagation is quite indefinitive and should ideally be tested in a control to compare on equal grounds.\n* The dependency on noise could also be harmful. Indeed, what if the neuromorphic device that would be considered offers a lower level of noise than what is needed?\n\nQUESTIONS:\n* Can the approach remain effective and competitive for deeper architectures? I.e. also in comparison to the networks used in equilibrium propagation?\n* Why are the authors not comparing to layer-wise contrastive methods such as successors of [1]?\n* There is no indication about the number of orthonormal basis vectors employed for the pre dimensionality reduction. This is a bit misleading and never is mentioned to sufficient capacity, which could be misleading to readers.\n* Can the method be applied to biologically more realistic models both on the neuron and architecture level (recurrent)? \n* What would be the effect of changes in the employed noise distribution introduced by dropout?\n\n[1] Löwe, S., O'Connor, P., & Veeling, B. (2019). Putting an end to end-to-end: Gradient-isolated learning of representations. Advances in neural information processing systems, 32.\n\nLIMITATIONS:\n* Primarlily employing shallow architectures: The authors are encouraged to investigate deeper architectures as this is necessary to properly present the method and strengthen the message\n* Although the use of noise is a positive point, it becomes at the same time a question whether the method can work in lower noise regimes too.\n\nThe authors are also encouraged to automatically check for typos.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}