Stochastic Forward-Forward Learning through Representational Dimensionality Compression
Authors: Zhichao Zhu
YANG QI
Hengyuan Ma
Wenlian Lu
Jianfeng Feng
Venue: NeurIPS 2025 poster ()
Keywords: Forward-Forward Algorithm
Effective Dimensionality
Energy-Based Learning
Neural Coding
Representational Geometry
Primary area: general_machine_learning
TL;DR: We propose a second-order goodness function based on effective dimensionality for Forward-Forward learning, eliminating the need for negative samples and showing that noise enhances performance and inference robustness.

Abstract: The Forward-Forward (FF) learning algorithm provides a bottom-up alternative to backpropagation (BP) for training neural networks, relying on a layer-wise "goodness" function with well-designed negative samples for contrastive learning. 
Existing goodness functions are typically defined as the sum of squared postsynaptic activations, neglecting correlated variability between neurons. In this work, we propose a novel goodness function termed dimensionality compression that uses the effective dimensionality (ED) of fluctuating neural responses to incorporate second-order statistical structure. Our objective minimizes ED for noisy copies of individual inputs while maximizing it across the sample distribution, promoting structured representations without the need to prepare negative samples. We demonstrate that this formulation achieves competitive performance compared to other non-BP methods.  Moreover, we show that noise plays a constructive role that can enhance generalization and improve inference when predictions are derived from the mean of squared output, which is equivalent to making predictions based on an energy term.  Our findings contribute to the development of more biologically plausible learning algorithms and suggest a natural fit for neuromorphic computing, where stochasticity is a computational resource rather than a nuisance. The code is available at https://github.com/ZhichaoZhu/StochasticForwardForward.

Stochastic Forward-Forward Learning through
Representational Dimensionality Compression
Zhichao Zhu
Yang Qi lang-qi@fuden edu. cn
zhi mhiihha_zhugee www.znuuuuuaneeeem edu. cn
Hengyuan Ma
Wenlian Lu
Jianfeng Fengl23*
hangyuanma210mm 10m fudan edu. cn
wenl renlian@fudan. edu, cn
j fffengffudan. edu. cn
Abstract
The Berward-Forward (FF) learning algorithm provides a bottom-up alternative backpropagation (BP) for training neural networks, relying "goodness" function with well-designed negative samples for contrastive learning. Existing goodness functions typically defined the sum of squared postsynap- tic activations, neglecting correlated variability between neurons. In this work, we propose novel goodness function termed dimensionality compression that uses the effective dimensionality (ED) fluctuating neural responses to incorporate second- order statistical structure. Our objective minimizes ED for noisy copies of individ- ual inputs while maximizing it across the sample distribution, promoting structured representations without the need prepare negative samples. We demonstrate that this formulation achieves competitive performance compared to other non-BP methods. Moreover, show that noise plays a constructive role that can enhance generalization and improve inference when predictions are derived from the mean squared output, which equivalent to making predictions based on an energy term. Our findings contribute to the development of more biologically plausible learning algorithms and suggest a natural fit for neuromorphic computing, where stochasticity is computational resource rather than nuisance. The code is avail- able stlhtts::/// // thub. com aon/ http:///ttttochtiiciccco wor.eaasstooooorrFFooooooooooooe orward
layer- wise
Introduction
Despite being central to the success of traditional deep learning, backpropagation (BP) poses chal- lenges for on-chip learning in neuromorphic systems, as requires global error signals and symmetric weight transport, both of which are widely regarded biologically implausible and difficult to imple- ment efficiently on neuromorphic hardware [15].. Therefore, the forward-forward (FF) learning algorithm proposed by Hinton [6] provides an elegant bottom-up alternative that each layer learns independently by maximizing a "gooddess"" measure of its activations, climinating the need for error backward propagation.
Although FF learning conceptually simple, its success relies on generating high-quality negative samples, which is highly task-specific and presents a significant practical challenge. Moreover, the goodness function the original FF learning the sum of squared postsynaptic activations and does not account for the role of noise, a ubiquitous feature in both biological neural systems 7 and
"Instittte of Science and Technology for Brain- Inspiredd Intelligence, Fudan University, Shanghai 200433, China. Key Laboratory of Computational Neuroscience and Brain-Inspinn Intelligence (Fudan University), Ministry of Education, China. 'MOE Frontiers Center for Brain Science, Fudan University, Shanghai 200433, China. Ji Hua Laboratory, Foshan 528200, China. *Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

neuromorphic computing hardware 19 10 Substantial evidence suggests that neuronal correlated variability could carry rich information |1:1 and can be considered a computational resource [[4] 18]. Leveraging noise and incorporating postsynaptic neuron correlation into FF learning could potentially lead to more biologically plausible and aardware-adapted alternatives.
In this work, we extend the FF framework by introducing a novel goodness function termed di- mensionality compression that is derived from the effective dimensionality (ED) [9] of neuronal responses. Essentially, ED depicts the second-order statistical structure of neural responses, which can be used to describe representational selectivity. Within-class responses should have low ED, indicating that samples are tied to a specific variable, while across-class responses should have high ED to avoid representation collapse. Instead of generating negative samples, we introduce noise to create isologues of the samples, with the objective of minimizing ED for each sample while maximizing it for all inputs. This promotes locally structured representations that are both compact and discriminative, bypassing the need for negative samples and fitting smoothly into the FF training pipeline. The numerical experiments on standard datasets demonstrate that the proposed method can achieve performance comparable to that of other non-BP methods. Furthermore, this approach inherently recommends using the mean of squared outputs rather than just averaging them for prediction, which means that variability in the outputs can also carry the information interested and can be interpreted as an extension of energy-based learning (EBL) 220 [2]. Collectively, our findings provide both theoretical link between dimensionality and representation learning and practical direction toward biologically plausible, noise-driven computation.
2 Background and related works
We begin by reformulating the learning problem of a network in classification tasks. Given an input- target pair (X, 1) from 7' targets, the input is processed through L blocks with X = f(( ,09) and ( X. A linear classifier W then produces class scores Y = WT and denotes Y, the score for the target i. 'hhe question posed is how to modify the model parameters in the absence of BP, while still guarantecing that the output Yú associated with the true target t achieves maximal discrimination.
Since learning performance is ultimately assessed through linear classifier, it intuitive to assume that if each block produces more discriminative representations, stacking such blocks can improve overall classification. For example, direct feedback alignment (DFA) (33 demonstrates that deep networks can learn effectively without requiring symmetric feedback connections. Instead, DFA delivers global error signals to each layer through fixed random feedback pathways, showing that precise gradient transmission is not essential for credit assignment.
While DFA relaxes the need for symmetric feedback by transmitting error signals through random fixed pathways. EBL (20 eliminates the requirement to propagate explicit errors altogether. Instead, EBL frames learning as minimizing an energy function that depends on local neuronal interactions. During training, the network first clamps the input and fixes the output to the target, allowing neural activities to evolve toward an equilibrium state that minimizes this energy. Weight updates are then computed locally based on the pre- and post-synaptic activities. However, because the energy landscape of one layer depends on others, finding a global equilibrium in deep architectures can be computationally demanding and slow to converge.
One can simplify the interaction between blocks by training the blocks independently from the bottom up. Greedy InfoMax (GIM) 24 optimizes individual blocks by encouraging each block to preserve information about its input, which can be interpreted as reinforeing each block learn slow features (25) from its inputs, allowing more scalable unsupervised representation learning without global backpropagation. Hinton's 's FF algorithm[6] offers simpler energy-based approach, using a scalar goodness function to distinguish 'positive" and 'negative" samples in each layer. Initially, this function, similar to the energy function in EBLs, is the square of postsynaptic neuronal activities. However, it is necessary to use high-quality negative samples for contrast learning, as optimizing the network with this function alone will fail because it encourages maximum neuron activation. Hinton also noted that binary objectives consisting of one "positive" and one "negative" sample inject limited information per update, which will slow the rate at which meaningful structure can be encoded the weights.

---

Recent advancements in FF-isppired algorithms have removed the requirement for negative samples by employing a supervised learning approach. The Cascaded Forward (CaFo) model [26] enhances the FF algorithm through the integration of convolutional layer blocks, allowing these blocks to independently generate label distributions without requiring negative samples. Papachristodoulouu et al. [27] further subdivided the channels within the convolutional layers into T groups, optimizing the mean activation of each group to be more pronounced for certain classes, thus improving linear separability. While these modifications can be effective, implementing such strong label-based supervision in a biologically brain can be challenging, as necessitates knowing a label for each input pattern.
Hebbian learning is acknowledged as biologically realistic mechanism for synaptic plasticity, enabling weight adjustments in an unsupervised manner. However, the principal obstacle lies in its limited efficacy, particularly when deployed large networks and more complex datasets. Although Journé et al. (28 and Nimmo and Mondragon [29] showed that combining unsupervised Hebbian learning with well-designed winner-take-all (WTA) mechanism yields satisfactory results, their success involves complex softmax activation functions combining with several optimization tricks. These complications hinder a deeper understanding of the core question: What should a neural block learn when limited to local information and how to define the goodness of neuronal response for goal achievement?
In this study, we aim to answer these questions by leveraging the continuous, second-order structure in neural responses, providing a more efficient and biologically grounded alternative to contrastive objectives.
3 Effective dimensionality goodness function
Historically, the effective dimensionality (ED) is introduced to describe the equivalent number of orthogonal dimensions that would produce the same overall pattern of covariation of a set of correlated variables [19]. In practice, ED is typically based on the covariance matrix of the data [30] 31. However, neural or network representations where the mean activity itself encodes task or stimulus information, it is more appropriate to consider the uncentered second moment as it reflects both the signal (mean configuration) and interaction structure (correlation). Therefore, in this work, we define
tr(E[x(07 x x0)
(E
X E X
ED(X"))
| VZ X
where X; are the eigenvalues of the uncentered covariance matrix of neural activity X²), If X'() has zero mean, this definition reduces to the standard ED based on the covariance matrix that gives good indication of the number of principal components needed to capture most of the variance of X!)) (Fig. 1a). When then mean is nonzero, ED based on ( x not only measures the diversity of local variations but also incorporates the global structure imposed by the mean configuration. By varying the mean and covariance of awo-dimensional distribution, we illustrate that ED is maximized for distributions whose dimensions are uncorrelated with identical mean and variance but will reduce otherwise (Fig. [D).
In particular, (Xx (D) is also a valid representation of the energy landscape of the (-th block. In conventional EBL, the energy term is typically defined as tr tr(Eix(() x(") that quantifies the total expected energy and the overall magnitude of population activity while neglecting the correlation structure between neurons. In contrast, ED incorporates the full spectral structure of TE[X ( x(), thus providing a correlation-au awwee measure of the energy landscape. In this sense, ED-based learning can be regarded as extension of EBL that preserves the connection to energy while emphasizing the structural geometry of the representation.
To illustrate this concept, we start by considering a population of neurons where each neuron is selectively responsive to particular type of stimuli (Fig. (Tb) and where the response of the neuron conditioned on specific input is function of some stimulus features s, commonly referred to as a tuning curve in neuroscience [12. Consider a representational space spanned by two neurons in Fig. D that are selectively activated for a specific feature informative about the input class. Then, neural responses to inputs within the same class are expected
more aligned along a specific direction (Fig. Tt). In such ideal case, the ED for within-clas responses (blue and orange
3

---

Nocran
Fiiing name
GD 2.0
Figure 1: Assessing the informativeness of neuronal responses through effective dimensionality
(ED). a. Illustration of what ED quantifies for a zero-mean Gaussian distribution. The eigenvalues X computed from the uncentered second moment. ED approaches 1 when variance is concentrated along a single principal direction (left) and increases toward 2 as variance becomes isotropic (right). b. Influence of mean and covariance on ED. In the left and middle panels #l2 0, of o 1 and
J22 = M1 0, o 5 are fixed while varying j41 and of respectively. In the right panel, 0, o² are fixed while varying the correlation coefficient P12- c. Example tuning curves showing neurons selectively responsive to some tnteeory-informatte categorical information. d. ED as a measure of class separability in a two-dimensional response space. Points represent noisy samples from two classes (blue and orange). Within-class responses form clusters with low ED, whereas their mixture (whose uncentered covariance represented by the dashed gray ellipse) exhibits higher ED, reflecting representational diversity.
features, forming a population code that encodes
dots) is expected to be close to one, indicating that inputs can be well explained by fewer variables compared the dimensionality of the representational space. In contrast, the ED for the mixed responses across classes (whose uncentered covariance is represented by the dashed gray circle) is expected to approach two. Therefore, inputs belonging different classes are explained by unique variables orthogonal to each other, leading to linearly separable representation. The learning goal of ED-based learning thus straightforward: we should minimize ED for responses within each class to promote consistency and robustness, while maximizing ED across all inputs to ensure diversity and discriminability.
The problem is, the label information is infeasible in the unsupervised setting. To compute ED in the absence of true class labels, we introduce noise into the computing process, which is an essential characteristic of both biological systems and neuromorphic hardware 7
14
17
I8 10
A crucial point is that introducing moderate noise does not hinder the identification of its class (Fig. A1). In other words, the essential features that form the concept of its class remain intact. This strategy enables unsupervised learning of robust and discriminative features, but bypasses the need for explicit class labels or negative sampling. Formally, let X' E BxF be a batch of data samples and use dropout create its noisy copies, X, by randomly setting the elements in X' to zero with probability p. For the /-th block, we denote its output (i.e. the corresponding neural responses) ( and x(") the neural responses a particular input sample in the batch. The objective function for each block is then consisting of a consistency term and a diversity term, defined as:
ED
Σ ED x!"
(2)
and
ED ED [IX
(3)
where B is the batch size, B[x() denotes the averaging noisy copies the input batch. ED describes the mean ED of neural responses to noisy realizations of individual input samples and

---

Phose tnering
Phane Iraring
Exteiitorm
Outpur ET1
Apc
Creating nony samples
ED,
E2 (x2)
Reguboriootoo on
Xl Drogroit
Figure 2: Network architecture and training pipeline. The first dropout layer generates N noisy variants per imput and remains active during inference, while the dropout in the linear classifier is used only for regularization during training. Batch normalization (BN) layers stabilize inputs and contain no trainable parameters. Each convolutional block includes a Fixed Orthonormal Projection (FOP) module that projects its output onto subspace with pregenerated random orthonormal basis before computing the dimensionality compression loss L. Training proceeds in two phases: (1) Each convolutional block is trained layer- wise for epochs using the proposed loss function L. (2) The convolutional blocks are then frozen, and a linear classifier is trained for 60 epochs using cross-entropy loss, where prediction score for each sample is computed the mean of squared classifier outputs over the noisy variants. The overall architecture and training pipeline are consistent across all experiments, except for the classifier's input dimensionality, which varies by dataset.
captures the noise variability, whereas ED2 describes the ED of the response distribution in different input samples and captures the variability of the data.
The dimensionality compression loss then formulated by merging the two components, with trade-off parameter that is assigned default value of 0.5.
I. = aED, (1 o))
Conceptually, minimizing ED; implicitly suggests a WTA dynamic 28 more distinct some directions compared to others, leading more compact representation. In contrast, maximizing ED, prevents the collapse of representation as it reinforces the overall responses that can be explained by as many directions as possible. In this way, the network is encouraged to features, leading to more linearly separable representation.
29 so that the responses are
learn
Notably, this learning objective also motivates modification of the inference procedure. Let Y; be the output of the linear classifier for noisy input sample X;. Rather than averaging the output, we propose using [E] the classification score. This approach is equivalent selecting the neuron that has the minimum energy, as inline with EBL.
4 Experimental design and results
Architecture, As shown in shown in Fig. we use a network architecture similar to that used by Journé et al. [28] and Nimmo and Mondragon 29 except the activation function chosen to verify the effectiveness of the proposed goodness function. The network consists of three convolutional blocks followed by a fully connected layer for classification. The first block has 96 channels, and the number of channels increases by 4 for each subsequent layer. Fach channel is regarded as a neuron due to weight sharing, and the corresponding feature map is treated as samples drawn from an unknown distribution conditioned on the input. See Appendix A1.I for details.
Datasets and data preprocessing. We use the 32] CIFAR-10 and CIFAR-100 datasets 33 to verify the effectiveness the proposed goodness function. For MNIST, we only use random crop for data augmentation. For CIFAR- 10 and CIFAR-100. whitening and then random crop and random horizontal flip for data augmentation.
first apply zero phase component
Training Pipeline. We divide the training process into two phases. In the first phase, convolutional blocks are trained using the proposed goodness function layer-wise for 3 epochs. Specifically, for each epoch, we train the first block with the proposed loss L, fix it, and then train the next block.
5

---

Table 1: Comparison of validation accuracy (%) of the proposed method against recent non-BP and FF-inspired approaches, along with BP, that adhere to our architecture across various datasets. The performance of the proposed method are estimated over 5 runs.
Validation Accuracy (%)
Method
MNIST
CIFAR1O
CIFARI00
BP
99.33 + 0.04
82.50 = 0.09
61.28 + 0.25
Original FF 66
98.73
59
CaFo FF 26
98.95
69,49
42.13
CwC FF [27
99.42 - 0.08
78.11 0.44
51.32
DFA 23
98.98 + 0.05
73.10 0.50
41.00 + 0.3
Soft Hebbian
28
99.35 + 0.03
80.31 = 0.14
56.00
Hard Hebbian 29
76
GIM 24
99.29 = 0.03
78.19 = 0.34
50.09 + 0.45
EBL 21
99.56
89.6
65.8
Proposed method
99.31 + 0.07
76.96. + 0.73
53.29 + 1.02
Reimplementation results.
This process is repeated for all blocks. Given that the number of neurons far exceeds both the number of samples and the number of classes, direct estimation of ED may become unreliable due the sparsity of samples. To address this, we propose projecting the output of each block onto a nower-dimensional subspace using a randomly generated set of orthogonal basis vectors before computing the dimensionality compression loss. This projection operation is expected to reduce the estimation variance while preserving the structural properties of the representation. We gradually reduce the projection dimensionality until it aligns with the number of output classes.
In the second phase, convolutional blocks are frozen, and we train the linear classifier for 60 epochs with the standard cross-entropy loss, where the prediction outputs over the noisy variants. The best validation accuracy is reported as we care more about the upper bound of the performance. All experiments utilize NVIDIA RTX 3090 GPU and an Intel Xeon(R) Gold 6226R CPU, as detailed in the Appendix A1.2
is based on the mean of squared
4.1 Proposed method achieves comparable performance with other non-BP methods
As the development of non-BP methods is still in its infancy, researchers use different network architecture, training protocols, and datasets to evaluate the performance of their methods, making it difficult to make a fair comparison. Therefore, we use other non-Bl BP methods' results reported in their papers for comparison. The results are summarized in Table [
We find that the proposed method achieves comparable performance with other non-BP methods on MNIST and CIFAR- 10 datasets, and show ability learn useful features in CIFAR100. In the realm of FF methods, the work by Hinton [6] is seen as the baseline. CaFo FF [26) modifies the initial goodness function by independently training each convolutional block for classification. Our method consistently surpasses these two methods in all datasets, as well as the DFA proposed by Nøkland 23 For CwC FF (27 which revises the convolutional framework by dividing the channels into C groups for supervised learning, our method can match its performance without labels and specific requirements for network architectures. Given that our network architecture mirrors that Hebbian learning combined with soft or hard WTA mechanisms 228 29), our results are directly comparable to those works and attain similar performance levels. In addition, as previously noted, optimizing ED naturally introduces competition mechanism. which can be considered principle that these methods strive to achieve. The GIM (4 also achieves performance similar to that of our method when reimplemented using this network architecture (see Appendix Ap1.)
general learning
For EBL methods, Scellier et al. 21 conducts a comparative study existing EBL approaches, utiliz- ing a five-layer convolutional Hopfield network to demonstrate that gquliibrium-backiee backpropagation (0) achieves superior performance. Although our findings are inferior compared theirs, the discrep- ancics may be attributed to the deeper network used and the ability of EBLs to leverage top-down

---

0.18
100
0-
Mean bernals
80
2=00
(8)
0.6
0 14
0.2
0:12
0
0.6
MNIST
CIFAR100
00 0.4 0.6 0.8
Figure 3: Effect of the trade-off factor a on weight optimization. a. Visualization of the first-layer convolutional kernels trained CIFAR-10 R- under different values of Each column shows the top 10 channels ranked by the standard deviation of their weights trained with different a. Cosine orthogonality score (COS, blue line) and mean standard deviation of first-layer kernels (orange line) as functions a. higher COS indicates greater diversity among channels' weights. c. Classification accuracy comparison for 0.0 and 0.5 (default). Error bars denote standard deviation across 5 independent training runs.
information for optimization. Unfortunately, adding more than three convolutional blocks in experiments leads to performance drop, which we attribute to the difficulty in preventing the collapse of class-specific features after a stack of highly nonlinear transformation, technical problem that needs to be solved in future works.
In contrast to BP under the same architecture and training protocol, the proposed method performs similarly on MNIST. However, the disparities widen with CIFAR- 10 and CIFAR-I00, frequent issue for non-BP methods. Since the power of BP comes from its ability to optimize network globally so that the final output ultimately meets the task's requirements, leaves room for future work to explore how top-down and bottom-up learning can be combined in more biological plausible manner, thus making the implementation on hardware more friendly.
Since one distinct feature of our method is the introduction of noise during training, we also conduct experiments by varying the noise strength (changing the dropout rate p) and sampling size assess its impact performance in the Appendix A1.3 These results demonstrate that both noise strength and sampling size significantly influence model performance. In general, moderate noise levels and appropriate sampling sizes yield optimal results.
In summary, the proposed method achieves performance comparable to that of other non-BP methods and even outperforms some of them. Although we currently are unable to demonstrate the efficiency of our method in deeper networks more complex datasets, the use of noise and the simplicity of the learning objective make it a promising direction to explore how noise can facilitate learning in a biologically plausible manner.
4.2 Compressing dimensionality leads to orthogonal weights
We next train the first convolutional blocks CIFARI0 by varying from to I with step increase of 0.1 to investigate the effect the trade-off factor a on weight optimization. Each column Fig B displays the weights of the ten channels with the highest standard deviation in their weights among all channels trained under a given a. The plots indicate a transition point at a = 0.5. When a > 0.5 applied for training, the channel weights collapse into similar pattern, suggesting the limited capacity of the layer to learn varied combination of input features. In contrast, when a S 0.5. diverse weight patterns emerge.
To quantify this observation, we compute the cosine orthogonality score (COS) of the kernels, defined
as
1 COS
Σ
(w;, wj) 11:
(5)
K
wy

---

100
(11)
Sup-sonpting
Accurscy (%)
Direct foword
Nome
Random
60
20
0
MN
CIFAR10 CIFAR100
MNIST
CIFAR1D
CIFAR100
MNIST
CIFAR10
CIFAR100
Figure 4: Factors affecting task performance. Classification accuracy under different inference strategies. E|Y²]; proposed method, using the mean squared outputs (energy) based on generated noisy samples as prediction score: E[Y]: uses the mean of outputs as prediction score. Direct forward: standard inference without noise, using raw inputs. b. Accuracy under different training schemes. Unsup: proposed method, where ED. is computed at the instance level based generated noisy samples. sumplimp generated noisy samples are further grouped by class labels before computing ED,,- Sup: computes ED, directly labeled data without the need to generate noisy samples. c. Accuracy under different projection strategies. Graded: block outputs are projected with gradually decreasing dimensions (30-20-10 for MNIST and CIFAR-10; 90-150-100 for CIFAR-100). Fixed: all blocks projected to a constant dimension equal to the number of classes. Random: projected randomly selected dimension per block. None: no projection.
where w; is the i-th channel and K <l=-1) is the number pairs {ij}} with e being the number of channels. As shown in Fig. 3b, the CoS (blue line) is consistently close to one if S 0.5 indicating that the kernels are fully orthogonal to each other. When a > 0.5, the COS decreases significantly and indicates that the weights of different channels tend to be similar, leading to highly redundant feature extraction. This consistent with the observation in Fig. 33 that all the weights of the channels are similar to each other when > 0.5.
This transition point also observed when using the mean standard deviation of the channels (Fig. 3P. orange line). When > 0.5, the average standard deviation decreases substantially, suggesting that the numerical values of the weights channel tend be identical, with low probability of forming distinctive structure for feature extraction. Similar trends are also observed with the same analysis on MNIST (Fig. A2
Clearly, setting a to exceed 0.5, the task would not succeed as the first block would be unable to explore the input's rich features effectively. Hence, simply decreasing ED, cannot facilitate learning. Therefore. we investigate the impact of ED. by assigning = 0.0 and performing the same experiments as outlined in the previous section to illustrate how task performance influenced. As shown in Fig. 33. although depending solely on ED, still results in some learning, the performance inferior the standard configuration, particularly for CIFAR10 and CIFAR100. Consequently, optimizing ED. is an indispensable component for learning.
4.3 Ablation study
In standard deep learning, the inference stage typically deactivates dropout and involves a standard feedforward pass through the network, with the network's output serving the prediction score. In Bayesian neural networks, the inference phase generally utilizes the mean of the outputs for the prediction score. Here, due the stochastic nature of both the goodness function and the training procedure, we advocate using the average energy of the outputs (EE[Y)) as the prediction score. We then compare these three inference strategies by running the second training phase with different prediction scores while the trained convolutional blocks are fixed. As shown in Fig. 4a the performance of the proposed method slightly better than that of the other two methods, indicating that both the mean and the variance can carry information about the labels, while the mean plays the primary role. Interestingly, when we apply t-distributed stochastic neighbor embedding (t-SNE) 34 to visualize the model output as defined by E[Y"] (see Fig. A3). the outputs within the same class exhibit a unique fluctuating direction. Moreover, similar classes are not only close proximity to each other, but also exhibit shared fluctuating directions.
8

---

b
90
14
Iofommatonn (mal))
In
Accursay
0.74
0.72
10
0.70
0.88
10
Figure Layerwise analysis representations after training a. Effective dimensionality (ED) of block outputs projected into a nower-dimensional space. FD2 and ED, are colored by blue and orange respectively and the shades denote one standard deviation of ED across classes. The horizontal dashed line marks the projection dimensionalityy and the black dashed line shows the compression ratio ED,/ED, b. Linear separability of each block's representation, measured by training linear classifier on the output of each block. Information decomposition of classifier outputs from each block, assuming a Gaussian mixture model. We report total mutual information (tot), linearly decodable information/in, and second-orde interaction terms (cor), where lin + cor. Results shown are from a randomly selected model trained on CIFAR-10.
We next investigate whether incorporating label information and grouping same-class samples during goodness optimization improve performance. Since our proposed unsupervised method relies on the generation of noisy variants of each sample, we compare it with two supervised alternatives: Suptsamplin uses labels and applies the same noise sampling as in the unsupervised setting while Sup only uses label information to compute L directly (within-class samples for ED, and batch data for ED&). As shown in Fig. 4p, the unsupervised method slightly outperforms the supervised one, suggesting that the model can learn effectively without labels and achieve comparable accuracy. Notably, noise sampling can improve performance, as Sup sampling is better than Sup. Although such sampling operation increases computational cost, the advance of neuromorphic hardware has the potential mitigate this by leveraging intrinsic physical noise.
Finally, we investigate the role of the projection scheme by considering three strategies, Graded (gradual reduction, as used in the main experiments), Fixed (set to the number of classes), and Random (arbitrary dimensionality no less than the number of classes), performance remains largely comparable. As shown Fig. 44. all three projection strategies yield similar performance. However, omitting projection entirely (None) leads to notable performance drop, indicating that projection is essential for task-related performance.
4.4 Higher compression ratio leads better performance
To better understand how the ED is related to task performance, we calculate the ED, and ED, of the outputs of each block after projecting it into the that each block has a distinct projection dimensionality indicated by the short horizontal dashed lines, observe that both ED2 and ED, decrease across the blocks. However, the reduction in ED, is more pronounced compared to EDa, resulting in an increase in the compression ratio (depicted by the black dashed line). From the perspective of such diwer-dimensional manifold, the input samples belonging to the same class can be gradually explained by fewer dimensions, while samples from different classes are more likely be explained by different dimensions.
subspace used in training (Fig. 5l). Ignoring
The compression ratio may serve as a valuable indicator of task-related performance efficiency. To demonstrate, train two linear classifiers on the outputs of the first two blocks respectively, each following the same protocol as that employed for the final block, to assess changes in linear separability across the blocks. As depicted in Fig. [b)), performance improves steadily with the addition of more stacked blocks, corresponding to the iccrease in compression ratio.
However, the gradually increased compression ratio in the projected space does not necessarily imply a similar trend in original space. Appendix A1.4 we empirically evaluated the activation sparsity in trained models using Hoyer's sparseness [35] and found that the sparsity may not increase monotonically with depth. In addition, dataset, projection strategy, and network depth can all affect the activation sparsity.

---

To better understand how information is represented across layers, we apply information breakdown analysis 36 to the output of each block after passes through the corresponding linear clas- sifier (see Appendix A1.5 for details). We model the classifier output as samples from a Gaussian mixture and estimate the mutual information between these outputs and the class labels. Although the Gaussian assumption may not fully capture the true distribution, it provides insights into the representational structure through moment matching. As shown in Fig. 5P), the mutual information 1ot is mainly contributed by the linearly decodable component Liin. with the second order interaction foor contributing modestly. This suggests that while most label-relevant information is accessible to the linear classifier, small portion remains embedded in neuron-tor to--uooon correlation and is not linearly separable. We perform a similar analysis on MNIST (Appendix Fig. [4].. Given the simplicity of the task, linear separability is nearly saturated across all layers, even as the final block exhibits a higher compression ratio than the previous one. Here, Toor is negligible compared to Tüin indicating that almost all the information relevant the task is captured by linear projections, consistent with the high classification accuracy observed.
5 Conclusion and discussion
We have shown that the proposed dimensionality compression loss enables effective unsupervised learning within the FF framework without requiring negative samples. By injecting noise to create stochastic variants of each input, the model is trained to minimize the effective dimensionality of responses within a class (EDz) while maximizing that across classes (EDa). Experiments on MNIST, CIFAR-10, 10, and CIFAR- 100 demonstrate competitive performance using a shallow three- layer CNN, and our ablation studies further indicate that noise benefits learning and that stronger dimensionality compression correlates with better classification accuracy. Despite promising results, our method does not yet achieve state-of-the-ard accuracy and has difficulty when scaling to large- scale datasets or deeper architectures. These remain important directions for future research along with the development of biologically plausible implementations on neuromorphic hardware.
The ED objective may also be applied end-to-end and conceptually relates to self-supervised learning methods such as Barlow Twins (39 and VICReg 40). However, unlike these approaches, which explicitly enforce feature decorrelation or variance covariance constraints, our method encourages noisy copies of the same sample to reside on a low-dimensional manifold while ensuring distinct representational directions across different inputs.
The high-level principle of our method also relates to predictive coding 41 which posits that the brain continuously generates predictions about the incoming sensory input and updates its internal model based on prediction errors. In our framework, minimizing ED, encourages stable represen- tations that suppress noise-induced variability, whereas maximizing ED,, maintains discriminative information across stimuli. Such a dual objective balances accurate prediction with representational diversity, aligning with recent advances in predictive coding related learning algorithms 24
There are two main advantages of the proposed method that makes it biological plausible. Firstly, noise is ubiquitous in the brain 7 and can significantly affect the way neural system represents and processes information [46] 8 13). an essential characteristic that distinguishes it from deterministic digital computing. In such stochastic system, we have measure its outputs multiple times to obtain a reliable estimate. Intuitively, we would like to average the output, which implicitly assumes that variability is just noise that should be ignored. While such firing rate coding is widely adopted in neuroscience, accumulating evidence suggests that variability itself can also be information carriers [ 47 48). Our proposed method uses noise to generate multiple variants of each input sample, which offers an alternative perspective on how the brain may utilize noise and how neuromorphic computing can exploit the intrinsic noise of physical devices to facilitate learning
and computation [8 13 16 [8].
Secondly, optimizing ED is relatively easy to implement based on biologically feasible mechanisms. For instance, WTA competition is widely observed in various brain regions (49/511, which can be used to reduce FD, by encouraging sparse response. Maximizing FD2 is more subtle but still achievable in biologically plausible neural circuits. For example, Bergoin et al. (52) demonstrate that inhibitory neurons and their plasticity can consolidate and selectively separate learned assemblies
and limit memory capacity. Future work could explore the design of local learning rules that balance the power of WTA and inhibitory competition to adaptively meet the proposed objective.
10

---

Acknowledgments and Disclosure of Funding
Supported by the National Key R&D Program of China (2019YTA0709502); Supported by National Natural Science Foundation of China (No. 62306078) and the Science & Technology Commission of Shanghai Municipality (No. 25LN3300700);; Supported the Hua Laboratory S&T Program (No. X250881UG250 and the Lingang Laboratory (No. LGL-1987); Supported by ZJ Lab and Shanghai Center for Brain Science and Brain-Inspired Technology; Supported by 111 Project
(No. B 110015).
References
[1] Catherine D. Schuman, Shruti R. Kulkarni, Maryam Parsa, J. Parker Mitchell, Prasanna Date, and Bill Kay. Opportunities for neuromorphic computing algorithms and applications. Nature Computational Science, 2(1):10-19, January 2022. doi: 10. 10.1038/s43588-00 -00-8-0--2----1------
[2] Su-in Yi, Jack D. Kendall, R. Stanley Williams, and Suhas Kumar. Activity-difference training of deep neural networks using memristor crossbars. Nature Electronics, November 2022. doi:
10. 00558//41928-02--000
[3] Alexander G Ororbia. Brain-inspired machine intelligence: survey of neurobiologically- plausible credit assignment. arXiv preprint arXiv:23/2.09227, 2023.
[4] Menachem Stern and Arvind Murugan. Learning without neurons in physical systems. An- nual Review of Condensed Matter Physics, 14(1):4174441 March 2023. doi: 10.1146/
annurv-commaaphy--00000-------
[5] Wei Wang, Yang Li, and Ming Wang. Difficulties and approaches in enabling learning-ing memory using crossbar arrays of memristors. Neuromorphic Computing and Engineering, 4(3): 032002, August 2024. doi: 10.
[6] Geoffrey Hinton. The forward-forward algorithm: some preliminary investigations. arXiv preprint arXiv:2212. 13345, 2022.
[7] Aldo Faisal, Luc Selen, and Daniel M Wolpert. Noise in the nervous system. Nature Reviews Neuroscience, 94))292-333, 2008.
[8] Gustavo Deco, Edmund T. Rolls, and Ranulfo Romo. Stochastic dynamics principle of brain function. Progress in Neurobiology, 88(1):1-16, May 2009. doi: 10.1016/j pneurroio.....oi.. 2009.01.006.
[9] Herbert Jaeger, Beatriz Noheda, and Wilfred G. van der Wiel. Toward formal theory for computing machines made out of whatever physics offers. Nature Communications, 14(1), August 2023. doi: (0.10338/41146--23--0055
[10] Giacomo Indiveri. Neuromorphic is dead. Long live neuromorphic. $0896627325007081. October 2025. doi: n.1006i.nerron.2255.000000
Neuron, page
[11] Emilio Salinas and Terrence J. Sejnowski. Correlated neuronal activity and the flow of neural in- formation. Nature Reviews Neuroscience, 2(8):539 550, August 2001. doi: 10. 1038/3300061...
[12] Nikolaus Kriegeskorte and Xue-Xin Wei. Neural tuning and representational geometry. Nature Reviews Neuroscience, 22(1 22(11):703-715 September 2021. doi: 10.1038/441533-02--000555
[13] Stefano Panzeri, Monica Moroni, Houman Safaai, and Christopher D. Harvey. The structures and functions of correlations in neural population codes. Nature Reviews Neuroscience, 23(9): 551-567, June 2022. doi:
[14] Wolfgang Maass. Noise as resource for computation and learning in networks of spiking neurons. Proceedings of the IEEE, 102(5):860-88)) 880, 2014.
[15] Zhichao Zhu, Yang Qi, Wenlian Lu, and Jianfeng Feng. Learning integrate parts for whole through correlated neural variability. PLOO Computational Biology, 1012441 September 2024. doi: 10.137 Jcbi 1012401.

---

[16] Zhichao Zhu, Yang Qi, Wenlian Lu, Zhigang Wang, Lu Cao, and Jianfeng Feng. Toward free-response paradigm of decision making in spiking neural networks. Neural Computation. 37(3):481-5211 February 2025. doi: 10. 10.1162/neco_a 0_7333
[17] Yang Qi. Zhichao Zhu, Yiming Wei, Lu Cao, Zhigang Wang, Jie Zhang, Wenlian Lu, and Jianfeng Feng. Toward stochastic neural computing. arXiv preprint arXiv:2305. 13982, 2023.
[18] Yang Qi. Zhichao Zhu, Yiming Wei, Lu Cao, Zhigang Wang, Jie Zhang, Wenlian Lu, and Jianfeng Feng. Learning and inference with correlated neural variability. PNAS Nexus, 4(10); pgaf284, October 2025. doi: 10.109//pnsnexuu/pppa2....
[19] Marco Del Giudice. Effective Dimensionality: Tutorial. 56(3):527-542. doi: 10.1080/
00273171.202011433611.
[20] Benjamin Scellier and Yoshua Bengio. Equilibrium propagation:
bridging the gap between energy-based models and backpropagation. Frontiers in Computational Neuroscience, 11, May 2017. doi: .t.389/fncom.2217.00g
[21] Benjamin Scellier, Maxence Ernoult, Jack Kendall, and Suhas Kumar. Energy-based learning algorithms for analog computing: comparative study. Advances in Neural Information Processing Systems, 36:52705-52731, 2023.
[22] Yuhang Song, Beren Millidge, Tommaso Salvatori. Thomas Lukasiewicz, Zhenghua Xu, and Rafal Bogacz. Inferring neural activity before plasticity a foundation for learn- ing beyond backpropagation. Nature Neuroscience, 27(():348-33 -58,, January 2024. doi:
[23] Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. Advances In Neural Information Processing Systems, 29, 2016.
[24] Sindy Löwe, Peter O' Connor, and Bastiaan Veeling. Putting An End to End-to-End: Gradient- Isolated Learning of Representations. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[25] Laurenz Wiskott and Terrence J. Sejnowski. Slow Feature Analysis: Unsupervised Learn- ing of Invariances. Neural Computation, 14(4):715-770, April 2002. doi: 10.1162/
089976602317318999
[26] Gongpei Zhao, Tao Wang, Yi Jin, Congyan Lang, Yidong Li, and Haibin Ling. The cascaded forward algorithm for neural network training. Pattern Recognition, 161:111292, 2025.
[27] Andreas Papachristoddulou. Christos Kyrkou, Stelios Timotheou, and Theocharis Theocharides. Convolutional channel-wise competitive learning for the Borward-forail algorithm. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 14536-14544, 2024.
[28) Adrien Journé, Hector Garcia Rodriguez, Qinghai Guo, and Timoleon Moraitis. Hebbian deep learning without feedback. arXiv preprint arXiv:2209. 11883, 2022.
[29] Julian Jimenez Nimmo and Esther Mondragon. Advancing the biological plausibility and efficacy of hebbian convolutional neural networks. arXiv preprint arXiv:2501.17266, 2025.
[30] Stefano Recanatesi, Matthew Farrell, Madhu Advani, Timothy Moore, Guillaume Lajoie, and Eric Shea-Brown Dimensionality compression and expansion in deep neural networks. arXiv preprint arXiv: 1906.00409 2019.
[31] Matthew Farrell, Stefano Recanatesi, Timothy Moore, Guillaume Lajoic, and Eric Shea-Brown. Brown. Gradient-based learning drives robust representations in recurrent neural networks by balancing compression and expansion. Nature Machine Intelligence, 4(6):564-573, June 2022. doi: 10.1038/<42256-22 022-00498-00 00498--.
[32] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86( (6(11))2278-2324, 1998.
12

---

[33] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[34] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9111), 2008.
[35] Patrik O. Hoyer. Non-negative matrix factorization with sparseness constraints. Journal of Machine Learning Research, 5(Nov):1457-146) 2004.
[36] G Pola, A Thiele, K-P Hoffmann, and S Panzeri. An exact method to quantify the information transmitted by different mechanisms of correlational coding. Network: Computation in Neural Systems, 14(1):35-60, January 2003. doi: (0.1888/0954-88884441111
[37] Aaron J Gutknecht, Michael Wibral, and Abdullah Makkeh. Bits and pieces: understanding information decomposition from part-whole relationships and formal logic. Proceedings of the Royal Society A, 477(2251):202100 10, 2021.
[38] Andrea I. Luppi, Fernando E. Rosas, Pedro A.M. Mediano, David K. Menon, and Emmanuel A. Stamatakis. Information decomposition and the informational architecture of the brain. Trends in Cognitive Sciences, 28 28(4):352-366 April 2024. doi: 10.1016/j.tic wic.:iic.s..........
[39] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny. Barlow Twins: Self- Supervised Learning via Redundancy Reduction. In Proceedings of the 38th International Conference on Machine Learning, pages 12-220. PMLR, July 2021.
[40] Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Aariance-Invariane Invariane--Cvvvriancc Regularization for Self-Supervised Learning. arXiv Preprint arXiv: 2105.04906 January 2022.
[41] Rajesh P. N. Rao and Dana H. Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 2(1):79-87, January 1999. doi: 1038/48/4
2(1):)99
[42] Karl Friston. The free-energy principle: a unified brain theory? Nature Reviews Neuroscience, 1222):127-138 February 2010. doi: 10. 1038//mm2787
[43] Georg B. Keller and Thomas D. Mrsic-Flogel. Predictive Processing: Canonical Cortical Computation. Neuron, 100(2):424445) October 2018. doi: 10.1016/j.ccuron2011 201.curroo..1..0.000.....
[44] Bernd Illing, Jean Ventura, Guillaume Bellec, and Wulfram Gerstner. Local plasticity rules can learn deep representations using self-supervised contrastive predictions. arXiv preprint arXiv:2010.08262. October 2021.
[45] Manu Srinath Halvagal and Friedemann Zenke. The combination of Hebbian and predictive plasticity learns invariant object representations in deep sensory networks. Nature Neuroscience, 26(11): (6(11):1906-1 1901-19965.1..1 November 2023. doi:
[46] Jaime De La Rocha, Brent Doiron, Eric Shea-Brown. Krešimir Josié, and Alex Reyes. Correla- tion between neural spike trains increases with firing rate. Nature, 448(7155):802-8806 806, August 2007. doi: 10. 00.1038/aatuee00028.
[47] Jonathan W. Pillow, Jonathon Shlens, Liam Paninski, Alexander Sher, Alan M. Litke, E. J. Chichilnisky, and Eero P. Simoncelli. Spatio-temporal correlations and visual signalling in a complete neuronal population. Nature, 454(7207):995-999, August 2008 doi: 10. nature07140.
[48] Mohamady El-Gaby, Hayley M. Reeve, Vitor Lopess-os-SSntos, Natalia Campo-Urriza, Pavel V. Perestenko, Alexander Morley, Lauren A. M. Strickland, István P. Lukáes, Ole Paulsen, and David Dupret. An emergent neural coactivity code for dynamic memory. Nature Neuroscience, 24(5):694-704, May 2021. doi: 10. 10000//441593300---082-----
[49] Wolfgang Maass. On the Computational Power of Tineer-Takee-Il, Neural Computation, 12 (11):2519 25325, November 2000. doi: 10. 10.11//0099977000000444277,,
[50] Rodney J. Douglas and Kevan A.C. Martin. Neuronal circuits of the neocortex. Annual Review of Neuroscience, 27(1):419445) 451, July 2004. doi: 10. 2166:mmurv.neumm2277770003..33.0
13

---

[51] T. Binzegger, R. J. Douglas, and K. A. C. Martin. Topology and dynamics of the canonical eircuit of cat VI. Neural Networks, 220)):0071-1078 October 2009, doi: 10.1016/j.neune) 2009.07.011.
[52] Raphaël Bergoin, Alessandro Torcini, Gustavo Deco, Mathias Quoy, and Gorka Zamora-López. Inhibitory neurons control the consolidation of neural assemblies via adaptation to selective stimuli. Scientific Reports, 13(1):6949 April 2023. doi: (0038//41598-022-34416
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main contributions and scope are included in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the Conclusion and discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have fully disclose all the information needed to reproduce the main exper- imental results of the paper. The code used in this paper is included in the Supplementary
Material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide scripts to reproduce all experimental results. See codes in the Supplementary Material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httpppscccppullccguuudddeo Coaee//dddeSebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details included in Section A1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
The results are accompanied by I-sigma error bars estimated 5 independent runs.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments utilize an NVIDIA RTX 3090 GPU and an Intel Xeon(R) Gold 6226R CPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We think our works will attract the interest of researchers who concerns biological plausible learning rules and prain-inspired computing. We disccused the potental applicaltion in the introduction and conclusion sections. This research has no immediate negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cited the original paper that produced the code package and the dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, datasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code is packed in anonymized zip file. See the supplementary material for details.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The LLM is used only for writing, editing, and formatting purposes.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neurre /0tp:///eccccooeeeeee////2////// for what should or should be described.
21

---

Appendix
A1
Details of the methods
A1.1 Network architecture
The network architecture used in this work is summarized in Table A1 For all experiments, the setting of the convolutional blocks same, and the only difference is the input dimension fully connected layer.
Table Al: The details of the network architecture.
Block index
Components
BatchNorm (No affine)
Dropout, p = 0.2, creating N = 20 noisy copies
5 standard conv, 96 channels, stride padding
ReLU
x MaxPooling, stride 2, padding
BatchNorm (No affine)
3 x depthwise conv, 384 channels, stride I, padding ReLU
4 x MaxPooling, stride 2, padding BatchNorm (No affine)
3 x 3 depthwise conv, 1536 channels, stride 1, padding
3
ReLU
2 x 2 AvgPooling, stride 2. padding 0
Flatten
4
Dropout, p 0.5, inplace, for Regularization
Linear, 13824 input dims for MNIST, 24576 for CIFARIO and CIFARI00
A1.2 Details of the experimental design
All models were trained using the AdamW optimizer with a learning rate of 0.001 and weight decay of 0.01. A cosine annealing learning rate schedule was applied, with maximum of 3 and 60 iterations for phase I and phase training, respectively. The batch size was fixed at 128 across all experiments.
During phase 1 training, given an input batch X' E BXCXHX after passing through the batch normalization layer of the first convolutional block, we applied dropout with a probability of 0.2 to generate N = 20 noisy variants per sample, resulting in (X E R |BBxXCCXHXX This dropout was only used in the first block and remained active during inference.
The output of each block was projected onto a predefined pover-dimensional space using randomly generated orthogonal basis vectors sampled from the Haar distribution (via SciPy). Under the default Graded setting, the projection dimensions were 30-20-10 for MNIST and CIFAR-10 and 90-150-100 for CIFAR- 100. The block then optimized using the proposed objective computed projected output. By default, a trade-off factor a = 0.5 was used. To ensure purely local optimization, we detached the output tensors from the computation graph before passing them the next block. preventing gradient flow across layers.
the
During the second phase of training, trained a linear classifier using cross-entropy loss. Due to the sampling in the first block, the classifier output Y had the shape of B x 20 x 10 for MNIST and CIFAR-10, and b 20 100 for CIFAR-100. The uncentered second moment across the sampling dimension was used as the prediction score, and the cross-entropy loss was computed against the ground truth labels.
To assess the role a (Fig a-b and Fig. A2 we varied a from 0 to increments of 0. when training the first block in MNIST and CIFAR 0 to investigate changes in channel weights. We also
22

---

trained models five times with = 1.0 and compared their task performance with the default setting (Fig. c)..
To study how inference strategies influence performance (Fig
4 a), we reused the trained convolutional blocks and retrained the classifier using two alternative strategies. The first used the mean of the output as the prediction score, denoted by [E[{] while the second used regular feedforward without the sampling operation.
We also evaluated other projection approaches. the Fixed setting, the projection dimension for each block was matched to the class count. The blocks used dimension of 10 for MNIST and CIFAR- 10. For CIFAR- 100, due to channel constraints, the dimensions were 90 for the first block and 100 for the second and third. In the Random setup, the projection dimensionality for each block was chosen randomly. For MNIST and CIFAR-10, 10, it varied between 10 and 60; for CIFAR-100, spanned 100 to 300 (except for the first block, set at 90). In the None configuration, the projection operation was omitted.
To minimize the effort of reimplementing Greedy InfoMax (GIM) 24 for comparison, we resued the scripts provided by the authors and modified the network architecture to align with ours, leaving the hyperparameters used for this method unchanged. The training procedure for phase I and phase remained the same as in the main experiments except for two differences. Firstly, for the MNIST dataset, to ensure that there were enough patches for prediction, while keeping other settings unchanged. Secondly, instead of layer-wise wise training, the scripts provided by the authors trained all blocks simultaneously in each iteration. For each dataset, trained models five times with GIM and reported the mean and standard deviation of the test accuracy.
scaled the imput images to 32 32
A1.3 The effect of noise strength and sampling size
To better understand the effect of noise, we conducted experiments by varying the probability of dropout p from 0.1 to 0.5 and the sampling size from 4 to 20 to empirically investigate how the noise level affects performance. Other training settings remained unchanged as in the main experiments. The results are summarized below.
Table A2: The effect of noise strength and sampling size when training on MNIST.
sample size
0.1
0.2
0.3
0.4
0.5
99.4
99.37
99.23
98.61
97.95
8
99.35
99.41
99.38
99.29
98.29
99.36
99.32
99.42
98.86
98.51
16
99.35
99.36
99.41
98.75
98.54
20
99.44
99.41
99.34
98.65
98.56
Table A3: The effect of noise strength and sampling size when training on CIFAR1O.
sample size p
0.1
0.2
0.3
0.4
0.5
74.87
74.64
75.6
74.68
74.54
76.57
77.02
75.8
77.1
75.73
12
76.8
77.32
77.24
76.33
76.2
16
76.6
76.87
77.62
77.66
76.7
20
77.19
76.97
76.75
77.12
77.12
Table A4: The effect of noise strength and sampling size when training on CIFARION
sample size
0.1
0.2
0.3
0.4
0.5
52.08
51.99
51.19
51.19
48.59
50.84
52.79
52.51
52.29
50.63
12
51.48
51.51
51.88
50.95
33.54
16
53.21
52.97
52.45
51.52
49.44
20
52.6
52.66
53.79
49.95
25.21
23

---

A1,4
The sparseness of neural activation
We empirically evaluated the activation sparsity in trained models with Hoyer's sparseness measure:
vd
1c 2
(xx)
vd
where r is the activation vector of a layer with d neurons. Here, the vector x is the flattened feature map of convolutional layer from N noisy variants of input sample and i the length of z. We then
calculated the mean and standard deviation of Hoyer's 's sparseness across five independently trained models for each dataset, with 100 samples per class. The results are summarized in Table A3 Here,
Table A5: Sparseness of neural activation across different layers.
Layer index
MNIST (FOP/NP)
(F
CIFARI00 (FOP/NP)
Layer
0.52 0.05 /0.29 0.06
0.34
0.04
0.55 1 0.08 / /0.49 E 0.05
Layer 2
0.24 = 0.02 / 0.45 + 0.07
0.29 0.05 + 0.10
0.42 + 0.06 / 0.38 + 0.09
Layer 3
0.46 0.03 / 0.87 0.04
0.43 1433:000 0.04 0.06
0.39 + 0.07 0.54 0.06
FOP means that the model was trained by first projecting the layer's output to a fixed dimensionality using the Graded projecting scheme, and NP means directly optimizing the layer's 's parameters without the projection operation. These results showed that the sparsity varied with datasets, projection strategy, and network depth. It could range from highly sparse (0.87) quite dense (0.24).
A1.5 Information breakdown analysis
The mutual information between the target t and the readout Y defined in terms of the difference between the readout entropy on all stimuli and the conditional entropy for given class of stimuli as follows.
heo (Y; () I(Y) h(Y(4). The entropy h(Y) and the conditional entropy (t( are given by
M(Y) - =- - p(Y) (o))oop((Y)l
(8)
MYY)) - Σx()) p((()) (ogP(()),
(9)
where the readout distribution p(Y(t) is modeled as Gaussian distribution with mean and covariance E conditioned on the stimulus class t. The readout distribution over all stimuli is calculated as
p(Y) Σr(Dp((1))
(10)
We used the information breakdown analysis 36 which further dissects mutual information To into three components, allowing us to assess the amount of contributions of individual readout components Iin signal similarity among readout components Isigsim and noise correlation in readouts Toor- The quantity Jim measures the total amount of information that would be transmitted if all readout components were independent, which is given by
Iim ΣINO)) - ([(X)))A(Y((0)))
(11)
where Yj is the j-th component of the readout. The quantity Isigim measures the information loss arising from the redundancy due to overlaps between the tuning curves of each readout component, which is given by
ssgsim = (((ind)) -- ((()).
(12)
where the independent population response Yind is defined by the distribution
p(Yinalt)
II pYY)t)).
(13)
24

---

Figure Al: Example of noisy image from the CIFAR-10 dataset. Each column represents five example images corrupted by randomly dropping its pixels with a probability increment of 0.05.
b
Mean sld of kemels
0.3
0.2
0.3
0.1
0.2
0.2
0.4
0.8
Figure A2: Extension of Fig analysss to MNIST dataset.
The last component Loor accounts for the rest part of Lott that is, the total amount of information due to noise correlations in the readout.
Loor Ltot
Ann
sigsim.
(14)
For simplicity, we absorb Isigsim in Iim and use Iin to represent the information that can be obtained using linear methods.
We initially used the trained linear classifier shown in Fig. and Fig A4 b derive the prediction scores [[[22] for each block. Subsequently, these scores were aggregated according to the labels of the inputs, allowing the calculation of their mean and covariance. Using Gaussian mixture model with 10 components, each sharing the derived mean and covariance, we generated 100,000 samples from each Gaussian component conduct the information breakdown analysis.
A2 Visualization and additional analysis
25

---

b
Dimaniionllly Compression
Dinenskoally Compression
BP
MNIST
GIAAR10
Figure A3: Comparison of outputs of models trained by the proposed method and backpropagation using T-SNE visualization. a and are results on MNIST and CIFAR- -1 datasets respectively.
30
0.592
Informaton (nar))
20
2.0
Assunicy
1.5
oor
Conpreession
0.900
1.0
10
1.5
0:988
00961
50n42
cOnKO
conv1
can2
00m33
Figure A4: Extension of Fig 5 Jnalysis to MNIST dataset.
26


Reviews:
[{"review_id": "pe5PQST6xb", "reviewer": "Reviewer_RdwQ", "summary": "The authors introduce, in the paper \"Stochastic Forward-Forward Learning through Representational Dimensionality Compression\", a novel variation of the forward-forward learning algorithm that expands upon the formulation of the layer-wise goodness. The new goodness function is proposed as a measurement of effective dimensionality which goes beyond the mere magnitude goodness that is used typically. This choice is well motivated and allows learning without the need for negative samples and instead cleverly apply noise. Their derivations are supported by experimental evidence on MNIST, CIFAR-10, and CIFAR-100.", "questions": "* Can the approach remain effective and competitive for deeper architectures? I.e. also in comparison to the networks used in equilibrium propagation?\n* Why are the authors not comparing to layer-wise contrastive methods such as successors of [1]?\n* There is no indication about the number of orthonormal basis vectors employed for the pre dimensionality reduction. This is a bit misleading and never is mentioned to sufficient capacity, which could be misleading to readers.\n* Can the method be applied to biologically more realistic models both on the neuron and architecture level (recurrent)? \n* What would be the effect of changes in the employed noise distribution introduced by dropout?\n\n[1] Löwe, S., O'Connor, P., & Veeling, B. (2019). Putting an end to end-to-end: Gradient-isolated learning of representations. Advances in neural information processing systems, 32.", "limitations": "* Primarlily employing shallow architectures: The authors are encouraged to investigate deeper architectures as this is necessary to properly present the method and strengthen the message\n* Although the use of noise is a positive point, it becomes at the same time a question whether the method can work in lower noise regimes too.\n\nThe authors are also encouraged to automatically check for typos.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n* As far as I know the literature, this dimensionality compression goodness function is novel and original in this setting. Hence, an insightful extension that leverages neuron interactions.\n* Clever introduction of noise to enable learning and circumvent the need for negative samples.\n* The authors provide sufficient insight in the form of analysis and ablations.\n\nWeaknesses:\n\n* The method effectiveness is demonstrated only on shallow architectures, especially considering that the non-BP methods from the literature consider much deeper networks.\n* Equilibrium Propagation: The authors note their performance is behind to that of equilibrium propagation, commenting that it might be a result of deeper networks and increased computation. Based on this the comparison to equilibrium propagation is quite indefinitive and should ideally be tested in a control to compare on equal grounds.\n* The dependency on noise could also be harmful. Indeed, what if the neuromorphic device that would be considered offers a lower level of noise than what is needed?", "was_revised": true, "final_justification": "I believe the authors present a good case for a follow up to the FF algorithm with a more powerful goodness function. They provide, after the rebuttal, a more complete analysis and comparisons. My conclusion is that the paper will benefit the community overall.", "rebuttal": "Thank you for your valuable commets. A point-by-point response to the weaknesses and questions are as follows:\n\n> **Weaknesss 1**: - The method effectiveness is demonstrated only on shallow architectures, especially considering that the non-BP methods from the literature consider much deeper networks.  \n**Weakness 2**: Equilibrium Propagation: The authors note their performance is behind to that of equilibrium propagation, commenting that it might be a result of deeper networks and increased computation. Based on this the comparison to equilibrium propagation is quite indefinitive and should ideally be tested in a control to compare on equal grounds  \n**Question 1**: _Can the approach remain effective and competitive for deeper architectures?_\n>\n\n**Response**: Weaknesses 1 and 2, along with Question 1, primarily concern the scalability of our method, so we address them together. While we agree that scalability to arbitrary architectures is important, our current experiments show that simply stacking more blocks does not improve performance. In fact, using deeper networks or the same architecture as in EBL leads to worse performance with our ED-based layerwise training compared to shallower models.\n\nThis performance drop is likely not due to depth alone. We suspect that this is due to that samples from the same class may not consistently activate the same subset of neurons. As network depth increases, this issue may be amplified, distorting representations and deviating from task-relevant goals. In contrast, EBL benefits from directly clamping outputs to desired labels, which enforces task alignment. We will revise the manuscript to clarify this performance gap compared to EBL.\n\nNevertheless, we believe our method can be scaled for deeper networks with enhancements like residual connections.\n\n---\n\n> Question 2: - **Why are the authors not comparing to layer-wise contrastive methods such as successors of [1]?**\n>\n\n**Response**: Our initial focus was on forward-forward learning, so we primarily reviewed works published after 2022, when the original FF paper [a] was introduced. As a result, we unfortunately overlooked the follow-up work [1].\n\nTo address this, we conducted the same experiments using the proposed Greedy InfoMax method, following its architecture and training protocol. The only minor change was resizing input images to 64×64, allowing us to use the authors' hyperparameters without further tuning.\n\nThe results are shown below:\n\n|  | Greedy InfoMax | Ours |\n| --- | --- | --- |\n| MNIST | 99.31 | 99.3 $ \\pm $ 0.07 |\n| CIFAR10 | 77.94 | 76.96 $ \\pm $ 0.73 |\n| CIFAR100 | 50.37 | 53.29 $ \\pm $ 1.02 |\n\n\nDue to time and computational constraints, we only ran each experiment once. Nevertheless, the results suggest that our method has the potential to achieve performance comparable to Greedy InfoMax.\n\n---\n\n> Q3: - **There is no indication about the number of orthonormal basis vectors employed for the pre dimensionality reduction. This is a bit misleading and never is mentioned to sufficient capacity, which could be misleading to readers**.\n>\n\nThank you for pointing this out. The number of orthonormal basis vectors used for projection was 30-20-10 for MNIST and CIFAR-10 and 90-150-100 for CIFAR-100, as mentioned in the Supplementary Material S1.2. \n\nWe will explicitly state the number of orthonormal basis vectors used in the experiments in Section 4 of the main text to avoid any confusion. In Fig. 2, we will add a dimensionality reduction module to clarify the preprocessing step before computing the ED loss. \n\n---\n\n> Q4: - **Can the method be applied to biologically more realistic models both on the neuron and architecture level (recurrent)?**\n>\n\nYes. To explore the applicability of our method to more biologically realistic models, we conducted experiments using the Moment Neural Network (MNN) [b], which captures the dynamics of firing statistics of spiking neural network (SNN) up to second order moments. Given noisy inputs with first and second moments $ (\\mu,\\Sigma) $, the MNN produces corresponding spike count moments $ (\\hat\\mu,\\hat C)=\\phi(\\mu,C $) through a nonlinear mapping.\n\nIn this framwork, $ ED_c $ can be computed directly.  For $ ED_d $ , we define the non-centered total covariance over a batch as:  \n$ \\mathbb E[XX^T] = \\frac{1}{B}\\sum_{i=1}^B (C_i + \\mu_i\\mu_i^T) $  \nWe tested this setup on MNIST using a one-hidden-layer MNN and achieved 92% accuracy. Although this is below state-of-the-art, it demonstrates that a biologically realistic model (SNNs) can potentially be optimized in a similar fashion due to the close correspondence between MNN and SNN.\n\nRegarding recurrent networks, we were unable to conduct experiments due to time constraints. However, prior empirical findings by Farrell et al. [c] show that ED in RNNs increases and then decreases during training with backpropagation. This suggests that optimizing ED in a principled way could yield competitive results in RNNs as well.\n\n> **Weakness 3** ：- The dependency on noise could also be harmful. Indeed, what if the neuromorphic device that would be considered offers a lower level of noise than what is needed? - What would be the effect of changes in the employed noise distribution introduced by dropout?  \n**Question 4**: - What would be the effect of changes in the employed noise distribution introduced by dropout?\n>\n\nTo better understand the effect of noise, we conducted experiments by varying the dropout probability (from 0.1 to 0.5) and the sampling size (from 4 to 20) to empirically investigate how the noise level affects performance. The results are summarized below:\n\n**MNIST**\n\n| sample size / P | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 |\n| --- | --- | --- | --- | --- | --- |\n| 4 | 99.4 | 99.37 | 99.23 | 98.61 | 97.95 |\n| 8 | 99.35 | 99.41 | 99.38 | 99.29 | 98.29 |\n| 12 | 99.36 | 99.32 | 99.42 | 98.86 | 98.51 |\n| 16 | 99.35 | 99.36 | 99.41 | 98.75 | 98.54 |\n| 20 | 99.44 | 99.41 | 99.34 | 98.65 | 98.56 |\n\n\n**CIFAR10:**\n\n| **sample size / P** | **0.1** | **0.2** | **0.3** | **0.4** | **0.5** |\n| --- | --- | --- | --- | --- | --- |\n| **4** | 74.87 | 74.64 | 75.6 | 74.68 | 74.54 |\n| **8** | 76.57 | 77.02 | 75.8 | 77.1 | 75.73 |\n| **12** | 76.8 | 77.32 | 77.24 | 76.33 | 76.2 |\n| **16** | 76.6 | 76.87 | 77.62 | 77.66 | 76.7 |\n| **20** | 77.19 | 76.97 | 76.75 | 77.12 | 77.12 |\n\n\n**CIFAR100**\n\n| **sample size / P** | **0.1** | **0.2** | **0.3** | **0.4** | **0.5** |\n| --- | --- | --- | --- | --- | --- |\n| **4** | 52.08 | 51.99 | 51.19 | 51.19 | 48.59 |\n| **8** | 50.84 | 52.79 | 52.51 | 52.29 | 50.63 |\n| **12** | 51.48 | 51.51 | 51.88 | 50.95 | 33.54 |\n| **16** | 53.21 | 52.97 | 52.45 | 51.52 | 49.44 |\n| **20** | 52.6 | 52.66 | 53.79 | 49.95 | 25.21 |\n\n\nThese results demonstrate that both noise strength (dropout probability) and sampling size significantly influence model performance. In general, moderate noise levels and appropriate sampling sizes yield optimal results.\n\nBeyond the numerical findings, we offer further commentary on this issue. Currently, digital computing hardwares often aim at eliminating noise. However, neuromorphic chips are designed to closely mimic the computational mechanisms of the brain, one of which is the highly irregular and stochastic nature of spiking activity. This inherent variability—often regarded as noise—has been shown to play a functional role in biological computation. Consequently, many studies explore how the brain harnesses such stochasticity to perform useful work. In the long term, we anticipate that neuromorphic hardware will likely evolve toward analog computation, embracing rather than suppressing intrinsic noise. In this context, the noise-dependent optimization paradigm may not be a limitation, but a reflection of biological realism and a design opportunity.\n\n---\n\n**Reference**  \n[a] Hinton, G. The Forward-Forward Algorithm: Some Preliminary Investigations. ArXiv (2022).  \n[b] Qi, Y. _et al._ Toward stochastic neural computing. ArXiv (2024).  \n[c] Farrell, M., Recanatesi, S., Moore, T., Lajoie, G. & Shea-Brown, E. Gradient-based learning drives robust representations in recurrent neural networks by balancing compression and expansion. _Nat Mach Intell_ **4**, 564–573 (2022)."}, {"review_id": "JjzKPZZ9dE", "reviewer": "Reviewer_5k7D", "summary": "The article proposes a variation of the forward-forward approach. In the original method, the variance of layer activations is used as a goodness-of-fit measure, with unbounded growth controlled by subtracting the objective evaluated on negative samples. To address the challenge of generating such negative samples, the current article introduces an alternative goodness-of-fit function based on effective dimensionality, defined as the ratio of the trace to the Frobenius norm of class-conditioned correlation matrices of hidden layer activations. The article reports empirical results on MNIST, CIFAR-10, and CIFAR-100 datasets.", "questions": "* Section 3, Line 99: \" a population of neurons where each\n100 neuron is selectively responsive to a particular type of stimuli\" is this linear separability condition envisioned for neurrons at all layers, or they are mostly for the final layer neurons?\n\n* Figure 1 (c): Is ED in these figures for the unconditional distributions, or class condition distributions? why should ED of class conditioned distributions depend on the center locations of class clusters?\n\n* What does ED_c in (2) measeure and what does ED_d in (3) measure? why should they be a goodness of fit functions? how do they guarantee that the class conditioned  distribution domains have low effective dimension?\n\n* Does the proposed loss lead to local learning rule? if not what is its advantage over the backpropagation algorithm?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n- Addressing the negative sample issue of the original forward-forward algorithm is valuable.\n- The performance on the benchmark datasets provide improvement over the original forward-forward algorithm.\n\n## Weaknesses\n\n- The article lacks clarity in the description, explanation, and motivation of the proposed approach. The mathematical notation—particularly concerning class conditioning—is imprecise and difficult to follow, which obscures the formulation of the proposed objective. The authors should first clearly define the class-conditioned objective and then demonstrate how it is transformed into its final unsupervised form, explicitly presenting the full loss function across all layers. Additionally, the article should provide a clear rationale for why this objective is expected to be effective.\n\n\n- The reason why forward-forward algorithm introduced is to provide a biologically plausible explanation for biological networks. It is not clear whether the proposed effective dimensionality loss leads to a biologically plausible structure and local learning rule.\n\n- Although the performance is better than the original forward-forward algorithm it is not better than other biologically plausible algorithms.\n\nThe article's presentation requires major improvement. Some examples are:\n\n* Section 2: Background work on forward-forward based and other backprop alternatives are not properly explained.\n\n* Page 2, Line 78-79: DFA stands for  Direct Feedback Alignment. Besides the statement \" ...  each block to align its output directly with the final target\" for DFA is not very accurate.\n\n* Page 3, Figure 1 caption (b): Incorrect sentence \"Within-class responses that colored blue and orange resepectively are tightly clustered, leads to a low ED\"\n\n* Page 3, Figure 1 caption (b): There is no gray ellipse visible? is it the  dashed circle?\n\n* Page 3, Figure 1 caption (d): Incorrect sentence \"The statistic factors that affecting ED\"\n\n\n* Page 3, Section 3:  The tuning curve has not been properly defined before its use. The authors could use conditional distribution based mathematical notation here.\n\n* Page 3: The authors could provide more motivation about the loss in (1). They can describe why it favors lower dimensions through the norm inequality on the eigenvalues.\n\n* Page 3, Section 3: Equation (1) should be first defined based on label's as it measures the effective dimensionality of the based on the conditional (conditioned on the labels) distributions. This needs to be mathematically precise, so that unsupervised version later defined could be better explained in reference to this definition.\n\n* Page 4: the loss functions in (2)-(4) are not clearly described, and why they should work is not clearly explained.\n\n* Page 14, Figure S1 caption: incorrect sentence \" randomly drop its pixes\".", "was_revised": true, "final_justification": "I believe the effective dimensionality–based objective proposed in this article is worthy of attention. My only remaining concern, however, relates to the biological network implementation of this objective. I wanted to clarify this point during the discussion—particularly regarding the WTA-based sparse implementation proposed by the authors in their response. The outcome of that discussion suggests that the biological network architecture and learning dynamics based on the ED objective remain an open question for future research.", "rebuttal": "Thank you for your valuable commets. Due to space limitations, we have focused our response on addressing Weaknesses 1-3, select aspects of Weakness 4, and Questions 1-4 in this rebuttal (while retaining reference numbering). We have also corrected all typographical and grammatical errors mentioned in Weakness 4.\n\n> **Weakness 1**.\n\n**Response**: We understand the mathematical notation in the original manuscript was a bit messy and may be ambiguous or inconsistent at times. Below is the definition of the learning objective with cleaner mathematical notations and a theoretical explanation of why it should work. \n\nDenote $X^{(l)}\\in \\mathbb{R}^{B\\times F}$ the neural response in block $l$ to a batch of noisy input samples (B is the batch size and F is the number of features or neurons). The covariance of the neural responses across random realizations of different inputs is $\\Sigma^{(l)}=\\mathbb{E}[X^{(l)T}X^{(l)}]$ where $\\mathbb{E}[\\cdot]$ denotes expectation over random realizations of the inputs. Note that in the noise-free setting the covariance is simply  $\\Sigma^{(l)}=X^{(l)T}X^{(l)}$ which captures the data variability. The ED is then defined as   \n$\\text{ED}(X^{(l)}) = \\frac{\\text{tr}(\\mathbb E[X^{(l)T}X^{(l)}])^2}{\\lVert\\mathbb E[X^{(l)T}X^{(l)}]\\rVert_F^2} = \\frac{(\\sum_{i=1}^{d} \\lambda_i)^2}{\\sum_{i=1}^{d} \\lambda_i^2},$  \nwhere $\\lambda_i$ are the eigenvalues of the uncentered covariance matrix $\\mathbb E[X^{(l)T}X^{(l)}]$.  \n\nLet $x\\in\\mathbb{R}^{B\\times F}$ be a batch of data samples and use dropout to create its noisy copies, $X$, by randomly setting the elements in x to zero with probability p.  \nFor the $l$-th block, we denote its output (i.e., the corresponding neural responses) as $X^{(l)}$.  The input block is therefore $X^{(0)}=x$. Further denote the neural responses to a particular input sample in the batch as $X_{i}^{(l)}$, that is, the i-th row of  $X^{(l)}$ .  \nThe goodness function for each block is then partioned into two terms, a consistency term and a diversity term, defined as:  \n\n$\\text{ED}_{c}=\\frac{1}{B}\\sum_i^B\\text{ED}\\left(X^{(l)}_i\\right)$, \n\nand   \n\n$\\text{ED}_{d}=\\text{ED}\\left(\\mathbb{E}[X^{(l)}]\\right)$  \n\nwhere $B$ is the batch size, $\\mathbb{E}[X^{(l)}]$ denotes averaging over noisy copies of the input batch. The consistency term describes the average ED of neural responses to noisy realizations of individual input samples and captures the noise variability, whereas the diversity term describes the ED of response distribution across different input samples and captures the data variability.\n\nTo understand why our method works, we examine an ideal scenario modeled by a Gaussian mixture model (GMM) with two modes, as in Fig. 1b of the main text. The blue and orange dots represent noisy responses for inputs belonging to classes a and b, respectively. The covariances of class a and class b are visualized by the corresponding blue and orange ellipses, while the overall non-centered covariance of the GMM is depicted by the dashed gray ellipse. In this case, $\\mathrm{ED}_c \\approx 1$ is computed based on the non-centered covariance within each class, whereas $\\mathrm{ED}_d \\approx 2$ is computed from the total non-centered covariance across all data.\n\nFurthermore, minimizing $\\mathrm{ED}_c$ promotes anisotropy in the statistical moments—i.e., it increases the difference in variance across different dimensions (as shown in Fig.1c). For example, given an input from class $a$, the corresponding response becomes more concentrated along one dimension and more dispersed along another. This asymmetry facilitates linear separability between the two classes.\n\n> **Weakness 2:** \n\n**Response**: Due word limitation, please see the response to the Question mentioned by Reviewer ck6c, where we explain the plausibility of optimizing ED within local circuits.\n\n> **Weakness 3**\n\n**Response:**  We appreciate the reviewer's concern regarding the performance gap between our method and other biologically plausible (non-BP) algorithms. While our method improves upon the original Forward-Forward (FF) algorithm, we acknowledge that it does not yet outperform the best existing non-BP methods (such as EBL).\n\nHowever, we would like to offer two clarifications:\n\n1. **Performance among non-BP methods remains comparable**: If we compare only among existing biologically plausible algorithms (excluding our method), their performance generally falls within a similar range. This suggests a broader challenge: non-BP methods as a class still struggle to match the performance of standard BP, likely due to the constraints imposed by biological plausibility (e.g., locality, lack of weight transport).\n2. Finding **what works** is currently more important than finding **what works best**: At this stage, the priority is to identify learning principles that allow biologically plausible models to function reliably across a variety of tasks. Our contribution lies in demonstrating that optimizing effective dimensionality offers a promising direction that enhances performance within the FF framework while remaining compatible with biological constraints. We believe this lays the groundwork for future improvements in efficiency.\n\nWe will revise the Discussion to more explicitly acknowledge this limitation and clarify our intended contribution.\n\n> Weakness 4.4\n\n**Response**: Correct. It refers to the dashed circle, though in general it is an ellipse - depending on the components of the gaussian mixture. Its center represents the mean of the gaussian mixture and the ellipse itself represents the **uncentered covariance** of the gaussian mixture. Note that this does not imply that the gaussian mixture itself has a unimodal shape.\n\n> Weakness 4.6\n\n**Response**: We understand the original manuscript lacks a precise definition of the tuning curve. In our case, if we let $X$ be the noisy neural response to an input sample $x$ with label $c$, then we can write the tuning curve (in mean firing rate) as $\\mu(c)=\\langle\\mathbb{E}[X]\\rangle_c$. Here, we use $\\mathbb{E}[\\cdot]$ to denote expectation over random realizations and $\\langle\\cdot\\rangle_c$ to denote averaging across all data samples with label $c$. One may also write out these expectation explicitly in terms of conditional distribution but we find that would be too cumbersome and it does not improve further the clarity.\n\n> Weakness 4.8\n\n**Response**: The ED in response to Weakness 1 is now more clearly defined. If we restrict the sample data to the same class, then the ${\\rm ED}$ formula in the revised notation would measure the effective dimensionality of that class.\n\n**Questions**:\n\n> **Question 1** Section 3, Line 99: \" a population of neurons where each 100 neuron is selectively responsive to a particular type of stimuli\" is this linear separability condition envisioned for neurrons at all layers, or they are mostly for the final layer neurons?\n>\n\n**Response**:  The statement refers to all layers, not just the final one. This passage provides high-level intuition for why it is possible to define a goodness function suitable for Forward-Forward (FF) learning.\n\nThe key idea is that since the final classification depends on the linear separability of the output representations, improving linear separability at each layer—through layerwise optimization—can cumulatively lead to good overall performance. While we do not enforce strict linear separability at every layer, encouraging this property throughout the network helps guide learning in the absence of backpropagation across layers.\n\nWe will clarify this point in the text to avoid confusion.\n\n> **Question 2**  Figure 1 (c): Is ED in these figures for the unconditional distributions, or class condition distributions? why should ED of class conditioned distributions depend on the center locations of class clusters?\n>\n\n**Response**:  In this panel, what variable the distribution is conditioned on is irrelevant. It is simply a graphical illustration of how ED would change based on parameters of a distribution - in this case the mean, variance, and correlation of a 2d gaussian.\n\n> Question 3: What does ED_c in (2) measeure and what does ED_d in (3) measure? why should they be a goodness of fit functions? how do they guarantee that the class conditioned distribution domains have low effective dimension?\n\n**Response**:  For the first two questions, please refer to our responses to Weakness 1 and Question 2. Regarding the third question, while we cannot guarantee that class-conditioned distributions will achieve low effective dimensionality (since optimization is unsupervised), we hypothesize that samples from the same class share similar structures, activating the same subset of neurons and thus reducing ED.\n\nOur experiments support this: (a) the method achieves reasonable classification performance, and (b) after training, the ratio $ \\text{ED}_d / \\langle \\text{ED}_c\\rangle $ increases, where $ \\langle \\text{ED}_c\\rangle $ is the average class-conditioned ED and $ \\text{ED}_d $ is the unconditional ED.\n\n> Question 4 Does the proposed loss lead to local learning rule? if not what is its advantage over the backpropagation algorithm?\n\n**Response**: No. The gradients are still calculated using BP, though there is no gradient flow across blocks. However, this approach could help guide biologically plausible learning rules in future work. For more details, please see response to the Question mentioned by Reviewer ck6c, where we explain the plausibility of optimizing ED within local circuits.\n\n**Reference**  \n[d] Kriegeskorte, N. & Wei, X.-X. Neural tuning and representational geometry. _Nat Rev Neurosci_ **22**, 703–718 (2021).  \n[e] Panzeri, S., Moroni, M., Safaai, H. & Harvey, C. D. The structures and functions of correlations in neural population codes. _Nat Rev Neurosci_ (2022)."}, {"review_id": "03SQofB67b", "reviewer": "Reviewer_ck6c", "summary": "The paper describes a local self-supervised learning method. The learning minimizes a loss applied layer-wise, and gradients are blocked across layers. The loss measures the dimensionality of the data using the effective dimension, which is the ratio between the square of the sum of eigenvalues of the covariance matrix, divided by the sum of the squared eigenvalues. The loss is the difference between two forms of ED, one applied to a noisy sample from the same input, and the second is applied across the batch dimension. The dimensionality is therefore maximal for different images in a batch, but minimal with a noisy version of the same image. The method is tested on image classification datasets (with a post-training linear classification accuracy). It performs relatively well in comparison with other layer-wise learning rules. It is reported that the kernel weights become orthogonal through the learning process", "questions": "I wonder what the specificity of the chosen loss function is in comparison with other learning algorithms. The algorithm is prominently compared with EBL, but EBL provides an alternative to gradient descent as a whole and is a supervised algorithm. What is the high-level intention of doing this comparison ? References like [1,2,3] are in the sense more similar to the proposed algorithm. If the key question is to provide a bio-plausible learning algorithm, how would he gradient descent step of the ED look in a simple case, would this indeed be apparent to a bio-plausible learning mechanism?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "While there have been many different approaches to layer-wise learning rules, I have seen ED being used to analyze data but not yet as a loss function. It is elegant and mathematically sound. \nThe learning algorithm is mathematically sound and well-explained, and it is great that it uses a simple loss function. The performance is tested on reasonable benchmarks, and the reported performance seems logical and reproducible. \n\nIn terms of weaknesses:\n\n- It is unfortunate that the loss function is not tested (with identical architecture and with similar analysis) against alternative loss functions. For instance, I wonder how the result compares to the same architecture but compared with the (more common) InfoNCE loss function. Methods like InfoNCE, which are usually applied end-to-end, can also be applied block-wise [1] and layer-wise [2,3], and vice versa, ED could be applied ETE. So it would be useful to know how ED compares to these methods.\n\n- One of the motivations is to make a \"non-contrastive\" algorithm. But I suspect that it is secretly a contrastive algorithm: the term ED_d applied on the batch dimension performs comparison on the batch dimension. In fact, it seems that the algorithm does not work for alpha=1, where this term is dropped. So I suspect that the contrastive component is needed. I do not think it is crucial for the message of the paper to claim that the method is non-contrastive, though.\n\n[1] https://arxiv.org/abs/1905.11786\n[2] https://arxiv.org/abs/2010.08262\n[3] https://www.nature.com/articles/s41593-023-01460-y", "was_revised": true, "final_justification": "The paper provides a useful analytical approach to layer-wise local learning. The additions promised by the author during the rebuttal will strengthen the paper.", "rebuttal": "Thank you for your valuable commets. A point-by-point response to the weaknesses and questions are as follows:\n\n> Weaknesses 1: - It is unfortunate that the loss function is not tested (with identical architecture and with similar analysis) against alternative loss functions. For instance, I wonder how the result compares to the same architecture but compared with the (more common) InfoNCE loss function. Methods like InfoNCE, which are usually applied end-to-end, can also be applied block-wise [1] and layer-wise [2,3], and vice versa, ED could be applied ETE. So it would be useful to know how ED compares to these methods.\n>\n\n**Response**:\n\nOur initial focus was on forward-forward learning, so we primarily reviewed works published after 2022, when the original FF paper [a] was introduced. As a result, we unfortunately overlooked the follow-up work [1].\n\nTo address this, we conducted the same experiments using the proposed Greedy InfoMax method, following its architecture and training protocol. The only minor change was resizing input images to 64×64, allowing us to use the authors' hyperparameters without further tuning.\n\nThe results are shown below:\n\n|  | Greedy InfoMax | Ours |\n| --- | --- | --- |\n| MNIST | 99.31 | 99.3 $ \\pm $ 0.07 |\n| CIFAR10 | 77.94 | 76.96 $ \\pm $ 0.73 |\n| CIFAR100 | 50.37 | 53.29 $ \\pm $ 1.02 |\n\n\nDue to time and computational constraints, we only ran each experiment once. Nevertheless, the results suggest that our method has the potential to achieve performance comparable to Greedy InfoMax.\n\nWhile we can also train a network end-to-end using ED, this field has been well studied that there are many successful solutions for unsupervised ETE learning. So we limit our discuss in non-BP methods.\n\n---\n\n> Weakness 2: - One of the motivations is to make a \"non-contrastive\" algorithm. But I suspect that it is secretly a contrastive algorithm: the term ED_d applied on the batch dimension performs comparison on the batch dimension. In fact, it seems that the algorithm does not work for alpha=1, where this term is dropped. So I suspect that the contrastive component is needed. I do not think it is crucial for the message of the paper to claim that the method is non-contrastive, though.\n>\n\n**Response**:  \nYou're correct that $ \\text{ED}_d $ implicitly encourages separation between samples, which could be viewed as a contrastive mechanism. However, our key contribution is not about being \"non-contrastive\" but rather proposing an alternative unsupervised feedforward learning approach that avoids explicit negative sample construction by leveraging noise. We will revise the introduction and Section 3 to clarify this and remove any misleading claims about non-contrastiveness.\n\n---\n\n> **Question**：I wonder what the specificity of the chosen loss function is in comparison with other learning algorithms. The algorithm is prominently compared with EBL, but EBL provides an alternative to gradient descent as a whole and is a supervised algorithm. What is the high-level intention of doing this comparison ? References like [1,2,3] are in the sense more similar to the proposed algorithm. If the key question is to provide a bio-plausible learning algorithm, how would he gradient descent step of the ED look in a simple case, would this indeed be apparent to a bio-plausible learning mechanism?\n>\n**Response**:\n**Why Compare with EBL?**  \nWe emphasize connections to Energy-Based Learning (EBL) for three reasons:\n\n1. Forward-Forward (FF) learning has roots in Boltzmann Machines and Hopfield Networks, making EBL a natural comparison.\n2.  EBL does not have to be supervised. It can optimize an energy function that similar to FF's layer-wise objective if we restrict the \"whole network\" as a single layer. Moreover, biological plausible rules such as Hebbian and anti-Hebbian learning, can also be framed as energy minimization by stabilizing correlated activity.\n3. There are many evidience suppprt that both the mean firing rate and the firing variability can affect neural coding and neural computation [b,f]. In our method, it is naturelly to suggest using $ E[X^2] $ as the metric for classication instead of $ E[X] $, which means we should jointly consider the mean and variance. Interestingly, this fomula also is regard as the energy term in EBL.\n\nAlthough an explicit weight update rule derived from ED would have non-local terms such as calculating the trace of covariance matrix, we would like to provide the following high-level rationale to support its biological plausibility:\n\n1. **Plausibility of optimizing ED within local circuits**:\n    - Minimizing $ \\text{ED}_c $ naturally leads to a sparse activation pattern, which can be implemented through a biologically plausible winner-takes-all (WTA) mechanism. In such a circuit, only a small subset of neurons respond strongly to a given input, resulting in a low effective dimensionality at the population level due to widespread silencing.\n    - Maximizing $ \\text{ED}_d $is more subtle but still achievable in plausible circuits. One scenario involves excitatory and inhibitory neurons operating on different timescales. When an input is clamped, the most active excitatory neurons drive activity in associated inhibitory neurons to maintain network balance. Upon switching from input A to B, the slower inhibitory dynamics can transiently suppress previously active excitatory neurons, allowing other neurons to become active via a renewed WTA competition. This mechanism effectively reshapes the population response to new inputs, thereby increasing $ \\text{ED}_d $.\n2. **Experimental relevance**: The effect of tuning curves [d] and higher-order statistics [e] on neural coding and representational capacity has been extensively studied in neuroscience. The optimization of effective dimensionality offers a unifying theoretical principle that could be tested against neural data in future works.\n\nWe will incorporate these points into Section 3 and the Discussion to better explain our motivation and why the proposed objective is compatible with biologically plausible learning mechanisms.\n\n**Reference**  \n[a] Hinton, G. The Forward-Forward Algorithm: Some Preliminary Investigations. ArXiv (2022).  \n[b] Qi, Y. _et al._ Toward stochastic neural computing. ArXiv (2024)  \n[d] Kriegeskorte, N. & Wei, X.-X. Neural tuning and representational geometry. _Nat Rev Neurosci_ **22**, 703–718 (2021).  \n[e] Panzeri, S., Moroni, M., Safaai, H. & Harvey, C. D. The structures and functions of correlations in neural population codes. _Nat Rev Neurosci_ (2022)"}, {"review_id": "srsuOogev9", "reviewer": "Reviewer_t7od", "summary": "The authors develop a technique to enhance non backpropagation based learning, through the use of contrastive samples that are generated without supervision. Essentially through noise, regularization losses and dimensionality reduction.", "questions": "n/a", "limitations": "Discussed in the conclusions.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\nThe paper presents a well-motivated and biologically plausible alternative to backpropagation by introducing a novel goodness function based on effective dimensionality (ED). This function captures second-order statistics of neural responses, promoting compact, class-consistent representations and diverse, globally discriminative features. They achieve this without requiring labels or manually crafted negative samples. The use of noise as a constructive mechanism to generate contrastive samples aligns well with principles from neuroscience and neuromorphic computing, providing a practical and theoretically grounded approach for unsupervised learning. Empirical results across MNIST, CIFAR-10, and CIFAR-100 show that the method achieves competitive performance compared to other non-backpropagation algorithms. Furthermore, the inclusion of ablation studies and orthogonality analyses helps support the framework's core claim, that optimizing representational structure via ED leads to more robust and linearly separable features.\n\n**Weaknesses:**\nDespite its conceptual elegance, the paper's empirical validation remains relatively limited. The model underperforms compared to equilibrium-based learning and backpropagation, particularly on more complex datasets, and it is tested only on shallow architectures. The reliance on manually chosen hyperparameters (e.g., the projection dimensionality and trade-off weight alpha) and the lack of large-scale or real-world benchmarks weaken the claim of generality. Additionally, while the theoretical motivation is sound, the exposition of experimental results lacks a clear narrative, and some of the key concepts, like the compression ratio and its link to performance, are underexplored in the broader context of learning dynamics. A more direct comparison to EBL and a clearer explanation of what is gained relative to prior methods would further strengthen the contribution.", "was_revised": true, "final_justification": "I hereby confirm my previous review, in that I find it to be an interesting contribution to the field of non gradient learning and biologically plausible alternatives to BP. I hope the authors have the time to run experiments on more complex datasets for the final submission and try to justify with metrics of speed or memory improvements what we gain if we lose a bit of accuracy.", "rebuttal": "Thank you for your valuable commets. A point-by-point response to the weaknesses and questions are as follows:\n\n>**Weakness1**: Despite its conceptual elegance, the paper's empirical validation remains relatively limited. The model underperforms compared to equilibrium-based learning and backpropagation, particularly on more complex datasets, and it is tested only on shallow architectures.  \n>\n\n**Response**:  While we agree that scalability to arbitrary architectures is important, our current experiments show that simply stacking more blocks does not improve performance. In fact, using deeper networks or the same architecture as in EBL leads to worse performance with our ED-based layerwise training compared to shallower models.\n\nThis performance drop is likely not due to depth alone. We suspect that this is due to that samples from the same class may not consistently activate the same subset of neurons. As network depth increases, this issue may be amplified, distorting representations and deviating from task-relevant goals. In contrast, EBL benefits from directly clamping outputs to desired labels, which enforces task alignment. We will revise the manuscript to clarify this performance gap compared to EBL. We will revise the manuscript to clarify this performance gap compared to EBL.\n\nNevertheless, we believe our method can be scaled for deeper networks with enhancements like residual connections.\n\n---\n\n> **Weakness 2**: The reliance on manually chosen hyperparameters (e.g., the projection dimensionality and trade-off weight alpha) and the lack of large-scale or real-world benchmarks weaken the claim of generality.\n>\n\nWhile our experiments show the method works across various projection dimensionalities (Fig 4c), we currently use hand-tuned settings because Farrell [c,f] revealed intrinsic ED patterns in BP-trained networks. The optimal configuration for projection dimensionality and $ \\alpha $ remains an open question. We acknowledge the need for more rigorous benchmarks and will clarify the scope of our claims regarding generality.\n\n> **Weakness 3**: Additionally, while the theoretical motivation is sound, the exposition of experimental results lacks a clear narrative, and some of the key concepts, like the compression ratio and its link to performance, are underexplored in the broader context of learning dynamics.\n>\n\n**Response**: We acknowledge the lack of theoretical analysis in the original manuscript. Further investigation of the learning dynamics during the weight update process is needed.\n\n> **Weakness4**:  A more direct comparison to EBL and a clearer explanation of what is gained relative to prior methods would further strengthen the contribution.\n>\n\nWe will include a more thorough literature review to cover the necessary backgrounds in the revision. \n\n**Reference**  \n[c] Farrell, M., Recanatesi, S., Moore, T., Lajoie, G. & Shea-Brown, E. Gradient-based learning drives robust representations in recurrent neural networks by balancing compression and expansion. _Nat Mach Intell_ **4**, 564–573 (2022).  \n[f] Farrell, M. Revealing Structure in Trained Neural Networks Through Dimensionality-Based Methods. PhD thesis, Univ. Washington (2020)."}]

Author rebuttal:

Meta-review: The paper describes a local self-supervised learning method based on a variation of the forward-forward approach. 
The loss used is based on the dimensionality of data, using the effective dimension, which is the ratio between the square of the sum of eigenvalues of the covariance matrix, divided by the sum of the squared eigenvalues.
The dimensionality is maximal for different images in a batch, but minimal with a noisy version of the same image.


Strengths identified by the reviewers include:

1. The dimensionality compression goodness function is novel and original in this setting.

2. Performance on the benchmark datasets provide improvement over the original forward-forward algorithm.

3. The introduction of noise to enable learning and circumvent the need for negative samples is of interest.


Weaknesses identified, which the authors should try to address, include:

1. Algorithm demonstrated only on shallow architectures.

2. The mathematical notation is imprecise.

3. Although the performance is better than the original forward-forward algorithm it is not better than other biologically plausible algorithms.

The authors in their rebuttal did a good job explaining some misunderstandings regarding biological plausibility and the effectiveness of the method with deeper architectures. This convinced some of the reviewers to upgrade their score.

After rebuttal, the paper received 1 Accept, and 3 Borderline accepts. As such I am inclined to recommend acceptance of the paper. The authors should make a reasonable effort to address the concerns of the reviewers in the final paper version.
Decision: Accept (poster)