DeeperForward: Enhanced Forward-Forward Training for Deeper and Better Performance
Authors: Liang Sun
Yang Zhang
Weizhao He
Jiajun Wen
Linlin Shen
Weicheng Xie
Venue: ICLR 2025 Poster (2025)
Keywords: backpropagation-free
forward-forward
forward learning
brain-inspired computing
Primary area: optimization
TL;DR: We propose DeeperForward which extends FF to CNNs up to 17 layers using a new design of goodness, improving performance with layer depth and achieving the best performance among recent FF methods.

Abstract: While backpropagation effectively trains models, it presents challenges related to bio-plausibility, resulting in high memory demands and limited parallelism. Recently, Hinton (2022) proposed the Forward-Forward (FF) algorithm for high-parallel local updates. FF leverages squared sums as the local update target, termed goodness, and decouples goodness by normalizing the vector length to extract new features. However, this design encounters issues with feature scaling and deactivated neurons, limiting its application mainly to shallow networks. This paper proposes a novel goodness design utilizing **layer normalization** and **mean goodness** to overcome these challenges, demonstrating performance improvements even in 17-layer CNNs. Experiments on CIFAR-10, MNIST, and Fashion-MNIST show significant advantages over existing FF-based algorithms, highlighting the potential of FF in deep models. Furthermore, the model parallel strategy is proposed to achieve highly efficient training based on the property of local updates.

Published as a conference paper at ICLR 2025
DEEPERFORWARD: ENHANCED FORWARD-FORWARD TRAINING FOR DEEPER AND BETTER PERFORMANCE
Liang Sun Yang Zhang Weizhao Jiajun Wen', Linlin Sh.n Weicheng Xie CComputer Vision Institute, School of Computer Science & Software Engineering, Shenzhen University National Engineering Laboratory for Big Data System Computing Technology. Shenzhen University "Guangdong Provincial Key Laboratory of Intelligent Information Processing Launliange heweizhao 220eeei2eaa22222maaaaaal 2022@ema11 ssu. edu. cn,
fyangzhang, wenjiajun, 11shen, e edu. en
Code: http::/ghthobccooooooyssssuueeet
ABSTRACT
While backpropagation effectively trains models, it presents challenges related to bio bio-plausibility. resulting in high memory demands and limited parallelism. Recently, Hinton (2022) proposed the Forward-Forward (FF) algorithm for high- parallel local updates. FF leverages squared sums the local update target, termed goodness, and decouples goodness by normalizing the vector length to extract new features. However, this design encounters issues with feature scaling and deactivated neurons, limiting its application mainly to shallow networks. This paper proposes a novel goodness design utilizing layer normalization and mean goodness to overcome these challenges, demonstrating performance improvements even in 17-layer CNNs. Experiments on CIFAR-IO, MNIST, and MNNIST show significant advantages over existing FF-based algorithms, highlighting the potential of FF in deep models. Furthermore, the model parallel strategy proposed to achieve highly efficient training based on the property of local updates.
INTRODUCTION
Backpropagation (BP) (Rumelhart al., 1986) has achieved significant success, serving as the prevailing paradigm for training complex structures like ResNet (He et al. 2016) and Transformers (Vaswani et 2017). However, no compelling evidence supports such mechanism existing in the brain, challenging the biological plausibility of BP. Critical challenges within BP consist of weight transport (Grossberg (987). nocal WWittiintton Bogacz 2019), freezing activity, and update locking problems (Jaderberg et al. |017] Czarnecki et al. 2017). The weight transport problem arises from reusing the same path in forward and backward passes. The non-local problem arises from global objective loss, while the brain relies on local signals for updates. The freezing activity problem and update locking problem contradict the real-time property in neural systems. Freezing activity involves maintaining intermediate states, leading to increased memory demands. The update locking problem prevents any update until all layers are activated, reducing parallelism in practice.
To tackle these challenges, various brain- inspired training methods (Ororbia 2023) have been developed to formulate a comprehensive theory of inference and learning in a biologically plausible
manner (Lillicrap al. 2016 Nøkland 2016 Dellaferrera & Kreiman, 2022 Ororbia al. 2023 Hinton 2022). Several of these methods are depicted in Figure | A recent breakthrough is the (FF) algorithm (Hinton 2022). as depicted in Figure [d). FF employs the squared sum of outputs, termed goodness, and fixes the output vector length via dividing by its vector length, thereby decoupling goodness within the output features and compelling subsequent layers to learn new features. Mathematically, the output's 's length and direction correspond to goodness and features. However, this design has limitations that confine current layer-wise FF studies to shallow models. The primary reasons why FF fails to achieve performance improvements in deeper networks are as follows:
Equua Contribution: Liang Sun and Yang Zhang. Comesponding author: Yang Zhang.

---

Published as a conference paper at ICLR 2025
c
PEPTAA
Ours
target
tanget
Yyux lrge label
Foug -M image wrong
barget
p
T
1%
W9
82
W5
1
4
15
WS
1010
h1
W
aw% -lis
Oa- (ETT AMTE" 4m - W. AET 11 = W5- (x
DWN; -Yes (Srray froa))
6, Mis Moss
9. Ras Auu
Figure 1: Comparison of several training methods. (a) BP employs traditional forward and backward passes represented by blue and green arrows respectively. (b) Feedback alignment (FA) uses an alternative backward pass for error passes. (c) PEPITA uses two forward passes based on input perturbation. (d) FF is implemented with two forward passes on positive data and negative data, respectively. (e) Ours simplifies the learning process by using single forward pass.
Feature scaling Normalization by vector length is uncommon in image classification tasks it does not ensure that features exhibit similar characteristics, such as identical means and standard deviations. To address this, layer normalization (Ba al. (0166) can be applied the input vector, but this leads to redundant normalization and compromises the mathematical significance of output direction a feature. Consequently, CwComp (Papachristodoulou et al., 2024) employs batch normalization as a substitute. However, this approach fails to decouple goodness and leaks goodness the next layer, hindering deeper layers from learning new features and causing overfitting.
Deactivated neurons Square goodness is highly sensitive to outliers, which can dominate and deactivate most neurons, distorting feature representation. Moreover, deactivated neurons do not contribute to weight updates during gradient calculations, leading to features represented by a limited subset of neurons and causing feature loss in deeper layers.
In this paper, DeeperForward is proposed to address the mentioned issues by redesigning goodness and features to better suit deep networks. We also enhance the convolution structure proposed by CwComp (Papachristodoulou al. 2024). enabling effective training of FF in deeper CNNs. Our main contributions are as follows:
We adopt the more widely used layer normalization (Ba et al. 2016 to ensure fixed mean and standard deviation of the output, replacing normalization based on vector length and effectively addressing feature scaling and redundant normalization issues.
Exploiting the property of layer normalization that maintains mean of zero, we propose using mean goodness as an alternative to squared goodness, thereby facilitating the decou- pling of goodness for enhanced feature extraction. This approach also ensures that weight updates are not hindered by deactivated neurons, allowing for the learning of richer features.
Based on the characteristics of layer-wise local updates, we introduce model parallel strategy that significantly enhances training efficiency on multiple GPUs.
Our method enhances FF to achieve improved performance in deeper networks. Experi- mental results indicate that approach, utilizing 17-layer CNN, outperforms existing layer-wise FF-based methods on CIFAR- MNIST, and Fashion-MNIST, achieving sub- stantial performance gains, particularly 8.1 11 % improvement CIFAR-10
RELATED WORK
2.1
CONVENTIONAL BRAIN-INSPIRED LEARNING RULES
Hebbian learning Hebbb 2005
Gerstner et al. 2014) updates synaptic plasticity determined by pre-
and post-synaptic neuron states (Löwel & Singer 1992). Based on the Hebbian rule, neural coding
2

---

Published as a conference paper at ICLR 2025
framework was proposed for learning generative models using the predictive coding (Ororbia & Kifer 2022 Rao & Ballard 1999). SoftHebb (Jouréé et al. 2023) proposes an algorithm based on theory for Hebbian learning in soft winner-take-all (WTA) networks. Hebbian learning is considered basic bio-plausible method with no target.
In target-based methods, feedback alignment (Lillicrap et al. 2016) and direct feedback alignment (Nokland 2016) replace backpropagation weights with a fixed random matrix to establish alternative error feedback connections, as shown in [b). Weight mirror (Akrout al. 2019) adjusts the feedback connection matrix, equivalent to the transport weight matrix. However, these methods still rely on global error. Target propagation (TP) (Bengio 2014 Bartunov al. 2018 and difference target propagation (DTP) (Lee et al. 2015 Ernoult et al. (0022 set local targets as the goal for local updates. Local representation alignment (LRA) (Ororbia et al., 2023) addresses the asymmetry problem through top-down signal transmission with Hebbian-like rules, further solving the non-local problem. To update unlocking, decoupled greedy learning (Belilovsky et al. (2220) optimizes a joint training objective to decouple the layer training with auxiliary networks. Avoiding using backward passes, forward propagation training method through time is proposed for recurrent neural networks (Kag & Saligrama 2021). PEPITA (Dellaeereraa & Kreiman 2022) achieves local updates by perturbing inputs with the error and employs a Hebbian-like like rule based on two forward passes with a fixed feedback matrix, as shown in Figure Tc). Despite these advancements, they partially suffer from the update locking problem.
2.2 BACKGROUND OF FORWARD-FORWARD ALGORITHM
Inspired by Boltzmann machines (Hinton et al. 1986 and noise contrastive estimation (NCE) (Gutmann & Hyvärinen 2010), the Forward-Forwart algorithm (FF) (Hinton 2022) introduces a greedy learning scheme via two forward passes, as shown in Figure [d), tackling the mentioned bio-implausible problems. FF uses the length of the output vector as a measure of goodness, where
goodness represents the score of positive data. Decoupling goodness from the output features is
important to prevent subsequent layers from relying solely on previous goodness. Therefore, FF extracts features by normalizing the vector length, denoted as,
y -Wx)
(1)
g = Eni.
z
(2)
Nタ +
Oc
AW,, 2x,yi 0g
(3)
where x denotes the input, represents the output after ReLU GGloro et al. 2011 with N elements, W is the weight matrix, y denotes the element of the vector y of hidden layer, and g denotes goodness. The features is the unit vector of y. is small constant. AW,, denotes the weight update term and L the loss function. The image with a real label regarded as positive data for optimizing to reach a high goodness value in each layer, and vice versa. During inference, image entails computing the goodness of each label and selecting the highest one through several iterations. The preliminary study of FF only works on small networks without weight-sharing structures.
Recently, several works have proposed some advanced FF-related algorithms. Symmetric backpropagation-fir free contrastive learning with FF (SymBa) (Lee & Song 2023) enhances per- formance through a gradient-symmetee
contrastive loss and a novel label embedding scheme. The predictive Forward-Forward algorithm (PFF) (Ororbia & Mali 2023) integrates FF with predictive coding presenting a promising brain-inspired algorithm for cl classifying, reconstructing, and synthe- sizing data patterns. However, these approaches are still limited models without weight-sharinn structures. The cascaded forward (CaFo) algorithm (Zhao et al. 2023) utilizes a series of random fixed convolutional kernels as the backbone and cascades a fully connected classifier for each ker- nel. However, it merely updates the classifiers, leaving the kernels unchanged. Forward-Forwart contrastive learning (FFCL) (Ahamed et al. 2023) introduces contrastive learning for convolutional models based on FF. However, this approach still prefers extra training by global errors. Recently, convolutional channel-win -wise competitive learning (CwComp) (Papachristodoulouu et al., 2023 2024) successfully extends FF into CNNs by grouping the features by channels for each class, and using a loss function inducing competitive learning between slass-specific features. Despite the advance- ments, these methods focus shallow networks within 4 layers. Currently, Trifecta (Dooms et al.)
3

---

Published as a conference paper at ICLR 2025
Backbone
Inpot image
Training Proces
Block 1
Block2
Block
Block
Block
ANss pooling
CW Com
OW
CWOOm
CW O.e
CW-Omm
Avenyy wanngyppding
CW
CW Ckns
C
Aveniy
(3)
"cat"
g13)
CW-Com
5:20
540
501
9:9
9:00
9
y0tn gil
1
9014
Signal Integrating & Pruning (SIP)
CW-
500
60
Channel Convolution (CW-Coomv)
Teatures
G classes
Testures
Group
goodzess
Figure Overview of DeeperForward. (a) Example network architecture for DeeperForward, including backbone and Signal Integrating and Pruming module. A VGG-like architecture displayed as an instance. (b) The training scheme of DeeperForward. (c) Modified channel-wise convolution (CW-Conv) from CwComp based mean goodness.
2023)
employs a two-layer block-wise backpropagation approach to replace single-layer updates in 12-layer CNN, using batch normalization. However, this integration with backpropagation still presents bio-plausibility issues, diminishing parallelism and contradicting the motivations behind the FF algorithm. Both Trifecta and CwComp facilitate easier training by leaking goodness, which can result in potential overfitting in deeper networks.
3
METHODOLOGY
This paper introduces DeeperForward, which extends the FF algorithm to 17-layer CNNs through a novel goodness design. This approach addresses the bio-plausibility issues of backpropagation and overcomes the limitations of FF concerning model size. FigureZiilustrates illlustrates the overall framework of method, including the architecture and training approach. The details of the new goodness design
are presented in Section 3.1 The network architecture is discussed in Section 3.2 while the training
process and advanced strategies for DeeperForward are outlined in Section 3.3
3.1 MEAN GOODNESS
FF uses squared goodness and normalization of the length, as described in Eq. 2 This method suffers from issues related to feature scaling, deactivated neurons, and redundant normalization. resulting in suboptimal performance in deep networks. Although CwComp (Papachristodoulou et al. 2024) improves performance using squared goodness and batch normalization, leaks goodness information, leading to overfitting in deeper networks. Considering these factors, adopt widely used layer normalization for better feature scaling with identical mean and standard deviations. To decouple goodness through normalization, of layer normalization that produces an output with mean of zero. Furthermore, mean goodness ensures that deactivated neurons do not hinder updates. The specific formula is as follows:
utilize the mean as goodness, leveraging the property
y ReLU(Wx), --2 g 9= E,Vi, I = Va² +
(5)
dL AW, Cr. 0g"
4

---

Published as a conference paper at ICLR 2025
where denotes the input, y represents the output after ReLU (Glorot et al. 2011). W is the weight matrix, 9 indicates goodness, a is the standard deviation, refers to the output features, AW; the weight update term, L is the local loss function, and C' constant.
From Eq. [] it evident that the output distribution z maintains a mean of zero, effectively eliminating goodness. This also ensures that the features share similar distribution, addressing the feature scaling issue. During weight updates, mean goodness (Eq. ( allows for updates even when the output neuron yj is zero, unlike squared goodness (Eq. 3). thereby solving the deactivated neurons problem.
3.2
ARCHITECTURE FOR DEEPERFORWALL
The architecture for DeeperForward, illustrated in Figure 2 a). incorporates modified classical CNN backbone, exemplified by the VGG-like model (Simonyan & Zisserman (014). It incorporates convolutional structure that combines channel-wise convolution (CW-Conv) with mean goodness, along with a Signal Inlegrating and Pruming (SIP) module to obtain the final results.
Channel- Wise Convolution with Mean Goodness To incorporate mean goodness into CNNs, combining convolution with mean goodness involves simply obtaining the output mean as goodness, followed by layer normalization to facilitate feature extraction. Formally, the goodness 9 and representation Z are defined as:
Σ h, Z= =ayyyerNorm (H), hCN
i = HWC
where H E HX denotes the hidden states after the convolution with ReLU (Glorot et al. 2011 E I. indicates the goodness, that is, the mean of H. The representation output Z is H going through layer normalization.
In multi-clas clatsiclass tasks, optimize the channel-wise convolution (CW-Conv) structure from CwComp (Papachristodoulou et al. 2023 2024). combining with our mean goodness to obtain goodness scores for all classes through single inference. The outputs are evenly grouped by channel, with each group representing a class. Goodness calculated for each group, followed by individual layer normalization to extract features, effectively implementing group normalization (Wu & He] (0188) the entire output, as illustrated in Figure (c). Formally, the channel-wise convolution with mean goodness for G classes can be described as:
G HWC
01
Σn, h. i= 1,1,....,
helt,
ý [11 $2..... ..., UG).
(8)
Z GroupNorm (H:)),
(9)
where H; E x and ÿJ denotes the hidden states and goodness for the i-th class, and ý stands for classification scores. GroupNorml (H; G) represents the group normalization of hidden states H by G groups. Z stores the representation feature maps.
Compared our method, CwComp performs classification training directly on the outputs after batch normalization without decoupling goodness, resulting in goodness leakage to the next layer and leading to overfitting in deeper layers.
Backhone The backbone is derived from the classical CNNs, leveraging their well-established structural advantages. We substitute the general convolutional kernels with CW-Conv modules to generate classification results locally. The representation from each CW-Conv module serves as the input for the next layer, denoted as (1) where signifies the layer number. The local classification result at the /-th layer represented as g()) In particular, the channel size kemnels must be a multiple of the class count. Furthermore, to maintain approximate zero mean of the representation, we adopt average pooling for downsampling, instead max pooling, as the latter tends to increase the mean value. In this architecture, each layer produces a classification score using CW-Cony. Moreover, the experiments in Appendix reveal that the CW-Con WW-Conv outperforms the fully connected (FC) layer in terms of performance. Consequently, the final FC layer is needless.
5

---

Published as a conference paper at ICLR 2025
Targets 10cooness
Target: chuhemesss
10c
CW-Coo: CD
10c
10c
10c
Auy Pooi
c
20cc
(c: dlcumds))
b
aakiittoon
concaleration
Figure 3: Residual structures: (a) Addition type for the shorteut channels match the target channels. (b) Concatenation type for shortcut channels differing from the target channels.
Residual Structure Residual structures are traditionally employed to facilitate error backpropaga- tion by providing shortcuts for easier learning. FF, it enables the integration of features different levels, enriching the representational capacity of deep networks. To adapt to the FF, we implement two parameter-free residual structures, the addition and concatenation types, as alternatives to the original parameterized versions, illustrated in Figure 3 To match the spatial dimensions, employ average pooling to the shortcut, as shown below, for downsampling.
(AvgPoolXX.), Z. X
(H,, W,) + (H, W), (H,, W.) (),
(10)
where Z, E R "xWW denotes the shortcut feature map after spatial dimension adjustment, and AvgPool(.) the average pooling operation to adjust X, from (H,, W,) to (H, W).
Two types of residual structures are adapted in different scenarios. If the shortcut matches the channel of target feature maps, the addition type employed. Otherwise, the concatenation type is used. As shown in Figure 3 two residual structures can be summarized as:
F(X:C) C) Z,, Concat CConcat(F(X;CC) Cc), C+ Cr.
C=
(11)
where F(;; C)) stands for the CW-Conv with C' channels output from input X, and Z, is the feature maps from shortcut. Concat(..-) denotes the concatenation operation on channel dimension. represents the final representation output. Particularly, the number of convolution channels is reduced to C' C. in concatenation type to ensure the channel of final output satisfies the target.
Signal Integrating and Pruning Module Inspired by synaptic pruning (Chechik et al. 1998 Neniskyte & Gross] 2017), where the brain forms excess synapses and then eliminates redundancies, we propose the Signal Infegrating and Pruning (SIP) module. The FF accumulates local goodness to obtain the final result, with experiments showing that the last three layers perform best on the test set. Similarly, separate subset of data from the training set, leave untrained, and evaluate accuracy on this subset to select the best layer combination, avoiding direct testing on the test set. However, for deep model with L layers, there are 2 combinations. To reduce complexity, we simplify the rule to accumulating layers between a chosen start layer and an end layer, reducing the combinations LL 1).2. The SIP module with layers can be described as:
j
O<SSESL,
(12)
where (( denotes classification scores from the i-th layer, and ÿ is the final result. S, E Z are integers and range from to L, representing the start and end layers to accumulated. After selection, layers beyond the end layer are no longer used and can be pruned.
3.3 FORPARDORWARD TRAINING SCHEME
Training Scheme We present DecperForward. a training strategy that optimizes the classification result at each layer through single forward pass, relying solely the local input-output states.

---

Published as a conference paper at ICLR 2025
2igure2[b) depicts the training procedure of DeeperForward. Local optimization leverages the classification results from CW-Conv as the local target. It utilizes a local cross-entropy loss for each layer to generate the update signal, preventing enror transportation across layers. The local optimization can be formulated as:
C (1 - y (og(sso (1)
V Vo Tow-CEg 6") yy)
(13)
where y:0) and denote the local classification result and the real label with G classes.
LeR denotes
(()
the cross-entropy loss, while vO(!) is the update of weights at the (-th layer, and V V oo C (f()) .y) is
the gradient of - with respect the kernel weights (()) The local optimization process solely relies the input and classification result of the individual CW-Conv. DeeperForward is compatible with general gradient- -based optimizers, such as Adam (Kingma Ba 2014). Owing to the local learning process, there is need store intermediate states, eliminating the freezing activity problem and the update locking problem. Additionally, the non-local problem and weight transport problem are addressed by the local loss optimization and the forward training scheme.
Model Parallel Strategy Our method enables model parallel strategy based on the parallelism of FF mentioned in (Aktemur et al., 2024), illustrated in Figure 4 Our strategy treats each convolutional layer as an independent component in the pipeline. Once convolutional layer processes a batch data, passes the results to next group, allowing the next batch to be processed without waiting for the entire network to complete. As shown in Figure 4 (a), this approach enables simultancous processing of multiple batches across different layers, achieving high parallelism.
Figure 4 b) illustrates an implementation example using multithreading techniques in multi-GPP setup. Our strategy assigns an independent thread for each convolutional layer to update, utilizing firs-in-first-out(FFfts
queues for data transfer. Different threads can be allocated to various GPUs, enabling model parallelism. Compared to the commonly used distributed data-parallel (DDP) (Li et al. (020)) technique in backpropagation, our approach offers several advantages for improved efficiency: (i) Each GPU does not need to store the entire network, and (ii) Data transfer between GPUs occurs only between layers on different GPUs, rather than across the entire network. Details of implementation are Appendix G
Liger'
Layar
F.
11
F,
U,
F,
U2
Laper 1
0.
Un
Inforece ni ninilatc
U, Uplata of the nirilaads
Fix
(GPU Threall
(GPU Thrsell
Figure 4: Model parallel strategy. (a) A pipeline program (the same color indicates operations on the same minibatch of data). (b) An implementation example based on the multi selti-ttreading technique.
Memory-saving Strategy Due to update locking problems, BP requires large amount of memory to store intermediate states throughout the process. Without limitation of update locking, this strategy achieves memory savings by promptly releasing memory after each layer's computation. The memory saving strategy is layer-by-layer update strategy, which consists of following steps: (1) Perform
computation and update weights in current layer. (2) Pass the output to the next layer. (3) Release
all intermediate states from the memory used by the current layer. (4) Repeat steps (1)-(3) layer by layer. More details are shown in Appendix H This memory-saving strategy is particularly suitable for scenarios with constrained memory resources, such edge computing.
7

---

Published as a conference paper at ICLR 2025
Table 1: Classification on CIFARIO, MNIST, and F-MNIST, evaluating performance compared BP and FF-related algorithms, Measurements of mean and standard deviation are for five trial runs. #; Reproduced results. With data augmentation. I: With block-wise backprop.
Type
Method
Arch.
#Layer
CIFAR10
MNIST
F-MNIST
PEPITA
CNN
52.57 = 0.36
98.01 0.09
DTP
CNN
89.38 = 0.20
98.93 + 0.04
90.35 + 0.11 88.13
recLRA
CNN
18
93.58
98.18
non-FF
SoftHebb
SoftHebb
80.31 0.14
99.35 0.03
F
MLP
46.04 = 0.18
97.16 0.10
SP
CNN
8
92.4
HPFF
CNN
110
91.04
Block-wise BP
SEDONA
CNN
93.87
BWBPF
CNN
152
95.52
BP
ResNet18-BP*
CNN
18
94.03 = 0.11f
99.58 + 0.02
93.78 + 0.06
MLP
59.00
98.69
SymBa
MLP
59.09
98.58
FF
CaFo
CNN
3
67.43
98.80
CwComp
CNN
78.11
0.44
99.42
0.08
92.31 = 0.32
TinyCNN-ours
CNN
79.49
0.29
99.50 0.05
91.83 + 0.06
Trifecta:
CNN
83.51 = 0.78
99.58 0.06
91.44
CwComp"
CNN
14
75.28 = 0.54
99.27 + 0.09
91.79
FF
CNN-ours
CNN
14
81.76
0.30
99.65 + 0.02
92.44 0.08
ResNet-ours
CNN
86.22
0.17
99.63 0.04
93.13 0.13
Table 2: Classification on CIFARIO0.
Res
ResNet-ours
ResNet-CHa3-ours
Accuracy 58.01 + 0.48
53.09 + 0.79
60.28 + 1.02
4 EXPERIMENT
4.1 DATASETS AND EXPERIMENT SETTINGS
To fully validate the effectiveness of DeeperForward, we conduct experiments on 3 datasets: MNIST (LeCun al. 1998). Bashion-MNIST (F- (F-MIIST) (Xiao et al. 2017). and CIFAR1O (Krizhevsky data augmentation. Specifically, the traming sets of MNIST and F-MINST
et al.] (209)) without are separated into two groups, 50,000 and 10,000 samples. The former group used for training and the latter group is used for pruning by Signal Integrating and Pruning (SIP) module. Similarly, CIFARIO 10's training set is split into 45,000 and 5,000 samples. All the samples in the datasets are resized to 32 x 32 pixels. Hyperparameters setting is detailed in Appendix B Our experiments are executed on Nvidia GTX Titan GPUs (12GB).
Arcuna (%)
(1g OOeROO
(%g Katandy
(tei)
Trslo
Tasls durkie
Trii dostccl
Epoch
Elocc
Epoch
Figure 5: Performances CIFARIO. (a) Shallow networks, compared to BP and FF-based methods. (b) Deep networks, compared to BP without data augmentation. (c,d) Comparison experiments for residual structures 10 without dropout. (c) Model performance during training. (d) The accuracy of each layer after training for 150 epochs.
8

---

Published as a conference paper at ICLR 2025
4.2 COMPARISONS OF DIFFERENT METHODS
We employ three CNN models to evaluate our method: a 4-ylayer tiny CNN, a VGG-like 14-layer CNN, and 17-layer ResNet-like CNN (He et al. 2016), as detailed in the Appendix A Our comparisons encompass both non-FF and FF methods for a comprehensive analysis. Non-FF brain- -rain-inspired
methods include advanced BP-free approaches such as PEPITA (Dellaferrera & Kreiman 2022)
DTP Ernoult et al. 2022). rec-LRRA (Ororbia et al. 2023). SoftHebb (Journé et al. 2023), F (Fügell
ct al., 2023), and Signal Propagation (Kohan et al. (024). We also compare with block-wise BP
method: HPFF Su et al. 2024), SEDONA(Pyeon et al. 2021), and BWBPF(Cheng et al. (024)) In
the FF methods, we consider FF (Hinton 2022), SymBa Lee & Song 2023), Calo (Zhao et al. 2023).
Trifecta (Dooms et al. 2023), and CwComp (Papachristodoulou et al. 2024). Since layer- wise FF-
based methods operate on shallow networks, we reproduce and extend CwComp (Papachristodoulouu
al. 2024) into the same 14-layer CNNs for comparison. Results are summarized in Table [ Our method outperforms FF-based methods in CIFAR10 and MNIST with shallow networks. The training curves on CIFARIO with FF-based methods in shallow networks are shown in Figure [sa).
a). As we
extend to 14 and 17 layers, our performance improves, whereas CwComp (Papachristodoulou et al. 2024) exhibits overfitting, leading to performance decline. Therefore, our method extended the capability of FF to train in deeper models. The results indicate that our design of mean goodness enhances the performance of FF, making it more suitable for deep CNN models. However, FF-based methods train greedily through classification objectives at each layer, indirectly extracting features. This results in weaker feature learning capabilities compared to BP, which directly learns intermediate layer features. Although Figure 55 shows that method's 's performance is close to BP without data augmentation, the improvement is limited after data augmentation, as detailed in Appendix C leading to larger gap with BP. These points are also directions worth exploring further.
Furthermore, we conduct a more challenging experiment on CIFAR 100, as shown in Table 2 ResNet- CHx3 is variant of ResNet with triple the number of channels. Table ( highlights the disparity between FeeperForward and BP on ResNet. The significant improvement of ResNet-CHx3 indicates that inadequate allocation of neurons to each class results in a sharp decline in performance.
4.3 PERFORMANCE OF SIGNAL INTEGRATING AND PRUNING MODULE
Table 3: Performance on Signal Integrating and Pruning (SIP) using 17-layer ResNet. (Start, End) denotes the selected layers by SIP, where Start and End represent start layer and end layer.
CIFARIO
MNIST
F-MNIST
ALL LAYERS
86.45
99.68
93.08
AFTER SIP
86.51
99.67
93.23
Startt, End)
(2,17)
(2.11)
(3,16)
We validate our pruning strategy using the Signal Integrating and Pruning (SIP) module by comparing the performance with a similar strategy in FF (Hinton) 2022) that accumulates all the goodness as the final result. In Table 3. we select the best trial results using ResNet for the SIP experiment comparison, showing that SIP improve performance in most cases. Interestingly, experiments on simpler tasks such as MNIST tend to retain fewer layers compared to more challenging tasks like CIFAR10. This observation shows the ability to adapt its depth based on the complexity of the task.
4.4 ABLATION STUDY
Table 4: Ablation study on 100 showing the mean performance from five experimental trials.
MEAN
SIP
RESIDUAL
ACCURACY
79.38
81.02
81.16
/
86.08
86.22
This method introduces mean goodness, the Signal Integrating and Pruning (SIP) module, and non-learned residual structure to optimize performance. To evaluate the contributions of each component, we conducted ablation experiments using a 17-layer ResNet architecture on C0, averaging results from five trials, shown in Table When mean goodness is omitted, utilize

---

Published as a conference paper at ICLR 2025
squared goodness and normalization of the vector length. For comparison with SIP, directly sum all layers as the final output. The removal of the residual structure involves excluding the shortcut connections. To provide more comprehensive analysis, Appendix discusses the differences in deactivated neurons between mean and square goodness.
The experimental results show that mean goodness achieves a substantial performance increase of 6.84% compared to squared goodness within the same network. The SIP module provides a slight performance boost while allowing for optimization of network size. The residual structure significantly enhances performance by integrating features at various levels, resulting in a more comprehensive feature representation. Figures (c) and (d) analyze the training curves and local classification performance with and without the residual structure, demonstrating that the residual connections facilitate improved learning in deeper layers.
4.5 PARALLEL PERFORMANCE OF DEEPERFORWARD
We evaluate the performance of model parallel strategy through time-consumption training CIFARIO, using 1, 2, and 4 GPUs. In multi-Gre AuUti case, layers are evenly grouped and assigned to different GPUs As point of comparison,
use BP with the widely adopted distributed data parallel (DDP) (Li et al. 2020) as baseline. As shown in Table our method outperforms BP with DDP in terms of training time. Notably, our approach achieves higher speedup with 2 GPUs, as inter-GPU communication occurs only between layers on different devices, unlike DDP, where communication involves the entire network. However, with GPUs, the speedup is lower than DDP. We observed drop in GPU utilization, caused by imbalanced computation across layers, leading to pipeline program bottlenecks. Future work could explore advanced pipeline techniques for optimization. This study demonstrates the feasibility and potential of model parallelism in our method.
Table Training time per epoch on CIFARIO (Speedup rate relative to GPU in parentheses).
METHOD
I GPU
2 GPUS
GPUS
BP-DDP
51.98s (1.0x)
32.70s (1.59x)
19.92s (2.61x)
OURS
36.38s (1.0x)
20.77s (1.75x)
14.68s (2.48x)
Moreover, training on CIFARIO with a batch size of 128 using the memory-saving strategy consumes minimum of 618.64MB of memory in practice, while BP in ResNet18 requires 1314.49MB.
Additionally, we experiment with deeper ResNet models with 33 and 100 layers but do not observe significant performance improvements, as detailed in Appendix [ Moreover, Appendix provides comparison classification performance between different convolutional layers and fully connected layers. Appendix [F] presents t-SNE (Van der Maaten & Hinton 2008) visualizations of the results the MNIST dataset.
5 CONCLUSION
This paper presents the DeeperForward algorithm, extending the Forward-Forwant approach to deeper networks with significant performance enhancements. We introduce a novel goodness design, com- bining mean goodness and layer normalization, which addresses key issues in the effective training of deep networks: feature scaling, redundant normalization, and deactivated neurons. Additionally, we propose model parallel strategy significantly improve training efficiency and memory-saving strategy suitable for cewurree-consrainnee environments. Experimental results demonstrate that our method substantially enhances the depth and performance of FF-based algorithms, highlighting the potential of FF terms of performance and parallelism.
Linitations. FeeperForward, similar FF, relies solely classification information for learning, lacking direct representation learning capabilities. This results slower convergence and weaker generalization. Future research should focus on enhancing feature extraction capabilities address these limitations. Additionally, as the number of categories increases, the convolutional structure grows, making it challenging implement on extensive datasets. Future work should aim develop more general structures that avoid excessively large models and multiple forward passes in FF.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work is supported by the National Natural Science Foundation of China under Grant 62176163; the Shenzhen Higher Education Stable Support Program General Project under Grant 2023111201552115000 and the Science and Technology Foundation of Shenzhen under Grant JCY1220210024094602007
REFERENCES
Md Atik Ahamed, Jin Chen, and Abdullah Al Zubaer Imran. FFCL: Forward- contrastive learning for improved medical image classification. In Medical Imaging with Deep Learning, short paper track, 2023.
Mohamed Akrout, Collin Wilson, Peter Humphreys, Timothy Lillicrap, and Douglas B Tweed. Deep learning without weight transport. Advances in neural information processing systems, 32, 2019.
Ege Aktemur, Ege Zorlutuna, Kaan Bilgili, Tacettin Emre Bok, Berrin Yanikoglu, and Suha Orhun Mutluergil. Going forward-forward in distributed deep learning. arXiv preprint arXiv:2404.08573, 2024.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1071v:1607.064500 2016.
Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey Hinton, and Timothy Lilli- crap. Assessing the scalability of biologically-motivacee deep learning algorithms and architectures. Advances in neural information processing systems, 31, 2018.
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Decoupled greedy learning of CNNs. In International Conference on Machine Learning, pp. 736-745. PMLR, 2020.
Yoshua Bengio. How auto-encoders could provide credit assignment in deep networks via target propagation. arXiv preprint arXiv:1407.7906, 2014.
Gal Chechik, Isaac Meilijson, and Eytan Ruppin. Synaptic pruning in development: computational account. Neural computation, 1757):175--177,, 1998.
Anzhe Cheng, Heng Ping, Zhenkun Wang, Xiongye Xiao, Chenzhong Yin, Shahin Nazarian, Mingxi Cheng, and Paul Bogdan. Unlocking deep learning: bp-free approach for parallel block-wise training of neural networks. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4235 4239, 2024. 10447377.
doi: D0.109/CASSP488485..000
Wojciech Marian Czarnecki, Grzegorz Šwirszez, Max Jaderberg, Simon Osindero, Oriol Vinyals, and Koray Kavukcuoglu. Understanding synthetic gradients and decoupled neural interfaces. Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, pp. 904-912. PMLR, 2017.
Giorgia Dellaferrera and Gabriel Kreiman. Error-driven input modulation: Solving the credit assignment problem without backward pass. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, pp. 4937-4955. PMLR, 2022.
Thomas Dooms, Ing Jyh Tsang, and Jose Oramas. The trifecta: Three simple techniques for training deeper forward networks, 2023. BRLhttps /// arxiv. org/ http//b/////1111111
Maxence M Ernoult, Fabrice Normandin, Abhinav Moudgil, Sean Spinney, Eugene Belilovsky, Irina Rish, Blake Richards, and Yoshua Bengio. Towards scaling difference target propagation by learning backprop targets. In International Conference on Machine Learning, pp. 5968-5987. PMLR, 2022.
Katharina Flügel, Daniel Coquelin, Marie Weiel, Charlotte Debus, Achim Streit, and Markus Götz. Feed-forwand optimization with delayed feedback for neural networks, 2023. URL https 7/ rxivv, wwg/ /2304 13372
11

---

Published as a conference paper at ICLR 2025
Wulfram Gerstner, Wernr M Kistler, Richard Naud, and Liam Paninski. Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press, 2014.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 315-323, JMLR Workshop and Conference Proceedings, 2011.
Stephen Grossberg. Competitive learning: From interactive activation to adaptive resonance. Cogni- tive science, 11(1):23 63, 1987.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Donald Olding Hebb. The organization of behavior: neuropsychological theory. Psychology press, 2005.
Geoffrey Hinton. The horwardforward algorithm: Some preliminary investigations, 2022.
Geoffrey E Hinton, Terrence J Sejnowski, et al. Learning and relearning in boltzmann machines. Parallel distributed processing: Explorations in the microstructure of cognition, 1(282-317):2. 1986.
Max Jaderberg. Wojciech Marian Czamecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, pp. 1627-1635. PMLR, 2017.
Adrien Journé, Hector Garcia Rodriguez, Qinghai Guo, and Timoleon Moraitis. Hebbian deep learning without feedback. In The Eleventh International Conference on Learning Representations, 2023. /tttts:://pperrv..... .w..ee//eeeeeeeeeeeeeeeeeeee.eee 1oruummd=_8gdMM-M22
Anil Kag and Venkatesh Saligrama. Training recurrent neural networks via forward propagation through time. In International Conference on Machine Learning, pp. 5189-5200, PMLR, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1eXiv:1412.6966 6980, 2014.
Adam Kohan, Edward A. Rietman, and Hava T. Siegelmann. Signal propagation: The framework for learning and inference in forward pass. IEEE Transactions Neural Networks and Learning Systems, 35(6):8555-8555 2024. doi: 109/TT9/N 2022..339.2002
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86( 96(11):2278-2344 1998.
Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2015, Porto, Portugal, September 7-11, 2015, Proceedings, Part 15, pp. 498-515. Springer, 2015.
Heung-Chare Lee and Jeonggeun Song. Symba: Symmetric backpropagation free contrastive leam- ing with forward-forwand algorithm for optimizing convergence. arXiv preprint arXiv:2303.084JJ1 2023.
12

---

Published as a conference paper at ICLR 2025
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704 2020.
Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin Akerman. Random synaptic feedback weights support enror backpropagation for deep learning. Nature communications, 7(1): 13276, 2016.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts, arXiv preprint arXiv:/608. 03983, 2016.
Siegrid Löwel and Wolf Singer. Selection intrinsic horizontal connections in the visual cortex by correlated neuronal activity. Science, 255(5041):209-212,, 1992.
Urte Neniskyte and Cornelius Gross. Errant gardeners: glial-celldependeet synaptic pruning and neurodevelopmental disorders. Nature Reviews Neuroscience, 18(1 1):658-670, 2017.
Arild Nokland. Direct feedback alignment provides learning in deep neural networks. Advances in neural information processing systems, 29, 2016.
Alexander Ororbia. Srain-inspired machine intelligence: A survey of neurobollogicaly--lassbbeee credit assignment. 12 2023. doi: 10.31219/osf /o...ff./xukqf..
Alexander Ororbia and Daniel Kifer. The neural coding framework for learning generative models. Nature communications, 13(1):2064, 2022.
Alexander Ororbia and Ankur Mali. The predictive forward-forward algorithm. arXiv preprint arXiv:2301.01452, 2023.
Alexander G Ororbia, Ankur Mali, Daniel Kifer, and C Lee Giles. Backpropagationn-fre deep learning with recursive local representation alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 9327-9335. 2023.
Andreas Papachristocoullou Christos Kyrkou, Stelios Timotheou, and Theocharis Theocharides. In- troducing Convolutional Channel-wist Goodness in Forward-Forweer Learning. In Proceedings of the European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN). Bruges, Belgium, 2023. i6doc.com publ. ISBN 978-2-87587-088-9 doi: 10. http////sann/22...222222 121. URL. /tttps:// org/ / 10. 4 4428/esanm hetp///saann//////// ES2023-121 ESANN 2023 proceedings, Bruges (Belgium) and online event, 4-6 October 2023.
Available from http://www.iddbc.com////
Andreas Papachristodoulou, Christos Kyrkou, Stelios Timotheou, and Thcocharis Theocharides. Convolutional channel-wise competitive learning for the forward-forward algorithm. Proceedings of the AAAI Conference Artificial Intelligence, 38(13) Mar. 2024. doi: 10.
1609/aaai.v38i/ v38113.2936., URL/attpss:///ssoo... aaai. org. Attpa/a./e//eeeeppp/////////// hhhpp/AAA/arttclaa F2lew293363
Myeongjang Pyeon, Jihwan Moon, Taeyoung Hahn, and Gunhee Kim. (SEDONA}: Search for decoupled neural networks toward greedy block-wise learning. In International Conference on Learning Representations, 2021.
Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: functional interpretation of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79-87, 1999.
David Rumelhart, Geoffrey E Hinton, and Ronald Williams. Learning representations by pock-propagating emrrr. nature, 323(6089) 1986.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rXiv:1409.1556. 2014.
Junhao Su, Chenghao He, Feiyu Zhu, Xiaojie Xu, Dongzhi Guan, and Chenyang Si. Hpff: Hierar- chical locally supervised learning with patch feature fusion. arXiv preprint arXiv:2407.05633 2024.
13

---

Published as a conference paper at ICLR 2025
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
James CR Whittington and Rafal Bogacz. Theories of error back-propagation in the brain. Trends in cognitive sciences, 23(3):235-22) 250, 2019.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp- 3-19, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1008.07747, 2017.
Gongpei Zhao, Tao Wang, Yidong Li, Yi Jin, Congyan Lang. and Haibin Ling, The cascaded forward algorithm for neural network training. arXiv preprint MXXiv:333300922: 2023.
14

---

Published as a conference paper at ICLR 2025
A NETWORK ARCHITECTURES
We perform experiments using the network architectures depicted in Figure which include Tiny- CNN, CNN, and ResNet. TinyCNN is shallow 4-layer CNN designed for performance comparison with FF-related methods that are effective in shallow networks. CNN adopts VGG-like (Simonyan Zisserman 2014) architecture with layers, while ResNet is modified version of ResNet18 (He et al. (016) with 17 layers and incorporates shortcut connections. CNN and ResNet are specifically chosen assess the efficacy of our method in training deep models.
TinyCNN
CV-CCme x sc p=z c=10
CV-Cam: m00
OV-CaR: x Sm prl, C- 10
Avergee A podling 2 2x2, 2,5
Avenge
CNN
Block
Block
Block3
Block 4
Block
CW-Cam:: c-70
CWCCam: s-l, c4)
H W x 4 280
CWCCamn 3 3, 5- p-c00
CO-Cam 3x 3, sml p=10000
CW-CaB3 3x3 3 S-1, p-1. 1x000
560
CWCCam x smd p-1, c=000
CW-Came 3 3, 1m1 p-1, c=000
CW-Camc 3x p=1, 0x30
pveragepoding x se2
Aveagpppiiigg x =2
H W x
3, s-1, p-1,
CW-Com:: 3, s-1, p-l.
2
Aveagepdding
CW-Cam:
ResNNet
Residual Block (s 1 2)
Avgoool
CW CW--Comn Comee 3 x 1--1 p-l, c-80
x400 7 7
W 00000 4 T
= x
Reskhall Flork 1: $=1
S=2
Reséhaal Block $=2
Resichal Hikkk S=2
Ci
CW-Come: x sel, p-l, e-
CW-Com: s-1, p-l, Cup
Hu x Win x Gi
Hon We 20m
Resichall Hlockk
Win Cin
CW-Come
CW-Camm
IIn
adkiition
concateration
Figure 6: The details of network architectures for the experiments. (Top): TinyCNN is a 4-layer shallow CNN with a comparable number of parameters to other FF-related algorithms. (Mid): CNN represents a VGG-like architecture with 14 layers. (Bottom): Our ResNet architecture is modified from ResNet18 and its residual block structure illustrated on the right.
15

---

Published as a conference paper at ICLR 2025
B HYPERPARAMETER SETTING
In our method, we utilize Adam (Kingma & Ba 2014) with a cosine annealing schedule (Loshchiloy Hutter 2016). reducing the learning rate from 0.08 0.008, with weight decay of 0.0005, Dropout with a rate of 20% is introduced after all hidden states. The models are initialized using He initialization (He al. 2015). Training is conducted for 150 epochs with a batch size of 128 for all datasets. The experiment with original goodness in Table uses the learning rate from 0.1 to 0.01 without dropout have better performance. The reproduced experiment of CwComp
(Papachristodoulou et al. 2024 uses the same setting original paper.
C DATA AUGMENTATION EXPERIMENT
We conduct experiments CIFAR10 with different augmentation settings based on the ResNet mentioned above architecture. We trained the model using the AdamW optimizer for 1000 epochs, starting with learning rate of 0.08 and halving it every 100 epochs. The experiment without augmentation uses a dropout rate of 20%. There are two augmentation experiments, including standard augmentation and heavy augmentation. The standard augmentation contains random horizontal flips and random crops for a size of 32 with a padding of 4. The enhanced augmentation uses kinds of augmentation techniques, including random horizontal flips, random rotation with degree of 10, random scaling from 0.8 to 1.2, and color jittering by up to 0.2 shifting on brightness, contrast, saturation, and hue. Note that the experiments with augmentation do not dropout, because, experimentally, the dropout causes the convergence too slow with data augmentation.
The training curve on the test is depicted in Figure 7 and the final performance is listed in Table As shown in Table standard augmentation has no significant impact on performance, while enhanced augmentation improves performance from 87.16% 88. 78%. However, compared the performance gains from data augmentation in traditional BP, the improvement with our method is less pronounced. Figure 7 exhibits that training without augmentation converges faster. At the 1 10thh epoch, no augmentation outperforms standard and enhanced augmentation. This result demonstrates that DeeperForward is not sensitive to data augmentation.
90
80
Accuracy (%)
70
w/o augmentation stardard augmentation enbanced augmentation
60
100
200
300
500
600
700
800
900
1000
Epech
Figure 7: Comparison in test accuracy for data augmentation experiments CIFAR10 for 1000 epochs. (Without augmentation): Training on original CIFARIO with a dropout rate of 20%. (Standard augmentation): Standard data augmentation without dropout using random horizontal flips and random crops. (Enhanced augmentation): Enhanced data augmentation without dropout using random horizontal flips, random rotation of 10 degrees, random scaling from 0.8 to 1 1.2, and color jitter.
16

---

Published as a conference paper at ICLR 2025
Table 6: Performances of different augmentation CIFAR1O.
No AUG.
STANDARD AUG.
ENHANCED AUG.
TEST Acc.
87.16
87.47
88.72
D SCALING EXPERIMENT
To evaluate the performance of our method in deeper models, we tested using deeper ResNet (He) et al. 2016) models CIFAR-10 10. We modified ResNet34 and ResNet101 by removing their fully connected layers, renamed as ResNet33 and ResNet100, and replacing the residual structures with those described in Figure 3 The first convolutional layer adjusted to have kernel size of 3 and 100 channels. ResNet33, like ResNet18, uses two kernel size 3 layers as the basic residual block, while ResNet100 employs bottleneck residual block with three convolutional layers of kernel sizes 1, 3, and I, where the third layer's 's output channels are four times the target number of channels. All models retain the original four stages, with corresponding channel numbers { (100, 200, 400, 800), each stage consisting of stacked residual blocks. In the ResNet33 structure, the four stages contain {3, 6, 3} blocks, while ResNet100 contains {3, 4, 23, 3} blocks.
Table 7 presents the performance of various network depths on CIFAR-10, including the ResNet17 structure described in the main text. The experiments were conducted with the experimental setting mentioned in Section and Appendix and trained for 150 epochs on CIFAR-10. The results show that the ResNet17 model performs the best, while increasing the network depth to 33 layers did not yield better results. The ResNet100, 100, which uses a bottleneck structure, performed worse. Tracking each layer's performance revealed significant drops after each I convolution layer, indicating that our method is unsuitable for the bottleneck structure. Additionally, the SIP layer retention results show that all layers were retained in ResNet33, whereas in ResNet100, only 32 layers were retained, with the latter layers being discarded due to poor performance.
Table 7: Performance on CIFARIO for scaling experiments.
RESNET17
RESNET33
RESNETION
ACCURACY
86.22
85.43
83.06
17

---

Published as a conference paper at ICLR 2025
E EXPERIMENTS FOR DIFFERENT TYPES OF LAYERS
Our work uses channel-wise convolution (CW-Conv) for classification instead fully connected (FC) layers. The experiments for different types of layers aim compare the classification performance of different layer types, including fully connected (FC) layer and two convolutional layers with different kernel sizes. The residual structure experiment's 's favorable results in deep layers of ResNet confirm the feasibility of classifying representations from these layers. Specifically, we replace the last layers of ResNet with the layers be tested and evaluate their local performance, as illustrated in Figure 8 The convolutional layer is the channel-wise convolution mentioned in the main body. The FC layer differs from the typical linear classifier, producing 1000 outputs divided equally into 10 groups for the 10 classes. Each class utilizes the mean of outputs from the corresponding group its classification score. Using a general linear classifier with 10 outputs yields poor convergence.
Table demonstrates that CW-Conv outperforms the FC layer significantly. Surprisingly, the experiment reveals that 3 3x3 kernels with fewer weights exhibit better performance. These findings suggest that the fully connected layer is superfluous in method.
Table & Performances on different types of layers on CIFAR1O.
FC LAYER
CONV 5x5
CONY 3x3
ACCURACY
76.92
82.85
83.25
Network Architecture
Layer Type
32 x2
FC Layer
Flatten
CW-Conv: 3 3, p=1,
12800
32 32
FC layer:
12800 x 1000
Residual Block 1: S =
1000
Group
10
Residual Block 2: =
Conv
CW-Conv: x s=1, p=2, c=1000
Residual Block 3: =
Conv 3 x 3
8 x 800
CW-Comy 3, s=1, p=1, c=1000
Test Layer
Figure 8: The network architecture setting of layer type experiments. (Left): The network architecture is modified by the ResNet model, replacing the last residual block with 3 types of layers. (Right): types of layer architectures in this experiment, including FC layer, a 3 3 channel-wise convolution, and 5 5 channel-wise convolution.
18

---

Published as a conference paper at ICLR 2025
F T-SNE VISUALIZATION
T-SNE (Van der Maaten & Hinton 2008), short for t-distributed stochastic neighbor embedding. dimensionality reduction technique commonly used for visualizing high-dimensional data in lower-dimensional space. It emphasizes the preservation of local structure, making it effective for revealing underlying patterns and clusters within complex datasets.
In our experimental setup, we employed t-SNE to analyze the MNIST dataset. We selected t-SNE for its capability to capture intricate relationships between data points and represent them in lower-dimensional space while preserving their local structures. The t-SNE analysis of the MNIST representation was validated using the hidden states of the 1 11th layer before group normalization in ResNet, which represents the last layer after the pruning operation conducted by the signal integrating and pruning (SIP) module.
The t-SNE visualization result is depicted in Figure 2 Before model training, the point clusters of each class in the MNIST dataset overlapped on boundaries. However, after training, a noticeable transformation occurred: the clusters became more distinctly separated, with discernible boundaries between them. This evolution underscores the model's 's ability to learn discriminative features and improve class separability.
Our utilization of t-SNE provided valuable insights into the dynamics of the MNIST dataset and highlighted the efficacy of our model in enhancing the separability of MNIST.
Initialization
Trained
Figure 9: The t-SNE visualization of the MNIST representation at the hidden state of the 11th layer before group normalization in ResNet trained on MNIST, both before (initialization) and after training (trained). Each color corresponds different class.
19

---

Published as a conference paper at ICLR 2025
G IMPLEMENTATION OF MODEL PARALLEL STRATEGY
Without the limitation of update locking, our method enables model parallel strategy using pipeline programming. Pipeline programming is a parallel computing paradigm where complex tasks are divided into series of independent and sequential stages. Each stage performs specific task and passes its results to the next stage. This programming model resembles a factory assembly line, where each worker is responsible for completing particular task in the process without waiting for the completion of the previous task. Tasks are divided into multiple stages that can execute simultaneously. thus speeding up the overall processing. By employing pipeline programming, can enhance the efficiency and performance of computer systems, achieving faster data processing and reduced response times. The advantages of this programming model include parallelized processing, reduced latency, increased throughput, and better utilization of hardware resources.
Figure I0 illustrates our method's model parallel strategy implemented using pipeline programming based on mulli--treeding techniques. The network is divided into a series of independent and sequential threads. Each layer is organized into thread to execute local optimization and generate data for the subsequent layer. As depicted in Figure I0, we employ first-in- first-in-first-out (FIFO) queues for communication among threads. The training set serves as the first FIFO designated for the initial layer. Each layer independently executes its specific task without waiting for the completion of other layers. This approach enables high parallelism, enhancing throughput, and optimizing hardware resource utilization. Algorithm provides detailed insights into the implementation of each thread's functionality.
For clearer presentation, simplify channel-win convolution to the following equation:
(Z. = CWConv(X:0,c)
(14)
where ý and Z. represent the classification output by the channel-wise convolution and the features extracted via layer normalization, respectively. X the input, O denotes the convolutional kernel parameters, and G is the number of classes.
Thread
Thread
RIFO
F
%ja
F
bout
FIRO
PIFO
Fin
Fout
FIFO
Fin
Fout
FIFO
CW-Com
CW-CouIV
LCB
V0
CcB
Figure 10: The implementation of the model parallel strategy of our method using pipeline program- ming based on the multi-threading technique. The network is segmented into a series of independent sequential threads, with each layer arranged within thread to execute local optimization and produce data for the subsequent layer. Communication among threads is facilitated by frst-in-first-on (FIFO) queues. It is important to note that we use the training set as the first FIFO, designated for the initial layer.
20

---

Published as a conference paper at ICLR 2025
Algorithm 1 A thread for local layer updates
Input: Input FIFO Fin- oulput FIFO Fott, FIFO for shortcut input F sin FIFO for shorteut output
F Fsoutt parameters of convolutional kernel 0, number of class G, learning rate 1), number of minibatch N.
2:
for = 1....... NN do
3:
Wait for Fin not empty
4:
(X,y) + Fin // pop data from Fin into (X,y)
5:
(Z.5) = CWConv(X;0.0o
6: // residual structure
7:
if residual structure existed then
8:
Wait for F Fsin not empty
9:
Z. + Fsin // pop data from Fsin into Z,
if Z, and Z differ in spatial dimension then
11:
Z, AvgPool(Z.) // downsampling Z, to match Z in spatial dimension
end if concatenation type then Z Concat (Z, Z.) else
13:
15:
7 + Zr
17:
end if
18:
end
19:
if output connect shortcut then
20:
Foout < push Z into F Fsout
21:
end if
22:
Fout < yZ.y) // push (Z,y) into Fout.
23:
I/ uupdate layer
24:
V0 = V VeLcE
0= n 00
26: end for
21

---

Published as a conference paper at ICLR 2025
H MEMORY-SAVING STRATEGY
Due to the update locking problems, BP requires a large amount of memory to store intermediate states throughout the process. Without this constraint, our method can employ memory-saving strategy, efficiently releasing memory once the update for a layer is completed, as shown in Algorithm 2 In this strategy, the memory requirement is determined by the layer with the largest memory demand. This memory-saving strategy is particularly suitable for scenarios with constrained memory resources, such as edge computing.
Algorithm 2 DeeperForward
Input: Dataset E [Bi, B2, Bv B, = (X,,,,), number of
layers L, number of classes G, learning rate
2: for = 1......, N.., do
(Z .)) B,
3:
4: for = 1, L do
(z ,()))
(WC)nv(ZU-1);@o
Eq. 14
6: v o CCE (p)
.))
Eq. 13
7: ( = (( - 17 vg
8: Release memory ((1 z(()) and VO(
end for
10: end for
22

---

Published as a conference paper at ICLR 2025
I DEACTIVATION NEURONS RATIO
To better explore the improvements of our method on the deactivated neurons issue, we conducted experiments and analysis the deactivated neurons ratio across different layers. The experiments compared our combination of mean goodness and layer normalization with the original square goodness and normalization of the vector length, both tested on the same 17-layer ResNet structure in the CIFARIO dataset. Both networks were trained with the same settings for 150 epochs on CIFAR1O. The average deactivated neurons ratio for each layer was tested on the CIFARIO test set, as shown in the Figure 11
The results show that our mean goodness design indeed significantly reduces the deactivated neurons ratio. Specifically, in the shallow layers, square goodness deactivates a large number of neurons by focusing on outliers. Moreover, we observed that the mean goodness deactivated neurons ratio increases as the depth of the network increases. This phenomenon aligns with our observation that high-level feature matching becomes sparser in deeper layers, which also explains why deeper networks (such as ResNet33 and ResNet100 in Appendix D) do not show improved performance extracting higher-level features from high-level features becomes increasingly difficult.
This experiment confirms that method effectively addresses the deactivated neurons issue present the original FF. Additionally, we further observe that high-level features extracted deeper layers relatively sparse, limiting the extraction of higher-level features. This results in deeper network layers failing to achieve optimal performance, which is a topic worthy of further exploration.
Deactivated Ratio of Mean and Square Goodness
100
MM
Deactivate Ratio (%)
80
70
60
Mean(Ours) urs)
50
Square
40
5
11
13
15
Layer
Figure 11: Comparison of mean goodness and square goodness in average deactivated neurons ratio per layer in 17-layer ResNet on CIFAR10's test set.
23

---

Published as a conference paper at ICLR 2025
J IMPACT STATEMENTS
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of work, none which we feel must be specifically highlighted here.
24


Reviews:
[{"review_id": "5IekM7WldH", "reviewer": "Reviewer_occo", "summary": "The current paper proposes a novel design of bio-logically plausible algorithms utilizing layer normalization and mean goodness, which extends the Forward-Forward to non-trivial networks. Extensive experiments across datasets and models validate the approach.", "questions": "Can you show that the portion of deactivate neurons is reduced?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper enables training conventional models with the FF algorithm. \n\nDiagrams are well designed, clear, and nice. \n\nThe explanation of the method is clear.\n\nThis approach scales well to multiple GPUs\n\nI like the well written background section.", "weaknesses": "I would like to see experiments on Imagenet or other large scale datasets. \n\nI'm interested in an experiment that shows that the portion of deactivated neurons is reduced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VZQRxS4VBc", "reviewer": "Reviewer_GH5m", "summary": "This work aims to improve forward-forward training and enable deeper networks by providing layer normalization and mean goodness to overcome certain limitations.  Experiments show that the performance can be improved even with 17-layer CNNs.  Datasets like CIFAR-10, MNIST, and Fashion-MNIST are involved in illustrating the benefits of the proposed DeeperForward method.  In addition, a model parallel strategy and memory saving strategy are proposed to enhance training efficiency and costs.", "questions": "See above as the questions are mainly addressing weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The brain-inspired forward-forward approach is certainly an interesting and novel area to explore. It’s great to see efforts advancing in this direction.\n\nThe writing and presentation are easy to follow.\n\nThe experiments in the study are substantial.", "weaknesses": "The main issue of the study is the performance of the proposed DeeperForward method.   Compared with sota models using BP, DeeperForward falls behind on Fashion MNIST and has an even bigger gap on CIFAR-10.  Its accuracy on MNIST is marginally comparable to other methods. However, that is not particularly strong considering the simplicity of MNIST.  The results suggest the exploration of FF is ongoing and still faces some challenges.\n\nLine 309, \"we employ a combination strategy for selecting consecutive layers\"  It is not clear what the strategy is and how this strategy reduces the complexity from O(2^L) to O(L^2).\n\nIt is not clear how the number of layers would impact the performance.  Is 17 the top limit? Would an architecture larger than 17 layers suffer from some kind of vanishing gradient problem?\n\nDespite some discussions in Section 4.5, the computational cost aspect of DeeperForward is inadequate. What is the total GPU time needed to complete a task, for example, CIFAR-10 classification?  That measure would allow comparison with other non-FF methods more comprehensively. This should be a critical part of this study. \n\nThe memory-saving strategy lacks details.  Appendix H does not provide any measures or evidence on how this strategy could reduce memory usage for DeeperForward.\n\nSimilarly, the parallel strategy needs more details as well.  What if the number of GPUs is more than 2?  Also, what would happen if the number of GPUs is greater than the number of layers?  Would the extra GPUs stay idle? Fig 4 does not provide such insight.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gek9MyM4QK", "reviewer": "Reviewer_YDPM", "summary": "The author proposed a novel approach, DeeperForward, to enhance the Forward-Forward algorithm. This paper introduced a goodness design based on layer norm and mean goodness to solve issues like feature scaling and deactivated neurons. This paper also demonstrates the effectiveness of DeeperForward using 17-layer CNN on various datasets. Additionally, this paper states a model-parallel strategy to reduce training time.", "questions": "See weakness above", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed modifications, especially mean goodness and layer normalization, effectively address the limitations of the FF algorithm in deeper architectures.\n\n2. The model-parallel strategy greatly enhances the training efficiency on multi-GPU setups, enabling the method to scale well across deeper networks.", "weaknesses": "1. **Dataset Size Limitations**: The reviewer acknowledges that DeeperForward has primarily been tested on relatively small datasets, such as CIFAR-10 and MNIST. While these datasets are widely used benchmarks, the reviewer agrees that evaluating DeeperForward on larger datasets, such as ImageNet, would provide more insight into its scalability and effectiveness. The reviewer recommends conducting experiments on larger datasets to assess their performance under different conditions thoroughly.\n\n2. **Choice of Architecture Depth**: The reviewer notes that the 17-layer CNN was selected as a balance between computational efficiency and depth sufficient to highlight the advantages of DeeperForward over existing methods in moderately deep networks. However, the reviewer wonders if testing DeeperForward on even deeper architectures (e.g., 50 or 100 layers) would provide valuable insights. The reviewer encourages extending the evaluation to larger models to assess its performance and generalization capabilities on very deep networks.\n\n3. **Accuracy and Comparison with Block-wise Learning**: The reviewer points out the lack of significant accuracy improvement in Table 1 compared to non-FF and other FF methods. While the method focuses on improving training efficiency and enabling deeper networks, the reviewer highlights recent works in block-wise learning[1,2] that have shown greater improvements in accuracy, surpassing backpropagation by over 5%. The reviewer is curious if DeeperForward could incorporate block-wise learning concepts to improve accuracy while retaining the benefits of the forward-forward training scheme.\n\n4. **Scalability on Multiple GPUs**: The reviewer observes that the speedup ratios in Table 5 reflect performance gains using one and two GPUs but would like to see tests on more GPUs to provide a more comprehensive view of DeeperForward's scalability. The reviewer suggests extending parallelization experiments to include setups with more GPUs to provide a broader evaluation of how the method scales with additional hardware resources.\n\n5. **Training Curve Convergence**: The reviewer notes concern regarding the convergence behavior shown in Figure 7. The training curve does not fully converge within the current number of epochs, and the reviewer would like to see further experiments to determine the number of epochs required for stable convergence. The reviewer recommends updating the results with extended training to provide a clearer picture of when and how the method reaches its optimal performance.\n\n[1] Cheng, Anzhe, et al. \"Unlocking Deep Learning: A BP-Free Approach for Parallel Block-Wise Training of Neural Networks.\" ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024.\n\n[2] Pyeon, M., Moon, J., Hahn, T. and Kim, G., 2020. Sedona: Search for decoupled neural networks toward greedy block-wise learning, in International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I4eqqeg1Ai", "reviewer": "Reviewer_eCNM", "summary": "This paper presents the DeeperForward algorithm, extending the Forward-Forward approach to deeper networks. The paper introduces a goodness design, combining mean goodness and layer normalization, which partially addresses issues in the effective training of deep networks: feature scaling, redundant normalization, and deactivated neurons. It’s a bit innovative but not very clear in expression.", "questions": "Q1. ‘This approach addresses the bio-plausibility issues of backpropagation and overcomes the limitations of FF concerning model size.’ In the Method section, the problem solved by this method is inconsistent with the problem you mentioned in the introduction that needs to be solved. Does model size refer to feature scaling? What is the relationship between the bio funding issues of backpropagation and deactivated neurons?\n\nQ2. ‘The pruning process aims to select an optimal combination of layers for refining the network and eliminating redundant layers.’ How is it selected and what are the criteria for selecting combinations?\n\nQ3. Where was Xr extracted from in Figure 3? Any layer? Because deeper layers contain higher semantic information, where Xr comes from is a critical issue for performance.\n\nQ4. The DeeperForward you proposed is similar in principle to the methods[1,2] in the Local Learning field. What are your strengths? \n\n[1] Xiong, Yuwen, Mengye Ren, and Raquel Urtasun. \"Loco: Local contrastive representation learning.\" Advances in neural information processing systems 33 (2020): 11142-11153.\n\n[2] Su, Junhao, et al. \"HPFF: Hierarchical Locally Supervised Learning with Patch Feature Fusion.\" arXiv preprint arXiv:2407.05638 (2024).\n\nIf the author resolves my questions, I will consider raising my rating.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The article proposes an innovative design based on the original FF method to solve the inherent problems of BP and FF.\n- From the experimental results, it can be seen that DeeperForward has good performance.", "weaknesses": "- The abstract lacks a brief description of the new parallel strategy.\n- Figure 1 is confusing. Firstly, it would be more appropriate to compare the performance gap between your method and SOTA in Figure 1. Secondly, your method only has a single forward propagation path, which is inconsistent with the enhanced FF in your title.\n- ‘However, this approach fails to decouple goodness and leaks goodness to the next layer, hindering deeper layers from learning new features and causing overfitting.’ Authors have repeatedly used this general and assertive conclusion in the introduction to evaluate the drawbacks of other methods, but I have not seen a specific reason. This is not very detailed and almost unconvincing. Moreover, it is evident that this conclusion has nothing to do with feature scaling.\n- ‘Although CwComp (Papachristodoulou et al., 2024) improves performance using squared goodness and batch normalization, it leaks goodness information, leading to overfitting in deeper networks.’ According to your description, the omission of goodness of fit information directly leads to overfitting, which is clearly not convincing. Any defects in the model can lead to common problems of overfitting. There is a lack of deeper reasons between the two.\n- ‘We adopt layer normalization for better feature scaling with identical mean and standard deviations, instead of L2 normalization.’ You have expressed the drawbacks of L2 normalization and proposed so-called alternative methods. But why do we need to use your method to replace it? What is the motivation? You cannot simply express that the original method has shortcomings but your method has advantages, and then your method can become an alternative solution without deeper motivation and logic.\n- The format of subheadings under section 3.2 is inconsistent.\n- From Sec4.1, it can be seen that GPU memory is sufficient, but there is a lack of experiments on large datasets such as ImageNet.\n- 8.Sec4.5 only compared GPU and speed with BP, but according to what you described earlier, you have made improvements based on FF. There is a lack of results compared to FF here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)