The Trifecta: Three simple techniques for training deeper Forward-Forward networks
Authors: Thomas Dooms
Ing Jyh Tsang
Jose Oramas
Venue: Accepted by TMLR ()

Abstract: Massive backpropagated models can outperform humans on a variety of tasks but suffer from high power consumption and poor generalisation. Local learning, which focuses on updating subsets of a model's parameters at a time, has emerged as a promising technique to address these issues. Recently, a novel local learning algorithm, called Forward-Forward, has received widespread attention due to its innovative approach to learning. Unfortunately, its application has been limited to smaller datasets due to scalability issues. To this end, we propose The Trifecta, a collection of three simple techniques that drastically improve the Forward-Forward algorithm on deeper networks. Our experiments demonstrate that our models are on par with similarly structured, backpropagation-based models in both training speed and test accuracy on simple datasets. Specifically, we achieve around 84\% accuracy on CIFAR-10, a notable improvement (25%) over the original FF algorithm.

Published in Transactions on Machine Learning Research (10/2024)
The Trifecta: Three simple techniques for training deeper Forward-Forward networks
Thomas Dooms
thomas. Sontaaddddoomtuuuuueeeeeeee
Deportment of Computer Science University of Antwerp
Ing Jyh Tsang
intoontsangg@imec....
Department Computer Science University of Antwerp, imec
Jose Oramas
jose. wwwwaaaawanterrrrrrrrere
Department of Computer Science
University of Antwerp, sqIRI/IDLab, imec
Reviewed on OpenReview: https: openreview. net/ forum? d7KPPuo0FF
Abstract
Massive backpropagated models can outperform humans on variety of tasks but suffer from high power consumption and poor generalization. Local learning, which focuses on updating subsets of a model's parameters at a time, has emerged as a promising technique to address those issucs. Recontly, a novel local lcarning algorithm called Forward-Forward has received widespread attention due to innovative approach to learning. Unfortunately, its application has been limited to smaller datasets due scalability issues. To this end, propose The Trifecta, collection of three simple techniques that drastically improve the Forward-Porwarr algorithm deeper networks. Our experiments demonstrate that our models are on par with similarly structured, backpropagation--aasd models in both training speed and test accuracy on simple datasets. Specifically, we achieve around 84% accuracy on CIFAR-10, a notable improvement (25%) over the original FF algorithm.
1 Introduction
Despile its pervasiveness, backpropagation not flawless; has inherent challenges that have puzzled re- searchers for decades. One such challenge is that it usually produces overfitted models, models that memorize the training set and fail to generalize to unseen data. (Erhan al. (010)). Over the years, rescarchers have found ways to mitigate the issue by introducing dropout (Srivastava et al. 2014h simply increasing the dataset size through augmentation techniques by collecting more data. However, with modern datasets reaching near-internet size (Gao et al. 2020 Schuhmann et al. 2022 Carlini et al. (021), this provisional solution cannot be scaled further. Morcover, contemporary models are tricky to scale both locally (Nagel et al. 2 due memory issues and across machines (Huang al. 2019) due to communication overhead. The need for sequential forward and backward passes leads to the phenomenon known as backward locking, hampering the efficient distribution of computation (Huo al.] (2018). Lastly, backpropagation struggles with vanishing exploding gradients, which affect learning and stability, especially in deeper networks Glorot & Bengio
2010: Bengio et al. 1994p. Rescarch has unveiled sev- eral mitigations (He et al. 2016 loffe & Szegedy 2015 He et al. 2015 Glorot & Bengio 2010), but this
[Hochreiter & Schmidhuber 1997
still remains an issue in large models [Chowdhery el. al. 2022
The combination of these systematic issucs and the biological implausibility of backpropagation [illicrap et al. (2016) has prompted the machine learning community to explore alternative learning algorithms. One
such approach is local learning (Bengio et. al. 2006 Hinton et al. 2006; Belilovsky et al. 2019 Nø kland

---

Published in Transactions on Machine Learning Research (10/2024)
2016b), which learns from nearby information, instead of full error signal. Limiting the information flow in the full network restricts the network's is ability to memorize inputs and helps to regularize gradient computations. Further, the reduced communication facilitates the efficient deployment of these networks
across several accelerator units [Gomez et al. 2022 Laskin et al. 2021
Layer-wise or greedy learning is a popular family of local algorithms that aims to learn each layer neural network independently (Bengio et al. 2006 Belilovsky et al., 2019). It avoids backward locking during training and enables non-differentiable operations in the forward process. However, it often requires auxiliary
classification heads or involves expensive mathematical operations (Belilovsky et al.) 2019 Löwe et al. 2019
No kland, 2016b). Recently, new local learning algorithm called Forward-Forward (FF) Hinnonn 2022) has
sparked interest in the machine learning communily. FF learns by maximizing or minimizing goodness from the activations of each layer for positive (genuine) and negative (bogus) samples, respectively. However, initial experiments show that this algorithm is unable to scale beyond the simplest of datasots. Therefore, this paper, we aim to address this issuc by pointing out three weak points of the algorithm and remodying them.
Loss Function In line with recent work (Lee & Song 2023 we find that the proposed loss function from Hinton (2022) has several weaknesses. Primarily, due to asymmetrics in the loss landscape, convergence is severely hindered. Moreover, the original loss function has a threshold parameter, which determines around which value of goodness the separation of positive and negative samples should occur. Our experiments show that this hyperparameter has a large impact on the final accuracy of the network, which undesirable.
Normalization Function
The objective function of each layer is to separate positive and negative samples much possible. This can cause issuc where layers can simply retain scale prior features to achieve high separation without learning anything useful, as noted by Hinton (0022). Their proposed solution is to normalize the features length-wise at the start of each layer such that any information about the previous prediction is removed. This work confirms the occurrence of this issue but claims that the solution suffers from several flaws such as information loss and instability.
Lack of Error Signals A drawback of many greedy or local learning algorithms is the lack of backward error signals. This means that layer may learn features that useful for its local objective but not necessarily for the next layer. Consider the extreme case where layer generates uniform feature vector (k 1) above or below the threshold according to the belief of the layer. Locally, the objective will be fulfilled, but subsequent layers will not acquire useful information that can help separate the data further. Consoquently, the local objective must be suitable, or such models will not be able to scale to complex tasks.
1.1 Contributions
We propose combination of straightforward techniques, termed The Trifecta, that matches the ac- curacy of our backpropagation baseline on several datasets by addressing the mentioned weaknesses.
We achieve around 84% accuracy CIFAR-10 with 12-layer model, demonstrating that FF can scale to more complex datascts and architectures.
We perform analysis two different loss functions for FF and discuss their differences in training stability. This reveals that using a dynamic threshold significantly helps convergence in deeper networks.
We conduct thorough investigation into different normalization functions in the FF algorithm, which demonstrates that batchnorm strikes good balance between feature recycling and information loss.
We provide an implementation of The Trifecta [ttt:://gttabb.om///ooomm//iiiie
2 Related Work
Before the advent of end-to-end backpropagation (E2EBP) (Krizhevsky al. 2012). : often simply called
backpropagation (BP), the investigation into algorithms to train neural networks was prevalent in machine
2

---

Published in Transactions on Machine Learning Research (10/2024)
Hinton et al. 2006p. Since the backpropagation breakthrough, research has mostly shifted to improving this algorithm. Recently, however, there has been a resurgence of interest in alternate learning methods due to the limitations of backpropagation (Jaderberg et al. 2017 No kland 2016b Huo et al., 2018 Löwe et al.) 2019). An early example of a localized approach in deep networks is the ioneption Net (Szegedy et al.) 2015), which incorporates multiple classification heads on intermediary layers throughout the network. Although the network still employs backpropagation, the in- termediate heads aid gradient flow, enabling the training of deeper networks. Several years ago, a research wave focused on truly local algorithms began with the introduction of Decoupled Neural Interfaces (DNIs) (Jaderberg et al.] 2017). DNTs employ independent sub-networks, called modules, allowing straightforward forward passes by sequentially passing inputs through all modules. The backward pass updates weights using synthetic gradients approximated using an auxiliary network trained ofline with true gradients. This decoupling of gradient flow permits asynchronous forward and backward passes. However, the drawback of this approach is diminished accuracy, even on small datascts. More recently, Forward Gradient techniques have shown more promise. These methods calculate directional derivatives based on either random directions (Baydin et al. 2022) or directions informed by local auxiliary networks (Fournier et al. 2023). The resulting directional derivative serves as an estimate of the gradient along the chosen direction. Concurrently, increas- ingly biologically inspired methods have emerged, starting with feedback aligmment (FA) (Lillicrap et al. 2016) that proposes to use random feedback matrices to resolve the implausibility of synaptic symmetry within backpropagation. Nø kland (2016a) alters this approach by using random matrices to directly update each layer, partly resolving the backward locking issue. Lastly, PEPITA (Dellaferrera & Kreiman 2022)) uses two forward passes to learn: a standard pass and a modulated pass. The first pass comparable to an ordinary forward pass in backpropagation and computes the network error for a given input. The original input is then perturbed by this error and used by the second pass to update the weight directly in forward manner.
learning research (Hinton & Sejnowski 1983
The Forward-Forward algorithm (Hinton The FF algorithu aims to overcome several limitations tied to backpropagation and provide a biologically plausible learning algorithm. This is achieved by adopting contrastive learning principles to separale positive and negative samples based on network activations. The lack of error signals and the generic objective make the algorithm very flexible in many scenarios where backpropagation is not feasible. Unfortumately, this approach often results in reduced accuracy on complex datasets. Some work has been done to improve the scaling characteristics of the FF algorithm. Specifically, the original loss function exhibits conditioning issues due to its suboptimal loss landscape, resulting in difficult optimization. To mitigate this, modified loss function known as SymBa directly utilizes the separation instead of the distance to a threshold. This leads to improved properties and significantly enhanced performance (Lee & Song 2023).
2022) has been proposed as a novel approach to deep learning.
This paper explores and improves the achieve this by proposing three minor modifications to the algorithm, including using the SymBa loss function [ee Song 2023), batch normalization (Toffe & Szegedy 2015), and overlapped local updates (Laskin et al. 2021). To the bost of our knowledge, we are the first to invostigate the effect of the latter two in Forward- Forward context. Further, we do not specifically focus on the biological plausibility but attempl to stay as close to the nature of the algorithm as possible. For instance, we employ convolutional neural networks (CNNs) (LeCun et al. 1989) due to their ubiquity in image processing, even though weight sharing is not biologically plausible Grossberg 1987).
algorithm in an image classification context. We
3
Background
The Forward-Forwatt algorithm is biologically plausible alternative to backpropagation. Instead of min- imizing the wross-entropy between the model's outputs and a one-hot encoding of labels, it replaces the backward pass from backpropagation with a second forward pass, hence the name. The first forward pass contains positive (real) data, and the second pass contains negative (bogus) data. The main idea is that each layer of the network should learn to distinguish this data as well as possible. To this end, Hinton (2022) considers the magnitude of hidden activations, which they call goodness. This goodness should be high for positive samples and low for negative samples. Generally, the l2 norm is used to determine the goodness, but variants exist.
3

---

Published in Transactions on Machine Learning Research (10/2024)
Importantly, the neural network structure and design remain the same; each layer modifies the incoming features according to their weights and forwards the result. to the next layer. Intuitively, instead of using a gradient of the global loss (computed at the output layer) with respect to the weights, a proxy loss (the goodness) computed at each layer and applied locally, obviating the need for backpropagation. This brings about three differences compared to ordinary supervised training, which we discuss in detail in this section.
3.1 Negative Sampling
At the core of the learning algorithm lies learning the distinction between positive and negative samples. Consequently, the nature and quality of negative samples strongly influence the learning process. While one can imagine multitude of ways to approach this, we focus creating negative and positive samples in supervised manner. Specifically by adding the label to the input. In this case, positive sample is a pair in which the impul and label match. Conversely, negalive sample has labels and inputs that don't Won't match. To distinguish matching pairs, the model has to learn the structure of the imputs. In some cases, one should be careful about how this pair is created. Consider a CNN-based model where the label encoded in a limited number of pixels; then, not all parts of the model could access this data. Due to the lack of backward signal, this may seriously hamper learning. While such contrastive approach has some drawbacks, often rooted in lower computational efficiency, it is much more flexible. For instance, one could focus on the negative pairs that the model often mispredicts are important to the task.
3.2 Weight Updates
Rather than performing forward prediction step followed by a backward update step, two forward prediction steps are performed on positive and negative data. Each layer updates itself locally (without receiving gradients) based on the generated goodness with the following loss function.
LFp(g gg n00) = E log exp(T- - s) + Σ log(1 exp(g neg - r))
Here, gmd* and geeg refer to the vector of the goodness values of the positive and negative samples in each batch, respectively, and т a threshold. This threshold defines how strongly the positive and negative samples should be separated. Finally, represents an entry in batch. This function will produce low loss values if both the goodness of positive samples is well above the threshold and well below the same threshold for negative samples.
To update each layer's weights, the optimizer step is performed using the gradient of this loss function using a standard optimizer [] The update rule for SGD, for example, would be:
W++1 W. LLrrr(( neg)
where W, represents the weights at time step i, is the learning rate, and Ler ne0)) is the gradient of the loss function with respect to the weights. We illustrate this algorithmically in
Algorithm Original Layer Update Step
1: hidden441 f layer, (hidden,)
activations + out. now (o..po@...attttnttttttttttt flaattensstart dim=1)
3: pos, neg + ieeeivations.chunk/
4: loss <- softplus(T pos). meann) + softplus(neg - r). meaan(
5: loss.optimize()
return dotach (hiden,)n)))
Alternatively since the update step only uses local weights, one could exact methods on small datasets.
4

---

Published in Transactions on Machine Learning Research (10/2024)
3.3 Evaluation
During evaluation, the goodness across multiple layers determines the certainty in the prediction of the provided sample. In the supervised setting, forward pass with each label must be performed to attain the logits of the n-way classification. This can be inefficient, especially in scenarios with numerous classes, such as ImageNet. (Deng et al. 2009). Alternatively, the features of the last layer in the network can be used to train a classifier; this mitigates the linear complexity of n-way classification Due to the novelty of the algorithm, different existing training and evaluation strategies are largely unexplored. Consequently, FF often exhibits lower accuracy and performance compared other local learning techniques. Nevertheless, this presents promising avenue for investigating learning algorithms and gaining insights into the learning process of deep networks in general. Appendix H discusses some experiments and ideas in this area.
This work focuses on the simplest version of the Forward-Forwan algorithm. We encourage the reader to refer to the original work (Hinton algorithrss.
2022 for a more complete intuition and connections to other learning
4 The Trifecta
The main contribution of this work is The Trifecta, which aims solve three observed weaknesses in the forward-Forward algorithm: the loss function, the normalization, and the lack of error signals. As our experimentation reveals, there is a positive synergy between the three components; combining the three yields higher gains than their separate improvements.
4.1 Symmetric Loss Function
Consistent with (Lee & Song 2023), we find that the proposed loss function in FF not equal for false positives and false negatives the same scale, leading to gradients that do not directly point toward the minimum loss. Additionally, the negative goodness is limited for many threshold values. The original paper (Hinton 2022) noted that minimizing goodness, instead of maximizing it for positive samples, yields higher accuracy. This suggests that it is easier for the network to distinguish negative examples very well according to how bogus they are. Appendix C provides a comprehensive mathematical analysis of these issues. To address these issues Lee & Song (2023) proposes the SymBa loss function that relies on directly optimizing for the separation gap, shown below.
is not possible to have negative goodness which means the separation of
Ly ( (
Σ log(1 + exply,"" 9, )re9))
This resolves the aforementioned issue. This approach works best when the positive and negative samples are highly correlated, which is the case with the supervised approach. Additionally, this function can be improved by scaling the separation with a constant term this penalizes wrong classifications further, akin to focal loss (Lin et al. 2018). Our work, therefore, opts to employ this loss function. In Appendix D we further discuss the advantages and disadvantages this loss function.
4.2 Batch Normalization
Prior to each layer, Hinton 2022 suggests normalizing the longth of the activations, akin to simplified This serves two purposes: it prevents prediction information from leaking
form of layer normalization (LN)
IIn the supervised approach, label must still be provided during evaluation. The original work Hinton 2022 uses 'neutral' label which is average of all labels. Limited experiments show that simply using random label works slightly better in practice.
IIn this work refer to this reduced form of normalization that does subtract the mean simply layernorm. All experiments in this regard are verified hold for both this reduced version and the standard PyTorch layernorm implementation Torch Contributors 2023).
5

---

Published in Transactions on Machine Learning Research (10/2024)
into the next layer and forces the network to learn completely new features at each layer. By scaling the network depth-wise, each layer has alternate insight into the data which is used by the final prediction.
However, we find this not to be the case in the original version of FF; deeper layers do not produce features that lead to higher accuracy. Instead, the accuracy plateaus by scaling the depth (Figure 1). We argue that this stems from the combination of the normalization and the lack of error signals. This disruptive normal- ization that removes all previous prediction information prohibits the network from creating incrementally useful and specific features observed in hierarchical learning in backpropagated networks. In combination with the fact that there are global error signals, the network is unable to optimize across layer boundaries and therefore cannot improve downstream accuracy. In essence, each layer has to fully re-separate the pro- vided features, leading to a collection of single-layer classifiers with shallow insight. This stack of classifiers operating on the output of their predecessor leads to high instability, especially at later layers, shown in Figure 2
Intuitively, we seek a transformation that strikes the balance between doing nothing, which leads to recycling features, and discarding all prediction information which leads to each layer having re-separate all features. To this end, propose to use batch normalization (BN) Gloffe & Szegedy 2015). Batch Normalization normalizes features within each batch without removing the information regarding the previous prediction. Further, as this normalization is dependent on batch statistics, which are not available to any single sample, any subsequent layer cannot exactly reconstruct the original features. Additionally, directly using batch- normalized features most likely will not lead to high goodness, especially these lead to a wrong prediction. This forces new features or insights to be created at each layer, which empirically demonstrated on the bottom of Figure again. This leads to lower weights Figure 10 drastically stabler gradients Figure 2 and therefore better generalization. This is discussed more rigorously and intuitively in Appendix
In essence, each layer has the ability to alter features instead of separating them
Lavemorm
Batchnorm
LAG11
1010'3
18019
1010
101522
L0Nd73 UNV 100 100rs
27
07-
2.
05
255
03
0.4
0.4
112.
033
07
01
01
Figure 1: Layerwise test accuracy for both normalization schemes. Both experiments use SymBa, but not overlapped local updates on the shallow CNN network (VFF-SymBa/s) on CIFAR-10.
4.3 Overlapping Local Updates
As means to introduce error signals into FF, the original work (Hinton 2022 work that uses the goodness of the subsequent and previous layers. However, for simple classification, this requires the network to be evaluated several times. Therefore, we propose using another technique from recent research (Huo et al. 2018 Laskin et al., (021) that uses semi-local error signals within the context of backpropagation. This approach involves training layers in groups in an alternating and overlapping pattern. Using group size simply results in greody lcarning while using group size 2 leads Overlapping Local Updates (OLU). more thorough explanation and further implementation details can be found in Appendix F
proposes recurrent net-

---

Published in Transactions on Machine Learning Research (10/2024)
Layer
Layer
GROTOm 2907170
50g
2001
20g 2030
R891
M00
st5p
Figure 2: &2 norm of the gradients of layers and 6 each step during training using the FCN network (VFF+SymBa/[) CIFAR-10. OLU not used due to its potential influence the results. Using batchnorm yields more stable gradients in deeper layers.
essence, this technique optimizes each layer with two alternating objectives. When the layer last in the group, is updated to satisfy the local objective: maximizing goodness. Conversely, when it first the group it updated to better satisfy the objective of the subsequent layer: help the subsequent layer maximize goodness. Intuitively, the former promotes local understanding of the data and the latter prompts the layer to increase the usefulness of its representations for the next layer. In essence, in accordance with the original idea of Hinton (2022). local weights are updated according to the beliefs of the subsequent layer. Many alternate techniques exist to incorporate error signals into local networks (Ororbia & Mali, 2023 Lillicrap et al. 2016) in a more biologically plausible manner, however, exploring these is deferred to future work.
In summary, these three changes result in the algorithm (shown in to update layer.
Algorithm Modified Layer Update Step
1: hidden;41 + hyver,hhdden,
2: (i mod 2 = 0) + (iteration mod 2 = 0) then
return hidden:41
end if
activations (ol..oow(2).laaeeennft)) dim=1)
pos, neg irtivattonsschuunk2
7: 8 + neg.mean(1) pos.mcan(1)
8: loss < softplus(a- (aaisss((---mmmmaea 6).mean()
9: loss.optimize()
10: return
5 Experimental Setup
Conforming to Hinton 2022), all experimentation in this work concentrates on image classification to measure
the capabilities the Forward- Forwardd algorithm, specifically on the MNIST, Fashion-MNIST. SVHN, CIFAR-10, and CIFAR-100 datascts (LeCun & Cortes [010 Xiao al. 2017 Netzer et al. 2011 Krizhevsky 2009). These datasets provide a good difficulty mix assess the behavior of the algorithm different scenarios.

---

Published in Transactions on Machine Learning Research (10/2024)
5.1 Training and Encoding
In this work; we opt to exclusively employ the supervised approach to the Forward-Forward algorithm described in Hinton (2022). Further, we opt to use CNNs due to their ubiquitous nature in image classification. This combination poses some problems as the one-hot label encoding (ID) cannot be trivially concatenated to an image (3D). Additionally, the encoding must be concatenated in such a way that a majority of convolutions can differentiate the labels | To this end, this work encodes the label into an additional channel of identical size as the images, this encoding is achieved through a learned matrix in the case of OLU and a random one without. In the case of C1F,R-10, the one-hot vector is multiplied by a 10 x 1024 embedding matrix (E), and this result is transformed into 32 32 matrix and concatenated fourth input channel (shown below). In practice, we found this to work well.
g,00 [x,, reshapo 32,32) 32,
9, [a,, reshapo Eū,, 32, 32)]
Where [| defines channel-wise concatenation, yı is the label corresponding to 2; and it is a wrong label.
5.2 Architecture
Selecting the appropriate architecture is a crucial consideration in deep learning. However, most design principles and architectural rules-of-thune thumb stem from backpropagation and are not blindly applicable to unexplored learning algorithms. For instance, in local learning setting, bottleneck layers most often result in information loss rather than compact representations in our experience. Currently, the Forward-Forwaad algorithm has only been used to train small conlecco networks and hon-weight-shaiing local receptive fields. In line with other work within local learning (No kland 2016b Belilovsky al., 2019), this work employs VGG-like CNNs (Simonyan & Zisserman 2015). However, some specific changes are made to adapt it to Forward-Forward. First, we omit the (linear) classification head because goodness is simply determined from aggregation of activations, operations within each layer. Specifically, we first perform normalization, followed by the convolution, the non-linearity, and finally an optional maxpool. This is simply an extension Hinton 2002 and permutations of this have not been explored further in this work g This work studies three architectures: two CNNs consisting of and 12 layers and an FCN with 6 layers. These networks respectively consist of 2.8 million, 8.5 million, and 25.1 million parameters (depending on the input). The CNN architectures are depicted in Figure 12
matter the dimensionality [ Second, use different ordering of
The FCN is simply stack of hidden layers with dimension of 2048.
5.3 Evaluation Metrics
The Forward-Forward algorithm allows for more fine-grained metrics to evaluate any given network compared to backpropagation. In essence, each layer has the same objective, in contrast to backpropagation where only the last layer has an objective, which can be used to examine the relations between layers. For instance, it is possible to inspect how well a single layer is able to classify based solely on its own goodness. We expect this accuracy to increase with depth, and failure to do so may indicate an issue with the previous layers. The same principle can be applied to any per-layer metric, such as the separation, loss, or gradient stability to gain insight into the behavior of the model. An important metric in evaluating learning algorithms is training accuracy. This indicates whether the algorithm overfitting or simply fails represent the dataset. However, in the supervised setting, this is impossible to determine efficiently without sampling all labels, which would slow down training time by the number of labels.
*Consider the case where the first ten pixels of the image used to encode the label. The majority of output neurons have information about the label, leaving them incapable of scparating positive from negative. PPrliiinary experiments
this work have shown that the goodness mechanism, in general, does not operate well when using very few features, such as classification head. For simplicity, we do apply any techniques such as the neer-normalization proposed in Hinton 2022 all networks.
8

---

Published in Transactions on Machine Learning Research (10/2024)
5.4 Accuracy
To measure the actual accuracy of the network, Hinton (022) reports that taking the average goodness of all layers except the first leads to the highest accuracy. Our experiments show this to be true in tiny networks trained with the original algorithm but when using The Trifecta on deeper networks, accuracy drastically degrades. This is because, on complex datasets, the carlier layers are significantly worse than the later ones and affect the stability of the ensemble. Therefore, all reported single-valued accuracies (Table (Table 2) are simply based on the goodness of the last layer. Nevertheless, in deep networks, aggregating the last few layers can be fruitful. By aggregating the last layers for goodness-based based classification, we can improve test accuracy by few decimal points. More information about evaluation can be found in Appendix H
6 Results
We perform and study two series of experiments evaluate the effectiveness of The Trifecta. The first is ablation of the three components of The Trifecta. The second is study into the impact of the architecture and training time. Where possible, we compare results to the vanilla (original) Forward algorithm (Hinton 2022), feedback alignment (Lillicrap et al., 2016), PEPITA (Dellaferrera & Kreiman 2022), and a backpropagation baseline. This baseline is trained on the deep architecture with an altered layer structure to accommodate backpropagation [ Further, a linear classification head is appended. Note that this simply serves as a baseline and far from SOTA. The same hyperparameters and architecture are used on all datasets for fair comparison. We describe cach model with the shorthand notation [algorithm]] (aarchhmmmaarhhhttteee //architocture] The considered algorithums consist of Vanilla FF (VFF), Trifecta FF (TFF), feedback alignment (FA), PEPITA (PEP), and backpropagation (BP). The architectures are the shallow CNN (s), the deep CNN (d), the FCN (f), and finally a tiny network consisting of 3 or fewer layers (t). For instance, VFF/s denotes the original FF algorithm on the shallow architecture.
6.1 Ablation of Trifecta Components
Table shows the accuracy achieved by multiple interplays of The Trifecta components. We perform this ablation with the shallow architecture on the CIFAR-10 dataset as it is the most challenging and, therefore, differentiates the algorithms best. The first row illustrates the importance of the loss function in our recre- ation of the original FF algorithm. This highlights the importance of as hyperparameter. The value T is chosen in line with other implementations (Lee & Song 2023 Pezeshki 2023) and T 10 is an optimized value we found for network on CIFAR-10. Additionally, utilizing the improved loss function further improves the accuracy and establishes this component the most crucial part of the Trifecta. The next rows show that the addition of each component yields significant gains to the final accuracy. The full Trifecta (the bottom right entry) achieves the highest accuracy by a significant. margin. The last column shows that the combination of all techniques outperforms the sum of parts, the differences compared to only SymBa are 6.60% for OLU, 9.29% for BN, and their combination is 16.01%.
6.2 Convergence Properties
We train the TFF/s and the TFF/4 /d architectures for 200 and 500 iterations respectively on four image classification datasets (Table 2). We compare our results at several stages of training to the available results from Hinton (2022) and a backpropagation bascline. When using backpropagation to fit trivial datasots, such as MNIST, unnecessarily deep networks are inadvisable as they lead to slower convergence and instability. Our experiments confirm this observation the deep model (99.58%) only slightly outperforms the shallow model (99.53%). A similar trend can be observed for the f-NNIST and SVHN datasets. On the other hand, the more challenging CIFAR-10 dataset exhibits the same behavior up to 100 epochs, where the shallow model (75.23%) convincingly outperforms the deep model (71.12%). However, the longer training time allows the deep model (83.51%) to surpass the shallow model (80.01%). A more detailed explanation of this phenomenon can be found in Appendix G Lastly, the learning speed of our algorithun very competilive,
"Backpoppgattin,, trained on the shallow architecture, shows slightly worse convergence properties and is therefore not shown. Further, on our setup, backpropagation converges after 100 epochs, showing further improvement longer training.
9

---

Published in Transactions on Machine Learning Research (10/2024)
which can be seen on the depicting accuracy after 10 epochs. These results highlight the potential of The Trifecta the Forward-Forward algorithm. Three slight changes have drastically improved the accuracies of the learning algorithm for all datasets: on simple datasets, our algorithm is only slightly behind our backpropagation baseline, and on more complex datascts, we obtain very competitive accuracics comparod other approaches.
Table 1: Test accuracy ablation with only specific components of The Trifecta using the shallow network CIFAR-10 after 100 epochs over runs. Referring to the original loss, the shorthand FTL (Fixed Threshold Loss) is used in this table.
FTL (T=2)
FTL (7=10)
SymBa (a=4)
None BN
29.27 2.49
44.28 + 1:28
59.22
0.49
41.02
37.31 :: 3.01
68.51
OLU
48.18
50.85 +: 0.63
65.82
OLU+BN
57.92 1.11
58.99 + 0.84
75.23 0.65
7 Discussion and Future Work
LOV13
LaVe 12jor
Lapert
100
200
380
430
500
opoch
Figure 3: The evolution of test accuracy (left) and scparation (right) throughout the training of the best- performing CIFAR-10 TFF/d /d model presented in Table 2
We can split the empirical impact of The Trifecta on the Forward-Forward algorithm into two categories. First, the progressive improvement is the amount by which each layer improves compared to its predecessor. Ideally, appending additional layers continues to improve accuracy. Second, the learning speed is the amount by which each layer improves over time. Trivially, the improvement cach layer must be as quick as possible reduce training time.
7.1 Progressive Improvement
The primary advantage of The Trifecta lies its ability to improve the accuracy upon going deeper into the network. In contrast to the original FF, which unable to learn further than few layers, especially when using CNNs (Fiure 4). In this work, we were able to scale to 12 layers on multiple datasets with considerable gains per layer. Above this threshold, our experiments did not show worthwhile improvement. Further, the progressive improvement of later layers is closely related to the learning rate. Figure 3 illustrates the accuracy and the scparation throughout training. These plots reveal the impact of a lower learning rate on the separation and stability of the later layers, as can be seen at epochs 200 and 400 when the learning rate schedule changes. This is also reflected in the accuracy (although difficult to see on the plot), as the difference between layers 6 and at epoch 200 2.6% and 3.7% at epoch 500. This work has only scratched the surface in terms of exploring the impact of learning rate schedules. More exotic learning schedules, which may involve freezing certain layers, may be highly fruitful to further improve accuracy.
10

---

Published in Transactions on Machine Learning Research (10/2024)
Table 2: Test accuracy comparison between learning algorithms on several datasets at certain epochs. We show the mean and standard deviation of 3 runs. Note that these are the accuracios for the last layer alone, achieve our best results by combining the last three layers outlined in Appendix H. The VFF/t tt and
PEP/ /t results are from Hinton 2022) and Dellaferrera & Kreiman (2022) respectively.
model
epochs
MNIST
F-MNIST
SVHN
CIFAR-10
CIFAR-100
FA/T
100
98.60 = 0.08
85.87 + 0.19 71.11 + 0.60
53.39 0.24
23.00 = 0.28
PEP/
100
98.29 =
56.33
1.35
27.56 =
VFF/t
60
98.6
VFF/t
500
99.4
59
TFP/s
10
97.44
0.49
76.81 + 2.37
84.62 + 1.34
50.32
0.32
11.32 1.57
TFF /s
100
99.23 0.11
88.00 1 0.67
92.57 1 0.67
75.23
0.65
28.71
0.42
TFF/s
200
99.53 0.04
91.38 + 0.38
94.25 + 0.54
80.01
0.00
35.79 = 0.17
TFF/d
100
99.32 = 0.11
85.62 + 0.53
91.49 + 0.34
71.12 = 0.77
24.42 = 0.55
TFF/d
200
99.28 = 0.03
87.90 + 0.12
92.60 = 0.27
78.99 = 1.16
28.88 = 0.80
TFF/d
500
99.58 # 0.06
91.44 + 0.49
94.31 = 0.07
83.51 = 0.78
35.26 = 0.23
BP/d
100
99.07 = 0.
93.61 + 0.11
94.01 + 0.20
89.32 +: 0.45
59.64 + 0.46
7.2 Learning Speed
Another strength of The Trifecta is its ability to learn more quickly. Each component in The Trifecta not only impacts the final accuracy of the model but also the speed at which this accuracy is achieved. All components contribute to this in their own way, leading to the strong synergy within The Trifecta. Specifically, our shallow CNN model able to cross the cape of 97% on MNIST and 50% on CIFAR-10 in less than 10 epochs. Again, the learning speed closely related to the learning rate of the layers. We are confident that finding tcchniques increase the lcarning rate, will further improve the speed of FF.
The focus of this work is to improve the accuracy of the FF algorithm in a supervised image classification setting using slight modifications. However, outside of The Trifecta and the original Newward-Fowwate paper (Hinton 2022), not much has yet been explored and several ideas for improving the algorithm still remain untried and await further most important for the future of FF: finding better evaluation strategies and improving general understanding of the learning characteristics.
We strongly believe that future work in the following two areas is
7.3 Evaluation Strategies.
Cltimately, FF is still held back due to several issues. In particular, performing non-binary classification using this algorithm is prohibitively slow as each class needs to be sampled separately to achieve high accuracy. Using FF for tasks with many classes, such as classifying ImageNet (Deng et al., 2009), is therefore strongly inadvisable. Nonetheless, alternate classification techniques that avoid sampling all classes separately by using candidate lists or one-pass classification (Hinton) 2022) seem promising but currently lead to lower accuracy and higher instability, respectively. Solving this would be huge boost toward the viability of Forward-Forwerr in more tasks.
7.4 General Understanding.
There are several gaps in knowledge about FF which we believe to be nontransferable from backpropagation. For instance, performing a thorough architecture search to determine which patterns and blocks work best. Bottleneck layers are not suitable for Forward-Forward but inverted bottlenecks (Sandler et al. 2018 may
be. Additionally, techniques such as residual connections (He et al., 2016) remain an intriguing avenue of
research that may push the algorithm to scale to increasingly complex tasks (touched upon in Appendix I.. Further, an investigation using well-established visual explanation techniques such as (Selvaraju et. al. 2019)
11

---

Published in Transactions on Machine Learning Research (10/2024)
is sure to reveal interesting properties about FF. Interestingly, these methods can be performed layer-wise to ascertain additional insight into the model behavior.
8 Closing Remarks
We proposed The Trifecta, three simple techniques that improve the accuracy and learning speed of FF- based methods. This enables the training of deeper FF-based archilectures that are competitive with their backpropagated counterparts. We note that, within our proposed improvements, there remain several open questions. We encourage further research to scrutinize our findings as well as explore how to further improve the FF algorithm. Contimued research into alternate learning algorithms may reveal generally applicable knowledge that will aid our understanding of machine learning and intelligence itself.
References
Atilmm Güneg Baydin, Barak A. Pearlmutter, Don Syme, Frank Wood, and Philip Torr. Gradients without backpropagation, 2022. URL https://axxxii wttp://g////...//////// 08587
Eugene Belilovsky, Michael Fickenberg, and Edouard Oyallon. Greedy layerwise learning scale to Im- ageNet. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 583-68 -933. PMLR, 09- 15 Jun 2019. URL https://prooceedingg/ mlr http.pess///////alloorrrysgaaaa
Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157 166, 1994. doi: 10.1109/72.279181.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Newral Information Processing Systems, volume 19. MIT Press, 2006. URL http:::/proocedings.eeeee neurips wwtpppps.ccppppppcpepeeppp fatpeerrffles/ppperrr
pdf
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprca, and Colin Raffel. Extracting training data from large language models, 2021. URL
http://aaab/22.7/00/0// 07805
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyaashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thamumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Eek, Jelf Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. In ArXiv, 2022.
Douglas
Giorgia Dellaferrera and Gabriel Kreiman. Error-driven input modulation: Solving the credit assignment problem without backward pass. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 4937-4955, PMLR, 17 23 Jul 2022. URL https://proceedding wwriiiinnnnnm.......mm 2 html
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10.100//CVPR:2205556665
12

---

Published in Transactions on Machine Learning Research (10/2024)
Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierro-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 11(19):625 660, 2010. URL http:///mt...ooppppppttt///aaaaa
Louis Fournier, Stéphane Rivaud, Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Can forward gradient match backpropagation?, 2023. URL
06968
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An S00gb dataset of diverse text for language modeling. 2020. URL
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics, 2010.
Aidan N. Gomez, Oscar Key, Kuba Perlin, Stephen Gou, Nick Frosst, Jeff Dean, and Yarin Gal. Interlocking backpropagation: Improving depthwise model-pparallelism. Journal of Machine Learning Research, 23 (171):1-28, 28, 2022. URL http:p/://hpll..orrr/ap/e///////
html
Stephen Grossberg. Competitive learning: From interactive activation to adaptive resonance. Cognétive Science, 11(1):23 63, 1987. ISSN 0364-0213. doi:
URL
http://www.ccienccirrcc.ccccc http::/www...ct.......c.aaaaaaaa
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In 2015 IEEE International Conference on Compuler Vision (ICCV), pp. 1026-1034, 2015. doi: 10.1109/ICCT 2035.123.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Tn 2016 TEFE Conference on Computer Vision and Patlern Recognition, (CVPR), pp. 770-776 2016. doi: N0.1109/CVPR.200
Geoffrey Hinton. The frward-forward algorithm: Some preliminary investigations. 2022. URL https:
7/arxiv. www.:oggg/////aaaae 13345
Geoffrey Hinton and Terrence Sejnowski. Optimal perceptual inference. In Proceedings of the TEEE Confer- ence on Computer Vision and Pattern Recognition, pp. 448 453, 01 1983.
Geoffrey E Hinton, Simon Osindero, and Yee Whyy Teh. A fast learning algorithm for deep belief nets. Neural Comput., 18(7):1527 1554, July 2006.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9:1735 80, 12 1997. doi: 10.1162/ncco http///cc..199..000 1997.9.8.1733
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lec. Jiquan Ngiam, Quoe V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In ArXiv, 2019.
Zhouyuan Huo, Bin Gu, qian Yang, and Heng Huang. Decoupled parallel backpropagation with convergence guarantee. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2098-2108010 20082 PMLR, 10 15 Jul 2018. URL httts::/
mlr www.pp/ee/e/oooessee0/ html
Sergey loffe and Christian Szegedy. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML''15, pp. 148-456, JML.R.org, 2015.
Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Doina Precup and Yee Whye Tch (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Reseorch, pp. 1627-1635. PMLR, 06-11 Aug 2017. URL https: mlr. /ttp:///...eeee/////aaarrrraaaaa html
13

---

Published in Transactions on Machine Learning Research (10/2024)
Alex Krizhevsky. Learning multiple layers of features from tiny images. pp. 32-33, 2009. URL https:
7/www. cs. toronto. edu/-kriz http:///ww//aaammaaaaaaaaaaeeeaa
Alex Krizhevsky, Πly Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger (eds.), Advonces in Newral Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL http://prooo// ott.gs....gg......o.ooooo///aaaa
.df
Michael Laskin, Luke Metz, Seth Nabarro, Mark Saroufim, Badreddine Noune, Carlo Luschi, Jascha Sohl- Dickstein, and Pieter Abbeel. Parallel training of deep networks with local updates. In ArXiv, 2021.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http://yann. com com/exdb/mniste
Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, R. Howard, Wayne Hubbard, and Lawrence
Jackel. Handwritten digit recognition with pack-propagation network. In D. Touretzky (ed.), Advances in 1989. URL wttpttps:/ppooeeaagae. htt.a//eee..aaaeeo.o//////ooooea
Newral Information Processing Systems, vohume 2.
Hleug-Chang Lee and Jeonggeun Song. Symba: Symmetric backpropagaation-fiee contrastive learning with wwtp:///wwrrabbb..o.oooo 08418.
Borward-forward algorithm for optimizing convergence. 2023. URL
Timothy P. Lillicrap, Daniel Cownden, Douglas B. Tweed, and Colin J. Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications, 7(1):13276, Nov 2016. doi: 10.1038 nc2mms13276. URL https: pettps:/ddoi.ooe org/10.
ISSN 2041-1723.
Tsung-Y: Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. 2018. URL https:
S. Lowe. Reimplementation of the forward-forward algorithm.
https://gittub..com//m
Forward 2023.
Sindy Löwe, Peter O' Connor, and Bastiaan Veeling. Putting an end to end-to-end: Gradient- isolated learning of representations. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL httpps://proceedingg 255300008822200040000546666666 Paper. pdf
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysci Bondarenko, Mart van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. In ArXiv, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. NTPS, 01 2011.
Arild No kland. Direct leedback alignment provides learning in deep neural networks. In D. Lee, M. Sugiyama, U. Luxburg, 1. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 29. Curran Associates, Inc., 2016a. URL htttps://proceeddggg wwwpppp.ccepapprrrrrro.. fiwes/ paper/ 2055/ffiee4000r44r00000000111112
Arild No kland. Direct feedback aligmment provides learning in deep neural networks. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol-
ume 29, Curran Associates, Inc., 2016b. URL httpp::/prooeeddggss http://rrrreddiig...eeiieurppppp wew.rrpps.ccppppr_fffeeeeeeee files/paperrr
.pff
Alexander Ororbia and Ankur Mali, The predictive forward-forward algorithm, 2023. URL https://arxiy org/abs/2301.014
M Pezeshki. gytorch-forward-forr 2023.
14

---

Published in Transactions on Machine Learning Research (10/2024)
PyTTrch Contributors. Pytorch layernorm documentation. http:////porrrr.oooooooo//////// torch. mn. LayerNorr..ttam 2023.
David E. Rumelhart and James L. McClelland. Learning Internal Representations by Error Propagation, pp. 318 362. 1987.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginox, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4510 4520, 2018. doi: 10.1109/CVPR.20 2018.00474.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kun- durthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models. In ArXiv, 2022.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual explanations from deep networks via gradient-based localization. Inter- national Journal of Computer Vision 3233 359, oct 2019. doi: 000007/11123-01--000 https://doii 10. 1ttp://.....001111122--0010--000
URL
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. In ArXio, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ArXiv, 2015.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting, Journal of Machine Learning Research, 15(56): 2014. URL http:///jilr.oog/pppprrsss// http://ppper//l///iiiaaaaaaaaa// html
C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Ra- binovich. Going deeper with convolutions. In 2015 JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9, Los Alamitos, CA, USA, jun 2015. IEEE Computer Society. doi: 10.1109/ 2VPP..015.779880
URL https:// doi www..icetctcttteerriiieey... wrw.iiett.oorog//// owg/10. 1109/CVPR CVPR 2015. 7298594
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. In ArXiv, 2017.
A Notation
Throughout the appendices, notational shorthands are used that will be outlined here. Additionally, for simplicity, we will be analyzing the case of single positive and negative example, instead of full batches.
First, we define a shorthand for the main component of the loss function, which called softplus.
{(x) softplus(x) = log(1 + exp(x))
Many functions can be used to calculate goodness, in line with the original work, we stick with the mean of squares, which can also be denoted as the square &g norm. positive and negative refer to the feature vectors from a ivsitive and negative sample respectively.
gos |lpositivell ge ||negativel
Often, we wish to examine the separation to the threshold, both for positive and negative examples.
800s_
pos
gneg
n5g
15

---

Published in Transactions on Machine Learning Research (10/2024)
In the same spirit, the overall separation between positive and negative is also usefui.
8 = 800s + gneg = goos - gneg
The following is a formulaic summary of the three main components of the paper.
SymBA: Lsn = soflphus(a-0) where a is a hyperparameter that defines how strongly the model is punished for wrong predictions.
OLU: (layer mod 2 = 0) @ (iteration mod 2 = 0) where @ refers a xor operation. This determines when the layer is updated in the update process seen in Figure 11
BN: - - -(ee"B//aaaa - 3 where the mean and variance taken over the batch dimension.
B Hyperparameter Discussion.
Similar backpropagation, training deeper networks requires more epochs. Specifically, we opted for 200 epochs for the shallow network and 500 for the deep network. Further, in line with Pezeshki (2023) ; Lowe (2023), we found the Forward-Forward algorithm is quite sensitive to the learning rate. Specifically, higher learning rates tend to lead to instability, and lower learning rates simply lead to slow convergence. We have found 10-3 to be a sensible default. For optimal results, however, we found a learning rate schedule to be highly beneficial. In our experiments, we train the deep network with a learning rate of 10-3 for the first 200 epochs, then 10-4 for the next 200 epochs and 10-4 for the last 100. The shallow network uses 10-3 for the first 150 epochs and 10 during the last 50. In terms of batch size, we found 1024 (512 positive and 512 negative) to work well but changing this slightly does not seem to noticeably influence accuracy. Further, both discussed loss functions have hyperparameter. For the original loss function, mention the value of F in each experiment explicitly. The SymBa loss has scaling parameter that penalizes incorrect classification, we use a scale of 4 as recommended by Lee & Song (2023). We find that values in the range [2; 8] for this parameter do not impact accuracy, only the separation. In terms of data augmentation, a random rotation and random crop are performed. Lastly, all networks are initialized using uniform Kaiming initialization He et al. (2015).
C Issues With the Original Loss Function
In this section of the appendix, following the previously introduced notation, we more formally indicate the issues with the original loss function. For convenience, the original loss function repeated.
Lep(g (ee A E(T - gros) + -egne9
(1)
This study inspects whether the loss function handles positive and negative samples equally. As stated earlier, in reality, the loss function is computed for a batch of positive and negative samples (shown in section 3 but will use goes and ge as the goodness of single positive and negative sample respectively.
1. Symmetry: False positives and false negatives have an equal loss.
LFF(T + 8000, + gnen) 2 LFF 6200 5
(2)
Intuitively, the left-hand side corresponds to the loss of correctly classifying a positive sample and wrongly classifying the nogative one (false positive). The right-hand side represents the inverse, where the positive sample misclassified (false negative). As the respective errors are the same, we expect. the loss function be the same. When filling in Equation 2 with Equation I we attain the following.
E(T (F + )200)) + E((T 5ne9) 7) E(T (T -8P0))) E((T gDEg) +))
(3)
16

---

Published in Transactions on Machine Learning Research (10/2024)
{(-8w0e) + E(8 {(8P0N) + E(-8ne))
(4)
We can see that this is the case iff 8res = gneg, i.e. when positive and negative samples are equally separated from the threshold.
2. Equal Gradients:
Negative and positive samples are equally pushed away from the threshold.
① LFF(T + 8008, , gneg) Oopos LFF(T - 8p00, 8"e
(5)
Zgneg
In essence, the first part computes what the partial derivative of the negative goodness is upon false positive. The second computes the same but for the positive goodness upon false negative. If any one of these is misclassified, derivatives of equal value are desired. To verify whether this holds for the loss function, the partial derivatives need to be computed first.
i Dōneg E((T + tneg) -) 08pos
{(T (T -)200))
(6)
θ 08pos {Bpoo))
08neg {(one9)
(7)
exp(8"eg) exp( (mey) +1
exp( (8p0s)
(8)
earpl (pos)
+1
Again, this is the case iff 8ras = gneg, when positive and negative samples are equally separated from the threshold.
Consequences. The loss function only satisfies our two criteria given a very specific condition: when 8res gueg Intuitively, one would expect this hold. However, goss increases, so does, exp(8""*)/1 + cxp(6""r) and similarly for 8"eg especially for small 8, which occurs most of the time. Therefore, if the separation of either positive negative increases, that derivative will start to dominate the gradient, leading to unstable equilibriun.
Such imbalance can occur simply due to the variance in batches or due to it being simpler to generate high or low goodness for the optimizer. Another factor that can cause this imbalance that goodness must be positive. This sets a hard limit for the separation of negative samples, namely gneg S т. In reality, this limit is never even reached due to the optimization process. Achieving low goodness values necessitates delicate equilibrium, rendering the network highly responsive to any minor alteration. Since the loss function's constituent terms aggregated, the gradients stemming from positive samples severely disturb this balance. Conversely, this is not an issue for the positive samples small disturbances from the negative gradients will not impact them as much. Scaling the threshold to be high enough to circumvent this behavior is not possible without repercussions; a high threshold will induce lots of instability in the network as all weights need to be higher, in some cases this may even lead to the inability to converge at all.
This issue exacerbated in weight-sharing architectures such as CNNs, given that it more difficult control the exact output of multiple neurons given single weight change. This can be observed in the following experiment where we simply employ the shallow CNN in combination with unaltered version of FF (VFF/s). Figure 4 clearly shows that the accuracy declines upon going deeper into the network, which is a large issue in many cases.
Finally, due to the abovementioned issues, finding a correct value for 7 is complicated. In limited experiments, we found the value that achieves the highest accuracy to be highly dependent on the architecture and dataset. However, all statements made or observed trends for given (such as Figure 4h are verified hold within the range [1:10].
17

---

Published in Transactions on Machine Learning Research (10/2024)
Layar
<poch
20
60
Figure 4: Layerwise accuracy of the best-performing VFF/s using the original loss function (r 2) Table 1
The experiment is performed CIFAR-10.
D Exploring the New Loss Function
This section of the appendix discusses the properties of the new loss function and their impact on the accuracy and stability of Forward-Forwee We conclude by outlining some alternatives and extensions to our approach. For convenience, the new loss function is repeated.
LSB(g geg) & Elgpoe ge
(9)
This function suffers from flaw that makes it possible to lower the loss function without learning new features. As the loss is simply calculated by the separation, scaling all parameters uniformly (given an already cxisting separation) will lower the loss, not by learning but by simply amplifying the features. Mathematically, we can note the following.
8 R=ReUU(((1,O:)) - R (( 0,))
(10)
This simply denotes that if the parameters of a layer 0, are able to achieve separation ö, an arbitrary separation can be achieved by choosing accordingly. Notice this only holds, if F is linear operation, which is the case for a convolutional or a dense layer. Moreover, ReLU only disregards nogative features which has no impact on scaling the positive features (like sigmoid would).
In theory, this uncontrolled scaling can have devastating effects on the stability of network. However, empirically, we find that this behavior is negligible in all experiments performed. We verify this using the three experiments listed below. To ensure the conjectured behavior has the highest chance to be observed, we perform these experiments on the model that achieves the highest accuracy on CIFAR-10, shown in Figure 3. This model has the most layers and is trained the longest.
Plot the Mean of Positive and Negative Goodness Throughout Training for Each Layer This experiment provides a comprchensive view of the evolution of the goodness which can be usod to indiroctly assess the stability of specific layers through training. Figure 5 shows that some layers have significantly higher goodness than others, especially the second layer. Regardless, knowing that this is the squared sum of thousands of features, all values remain within the expected range for neural networks and do not demonstrate this uncontrolled scaling.
Measure the Maximal Weight for Each Layer After Training. The magnitude of weights is only an indirect indication of stability or other common metrics. Nevertheless, low maximal weight ensures that the network suffers less from phenomena such as overfitting or instability. This experiment, depicted in Figure 6. reveals that the highest weight has value of 0.3 which occurs in the first layer. After the second
18

---

Published in Transactions on Machine Learning Research (10/2024)
layer, there is no layer with weight larger than 0.2. After more epochs, there is significant difference in maximal weight. This further refutes the notion that the network is simply scaling its weights to achieve higher separation.
Measure the Maximal Goodness Value at Test Time. The first experiment examined the mean values of goodness training time assess the stability of the network. However, simply examining the mean does not provide information about outliers, which may exclusively cause the issue. Comparably, examining training samples is inadequate the model may exclusively become unstable outside of the training distribution. Therefore, this experiment studies the 'worst case' of the model; the maximal goodness at test time. We measure a maximal goodness of about 160 in the first three layers, then regressing to a maximum of 90 in the later layers, depicted in Figure 6 This, again, is within the realm of reason. Generally, longer training times do not severely impact the maximal activations except for the last layer. This effect can be seen a lesser extent Figure 5 (green line). We suspect this phenomenon is specific to this exact architecture (as it was not observed clsewhere) and therefore do not draw conclusions about its origins. Lastly, the peak al the second layer seems to be simply incidental and does nol occur for different networks hotwooks/daaasen /daaasets.
Positive
Negative
layer
layer
1ayer
layer
layer
layer
layer
layer
layer
layer 12
layer
Layer
layer
Layer
Layer
12
g00dies
60
60
20
20
spoch
100
200
300
400
100
200
305
Figure 5: The evolution of positive (left) and negative (right) goodness during training for the best-performing CIFAR-10 TFF/d model presented in Table
Goodness
Weight
200
200
140
500
0.25
500
0.20
60
0.30
40
0.05
20
layer
layer
layer
layer
layer
layer
layer
layer
layer
layer
Figure 6: Layerwise maximal goodness (left) and maximal weight (right) at three different epochs for the best-performing CIFAR-10 TFF/d model presented in Table 2
19

---

Published in Transactions on Machine Learning Research (10/2024)
Consequences. We hypothesize that the explosion of weights practically does not occur due to variance in the network, known as Internal Covariate Shift (ICS) (Toffe & Szegedy 2015). In effect, it seems to be better to learn robust features instead of scaling highly specific ones to achieve the best separation. As the first two layers are barely subject to such variance, the optimization is more prone to increase their weights and therefore goodness. However, no further experimentation performed to fully back up this hypothesis.
The only adaptation this work has made to avoid issues related to high values is the method of calculating the loss itself. Namely, the term exp(g"0e - -g"me neeo) can lead to numeric instability, especially when working with half-width floating point values. Therefore, we approximate log(1 + ezplg - gooe - after a specific point
gnea)) with simply
Alternatives. A possible remedy is to find a loss function L that satisfies the condition Vo: L(pos, neg) = L(a pos, a neg). If the loss remained identical regardless of any a, there would no longer be an incentive to scale up all parameters. However, consider the case where a 0, this would imply Vz,y y L(0, 0) = L(x, y) which only a constant function satisfies. Consequently, is not feasible to achieve such behavior given the current approach.
Alternatively, a possible constraint can be put on the growth of weights by imposing weight decay. Limited experiments were performed with decay values ranging from 10-4 to 10-6 without success. In all cases, accuracy was significantly lower.
E Normalization Deep Dive
This appendix accompanies section 4 and provides additional plots and insights about the impact of nor- malization. Specifically, the statements made in that section will be substantiated further here.
In contrast to the majority of other experiments in this work, this section uses the FCN architecture consisting of 6 layers that all have hidden dimension of 2048. This choice is made to ensure that any trends throughout layers are the product of the learning algorithm and not the architecture (for example side effects of a maxpool). The observations made in this section are also verified to hold for CNNs, but not shown.
No Normalization Leads to Feature Recycling. We empirically demonstrate the feature recycling phenomenon as denoted in (Hinton) 2022). The recycling behavior stems from the reuse of features that already produce high/low goodness depending on the sample. Any subsequent layer can therefore simply perform an identity operation or even a permutation and produce low loss without learning anything new
The following plots Figgure 7) demonstrate that networks trained without normalization, have substantially lower gradients, especially after a few layers. Compared to the other normalization strategies, the right- hand side of Figure 7 shows that the gradients of the network without normalization are several orders of magnitude lower. Further, the left-hand side shows that the magnitude the gradients decreases throughout layers.
Layer Normalization Leads to Information Loss. Information loss is often quantified in terms of mutual information. Specifically, if an embedding has low mutual information with the input, it is said to have lost information. Informally, mutual information conveys how much information is shared between two distributions. Or alternatively, how much information does sampling one distribution reveal about the other and vice-versa. Consider a very narrow (bottleneck) layer that only has very few features, generally, it will be impossible to redetermine the original input exactly. However, this phenomenon occurs not only when reducing dimensionality but at many locations. For instance: a ReLU operation loses all negative information, and numeric approximation (which occurs almost every floating point computation) loses information too.
The mutual information between the input and the label is generally low, from the label alone, the exact input cannot be determined. Therefore, at some point, the network will lose information. It has been shown that modern networks retain most information until the classification head aggressively narrows dimensionality (Shwartz-Ziv & Tishby 2017). This allows later layers to manipulate data to become even more separable
8The Torch SoftPlus function has this functionality built-in by default with a threshold of 20.
20

---

Published in Transactions on Machine Learning Research (10/2024)
Gradient L2 (no norm)
Gradient L2 (layer 6)
layer
layer
Zayer
layer
Layer
loyer B
no nome
batch nore
Layer nore
23.8
15e0
26
4K 6k 86
8k.
Figure 7: Layerwise gradient & norms without normalization on the FCN network (left) and & norm of the gradients of layer 6 compared to other normalization schemes (right). Both plots are on CIFAR-10 with the SFFnSy model. (The right figure is the same as Figure supplemented with the no normalization model.)
while retaining holistic picture of the input. In ordinary neural networks, mutual information is not essential to consider. There is simply objective and the entire update is to better satisfy this objective. In this context, it is in the best interest of lower layers to relain information. In the case of FF however, each local objective has no incentive to retain information. Therefore, we need to ensure information loss is diminishod.
However, this paper does not focus on the mathematical properties and behavior bounds but rather takes an empirical approach. We measure the usefulness of representations by appending multi-layer classifier top of each layer separately (Figure S). The original network pre-trained with VyF++y and the classifier heads are trained with backpropagation for 100 epochs. Intuitively, architecture and learning method can use the produced features by each layer for classification. This reveals that normalization affects the downstream usefulness of representations quite strongly. Concretely, as depth increases, the usefulness of layer normalization features quickly decays while no normalization retains them best, batch normalization stays in the middle. This relates back to finding the balance between forcing the network learn new fcatures (layer normalization) and simply reusing features (no normalization).
measure how well-tested
Accuracy
layer norm batch norm
no norm
layer
layer
layer
Figure 8: Layerwise test accuracy auxiliary backpropagated classifier for layernorm, batchnorm, and no normalization. The experiment is performed on the VFF+SymBa/f model after 100 epochs on CIFAR-10.
21

---

Published in Transactions on Machine Learning Research (10/2024)
Batch Normalization Leads to Lower Variance. Figure 2 depicts the & norm of the gradients in each batch. The plots clearly indicate the increased variance of layer normalization in comparison to batch normalization. As additional evidence, it is possible to examine the weights of the models themselves. There are a multitude techniques to visualize the distribution of weights. We opt to mcasure the maximal weight al each layer and the maximal goodness for each layer (on the full test set), depicled respectively in the left and right plots of Figure 0 This reveals that the network with layer normalization has higher maximal weights at each layer. Additionally, while less noticeable, the maximal goodness values more constant throughout the batchnorm network.
weight
Max Goodness
10
normal ization
layer norm batch
layer norm
800
batch norm
norm
none
0.8
0.6
400
200
layer
layer
layer
layer
layer
Figure 9: Layerwise maximal weight (left) and maximal goodness (right) for layernorm, batchnorm, and no normalization. The metrics from the same run depicted in Figure
Maximal values are good for putting bounds on the variance of a network but give a narrow view of the distribution. Simply looking at the weights themselves FFguuee 10) gives qualitative insight into the differences of distribution: batchnorm has drastically less spread out weights.
Layernorm
Batchnorm
0.2
0.1
0.0
0.0
-0.3
-03
Figure 10: A visual representation the weights at layer of the VFF+SymBa/t model. The y-axis corresponds to the output neurons and the x-axis to the input neurons. Additionally, the values are clamped within [-0.3; 0.3] for visual clarity. The metries are from the same run as depicted Figure 8
22

---

Published in Transactions on Machine Learning Research (10/2024)
Single Sample Training. A notable advantage of layer normalization over batch normalization lies in the fact that there is no constraint on batch sizes. In a biological setting, which FF endeavors to resemble, learning by normalizing over a batch of training samples is implausible. However, comparable to evaluation using batch normalization, this can be solved by using rolling averages for the full training process. Some preliminary experiments have shown that this does not significantly degrade accuracy. Specifically, when using the same settings as in Table 1 with the full TFF, the resulting model achieves an accuracy 73.3%, only 2% lower than originally.
F Error Signals and Overlapping Updates
This appendix will provide overview of the role and implementation of Overlapping Local Updates (OLU) within FF. Prior to this, detailed explanation of OLU is given.
Details of OLU. Intuitively, OLU is a form of truncated backpropagation where two layers are updated at once. At each step in the learning process, instead of simply updating the current layer given the local objective function, the previous layer also updated. In the context FF, there are two possible implementations. Both approaches are depicted in Figure 11
Approach 1: Alternatingly update group of 2 layers each epoch (used in this work).
Approach 2: Update both the current and previous layer at each step.
In terms accuracy, both techniques are equal. The discrepancy lies in their runtime; we found the first approach be faster than the second on the sctup we used. Therefore, we opted for the first approach. However, expect, with some luning, both techniques can perform equally fast.
Approach
Approach 2
AuX
AuX
Aux
Aux
Aux
AuX
Aux
Aux
Even Iterations
Aux
Aux
Uneven Iterations
Figure 11: A visual representation of two possible OLU implementations. The red lines represent error signals. At even iterations, even objectives are optimized and at uneven iterations, uneven objectives are optimized.
Alternatives to OLU. In the case of this work, OLU was inspired by the concept that neurons not only send signals to other neurons within their layer but also to neighboring neurons, such as the previous layer. OLU takes the simplest approach from a contemporary research standpoint and allows a full error signal be locally propagatod. However, more biologically plausible alternatives exist, in which random synaptic feedback can be used locally update neurons (Lillicrap al. 2016). We encourage future work in this area to explore this technique.
23

---

Published in Transactions on Machine Learning Research (10/2024)
G Architecture Discussion
The main role of this appendix is to depict and describe the two main architectures used throughout this work. However, before that, a collection of empirical observations will be provided that led us toward this design. These observations will be contrasted with backpropagation and outline the differences and similarities to this algorithm.
During the experimentation phase of this work, the viability of training several architectures with FF was tested. This manual search focussed simple CNNs, meaning only combining convolutions and maxpool operations. This exploration entails more 'hyperparametens' than might be expected. For instance, techni- cally FF is proposed a layer-wise learning algorithm. However, it can be used as general algorithm to train entire 'blocks', such as (He et al., 2016 Szegedy et al.) 2013), or simply a subnetwork. However, this work limits itself to layerwise training and defers this avenuc of research to future work.
When training simple CNNs, two parameters of the network are of importance: the width and the depth. Varying these parameters directly influences the learning speed (time to reach a threshold in accuracy) as well as the final accuracy (accuracy convergence after long training). The learning speed difference between shallow and deep networks is a well-known phenomenon in backpropagation (especially in non-residual architectures). This stems from the increasing instability of gradients (He et al. 2016). In local learning, as no global gradients are used, this issue does not arise.
The accuracy of the local algorithms mostly determined by the width (the number of features or channels) of the layers. There exist several architectures that intentionally roduce the number of features of certain layers, such as autoencoders [Rumelhart & McClelland] 1987). These networks can be trained with backpropagation which will result in these narrow layers containing compressed representation of the imput. However, in local learning regimes, this bottleneck simply leads to information loss as the local objective struggles to retain useful downstream information. Conversely, scaling the width of a layer severely impacts the lcarning speed, intuitively, more features lead to a more challenging optimization process as local minima are harder to find. Therefore, the dimensionality of the features after each block of paramount importance to the final accuracy.
When using local learning CNNs, this balance in work width can be balanced through lossy operations such maxpool operations. Therefore, their locations severely impact the learning characteristies of the network. This disparity causes a large difference between the shallow and deep networks used in this work. As can be seen on Figure 12 the shallow network is not simply a subset of the deep network. Empirically, as discussed in section 6 we confirm the expected tradeoff between learning speed and final accuracy between these architectures.
As explained in |section 3 each layer of our architecture deviates from the ordinary order of operations. Particularly, each layer first normalizes, followed by the convolution and the non-lincarity. Lastly, an optional max pool is performed before the calculation of the goodness, which is explicitly shown in Figure 12] This figure shows the progression of channels through the architecture, where C is the number of input channels.
H Evaluation Strategies
As originally discussod in scction 3, there are several mothods to evaluate with a network trainod in a supervised FF setting. This appendix explores some such methods and compares their effect on the final accuracy.
Goodness-Base Classification. This evaluation method stands out as the prominent classification tech- nique to achieve the highest possible accuracy with FF (Hinton 2022). This stems from the fact that this technique leverages the exact purpose the network was trained for It evaluates the unseen example with all possible labels in the datasct and selects the label with the highest goodness, as this is the most positive. The main disadvantage, however, is that it requires an evaluation of the network for each possible label, which is prohibitively expensive in most cases.
24

---

Published in Transactions on Machine Learning Research (10/2024)
512 512
2x2 Maxpool
Maxpool
256 512
256 256
256-- 256
2x2 Maxpool
2x2 Maxpool
128 256
128 256
128 128
x
128 - 128
2x2 Maxpool
2x2 Maxpool
64 128
C 128
C - 64
Input
Input
Figure 12: The two CNN architectures used throughout this work. The shallow network (left) has 2.8 million parameters and the deep network (right) has 8.5 million parameters.
Feature-Basoo Classification. This method does not nccessitate sampling each class individually. Instead, it utilizes a neutral label, which is the average of all class labels, to generate features that represent all classes. These features are then used by an auxiliary network to make a single prediction encompassing all classes. In line with (Hinton 2022), we find that this method generally suffers from instability and therefore does not achieve the accuracy of the aforementioned technique. We have performed several experiments to remody this without success: freeze the whole network except the auxiliary head, normal negative labels instead of neutral ones, use very low learning rates, and use classifier consisting of multiple layers.
Ensemble Classification. The fact that the objective of each layer is the same can be leveraged in both aforementioned techniques. Unlike ordinary backpropagated networks, which only use the last layer to generate a prediction, any layer or combination of layers of the network can be used for evaluation. essence, the goodness or features from multiple layers can aggregated into an ensemble that more resilient noise and generally achieves higher accuracy. This aggregation can be a straightforward average even small classifier of any kind. To predict, the highest value of the mean goodness is used, instead of single goodness value, which results in more stability. This is depicted in Figure 13 In this work, use the average of the goodness to make the prediction. With this technique, we are able to augment the test accuracy of all our deep models by few decimals, shown in Table 3 The shallow models do not benefit from averaging the last fow layers as the accuracy of the last layer is significantly higher than its predecessors.
Residuals
Residual connections He al. (2016) have proven vital train deeper networks with backpropagation. Therefore, utilizing them to improve FF seems like clear path to improve the scaling behavior. The
25

---

Published in Transactions on Machine Learning Research (10/2024)
Table 3: Comparison of two proposed evaluation strategies on the accuracy, The shown models are the the FFF/d models at 500 epochs from Table 2
evaluation only last layer
MNIST
F-MNIST
SVHN
CIFAR-10
99.58 + 0.06
91.38 = 0.38
94.31 + 0.07
83.51 + 0.78
three last. layers
99.62 0.02
91.51 1 0.27
94.36 1 0.04
83.72 1: 0.80
Loss
Loss
Feature-Based Classification
Classification
Layer
L2 norm
Layer
L2 norm
Head
Layer
L2 norm
Layer
Lg
Head
Layer
Lg
Layer
Neutral
Input
Label
Input
N
Figure 13: A depiction of the two prominent evaluation techniques for Forward-Forward. The black arrow represents data flow and the red arrows represent gradient flow. Goodness- based classification (right) requires the network to be evaluated N times according to the amount of label. The 'head' in this figure can range from a simple average to a small classifier.
motivation between residual connections is twofold: it provides sensible (indirect) initialization for each layer, namely the identity matrix, and allows the backward gradients to bypass each layer. This results in a smoother convergence, eSpecially for deeper networks.
Within FF, both motivations no longer hold. First, adding the identity to each layer is not desirable as this may discourage the network from learning new features. Second, there is no need for backward gradients to bypass layers, there is full backpropagation. Additionally, the models used within this work are too shallow enough to fully benefit from residuals, even if they were trained with backprop. Interostingly, preliminary experiments using TFF/s and TFF/d CIFAR-10 with residual connections paint more neutral picture. Specifically, early during training (the first 30 epochs), the use of residual connection yields improved results. However, after 100 epochs the accuracy of the small and deep models are respectively 4% and 2% lower. Therefore, we believe residual connections to hold some promise, if modified in a sensible manner to the requirements FF.
26


Reviews:
[{"review_id": "3BrlfXtOXR", "reviewer": "Reviewer_YJVK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The tackled problem is hard, important, and interesting, especially with potential application to large language models, for which one cannot resort to vanilla backpropagation. To the best of my knowledge, making this type of backpropagation-free algorithm work in practice can be very challenging, and this paper provides tricks to obtain performances significantly better than previous backpropagation-free algorithms.\n\n\n\nWeaknesses:\nIntro: - To the best of my knowledge, backpropagation is a way of computing gradients, using the chain rule and computing the derivatives in an efficient reverse order. IMO it is weird to say ´backpropagation is overfitting´ since it is a way to compute the gradient, the exact gradient computed with the forward mode would yield the exact same result as backpropagation.\n\n\nBackground:\n- IMO more details are required when introducing the Forward Forward algorithm, maybe recall the algorithm?\n- More generally I can understand the ideas of the paper, which are very well explained with words, but it feels like the paper would benefit from more concrete ¨maths¨, e.g., more formal equations and algorithms. I would love to see the exact updates of the algorithm and a precise comparison of the updates.\n\n\nLiterature Review: Could the authors contextualize their work compared to [1], where good performances on CIFAR are reported with local updates?\n\n[1] Fournier, Louis, et al. \"Can forward gradient match backpropagation?.\" International Conference on Machine Learning. PMLR, 2023.\n\n\n\n\nMinor:\n- Harmonize bibliography style: chose FirstName LastName or F. LastName, e.g. Y. Bengio should be Yoshua Bengio in Appendix \n- ~~I would recommend using matplotlib for the figures instead of wandb output. Very very minor: in particular I would recommend using the viridis colormap for readability of the indices of the layers~~", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The goal of the paper is to study alternatives to backpropagation for training neural networks. First, the paper recalls a recent algorithm, the Forward Forward algorithm (Hinton 2022), and highlights three of its issues: the loss function, the normalization, and the potential lack of signal. The authors propose to fix each of these 3 issues, hence the ´trifecta´ coining.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- clarification on  ´backpropagation is overfitting´\n- more detail on what is done in the Forward Forward algorithm, and the proposed algorithm: add precise equations and algorithms\n- comments on the difference with the paper \"Can forward gradient match backpropagation?.\"", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vYiTxJ1okQ", "reviewer": "Reviewer_R5wW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- Discovering the positive synergy between these three components can give insights into how we should combine different techniques used in the FF algo.\n- The paper provides an thorough related works section and does a good job of placing it within the larger body of literature.\n- The results are quite impressive for datasets such as CIFAR-10.\n\n### Weaknesses and Requested Changes\n\n- I wouldn't really consider SymBa a contribution of this paper. I suggest removing SymBa as part of the Trifecta as it was already proposed by Lee & Song (2023).\n- Many details about how Batch Normalization and Overlapping Local Updates helps are left to the Appendix. These explanations should be in the main body if they're part of the core contributions.\n- The plots in Figure 2 would benefit from some smoothing of the curves so that we can see the general trend.\n- I am not totally familiar with the literature on local-learning but I am sure you could train the network to have high/low goodness at each layer by using backpropagation as well. Say by setting the loss as the average of the separation gap across all layers? How would this compare in terms of test accuracy to the current Forward-Forward implementations?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper highlights three weaknesses of the original Forward-Forward (FF) algorithm. These include, the loss function, the normalization function and the lack of error signal. The paper then proposes three simple techniques that improves the performance of the FF algorithm on deeper neural networks. They show that their techniques allow FF to reach training speeds and test accuracy that is on par with traditional backpropogation on simple datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested changes listed above.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ko2aKpEE8X", "reviewer": "Reviewer_j1gm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n\n- The problem under study is of significant importance given the rapidly increasing scale of neural networks and the demand for memory-efficient alternatives to the backpropagation algorithm.\n\n- The paper is clearly written and well-structured, providing both theoretical intuition and experimental evidence to support the importance of each component of Trifecta.\n\n- The proposed Trifecta methodology significantly advances the performance of the forward-forward learning algorithm compared to prior work on several datasets. Trifecta not only improves performance but also accelerates convergence.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work examines the issues of the recently proposed Forward-Forward learning algorithm and suggests certain modifications to the original training procedure, including the choice of the loss function, feature normalization, and update strategy. These modifications significantly enhance the performance of the algorithm. The proposed Trifecta prescription is validated on several image classification datasets and CNN/MLP architectures.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- In my opinion, the background on the forward-forward algorithm should be slightly expanded for readers not familiar with the topic. While the loss function is mentioned, the exact update rule (optimizer step) is not provided. I suggest adding this for the sake of completeness.\n\n- Whereas the extension of the approach to residual networks is left for future work, is there a significant challenge with the application of forward-forward to small residual networks? There are small 9-layer ResNets [1] that can be rapidly trained on the CIFAR-10 dataset to achieve pretty good accuracy. It would be interesting to examine how the FF algorithm works with residual connections.\n\n- [Minor] I think it would be better to have a sequential color scheme for the plots in Figures 1 and 3, such that a darker/brighter tone (or any other rule) indicates deeper layers, making them easier to discern.\n\n[1] https://github.com/apple/ml-cifar-10-faster", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the claims are mostly well-supported and there is strong empirical evidence for their proposed method.

However, 2/3 of the reviewers feel that the forward-forward (FF) algorithm, on which the current submission builds heavily on, is not properly described and lacks clarity. The authors did provide a little more background but I agree that the FF algorithm needs much more detail than what is provided. Currently there is less than a page devoted to this algorithm, which feels inadequate given the centrality to understanding, appreciating, and reproducing this work. If the authors are concerned with space, at a minimum I would suggest adding a lot more detail to the appendix.

Given that all reviewers would be supportive of an acceptance as long as more details on FF are provided, I am recommending an acceptance with minor revisions, where it should be made clear to the authors that I will only accept the camera-ready version as long as substantially more details are provided for FF.
Decision: Accept (poster)