Learning Using a Single Forward Pass
Authors: Aditya Somasundaram
Pushkal Mishra
Ayon Borthakur
Venue: Accepted by TMLR ()

Abstract: We propose a learning algorithm to overcome the limitations of traditional backpropagation in resource-constrained environments: Solo Pass Embedded Learning Algorithm (SPELA). SPELA operates with local loss functions to update weights, significantly saving on resources
allocated to the propagation of gradients and storing computational graphs while being sufficiently accurate. Consequently, SPELA can closely match backpropagation using less memory. Moreover, SPELA can effectively fine-tune pre-trained image recognition models
for new tasks. Further, SPELA is extended with significant modifications to train CNN networks, which we evaluate on CIFAR-10, CIFAR-100, and SVHN 10 datasets, showing equivalent performance compared to backpropagation. Our results indicate that SPELA, with its features such as local learning and early exit, is a potential candidate for learning in resource-constrained edge AI applications.

Published in Transactions on Machine Learning Research (06/2025)
Learning Using a Single Forward Pass
Aditya Somasundaram Columbia University
as as7458@columbiaa.c edu
Pushkal Mishra'
pumishra edu
University of California San Diego
Ayon Borthakur IIT Guwahati
ayon, wwwtboorttaa@iiii.. oc.in
Reviewed on OpenReview: https: // openreview. net/ forum? id= EDQ8Q0Gqjr
Abstract
We propose learning algorithm to overcome the limitations of traditional backpropagation in cooourceconntrained environments: Solo Pass Embedded Learning Algorithm (SPELA). SPELA operates with local loss functions to update weights, significantly saving on resources allocated to the propagation gradients and storing computational graphs while being sufficiently accurate. Consequently, SPELA can closely match backpropagation using less memory. Moreover, SPELA effectively fine-tune pre-trained image recognition models for tasks. Further, SPELA is extended with significant modifications to train CNN nelworks, which evaluate on CIFAR-10, CIFAR-100, and SVHN 10 datasets, showing equivalent performance compared to backpropagation. Our results indicate that SPELA, with its features such as local learning and carly exit, is a potential candidate for learning resource-constrained edge Al applications.
1 Introduction
Backpropagation (BP) is a long-standing, fundamental algorithm for training deep neural networks (NNs) (Werbos) 1990 Rumelhart et al., 1986; LeCun et al. 1989 2015). It is widely used in training multi- layer perceptrons (MLP), convolutional neural networks (CNN), recurrent neural networks (RNN), and now transformers. It loss minimization problem involving thousands, not millions, of parameters. Data is first propagated through the network using forward passes, and the entire computational graph is stored. The difference (error) between the final layer output and the label is utilized to update the weight matrices of the network. The error is then propagated backward from the final layer using the chain rule of differentiation, which uses the stored computational graph to compute the associated gradients for each layer.
Careful observations of backpropagation drive home the point of no free lunch. Backpropagation works within a global learning framework, i.e., a single weight update requires knowledge of the gradient of every parameter gglobllnnon-lon learning problem) (Whittington Bogacz 2019). Backpropagation requires the storage of neural activations computed in the forward pass for use in the subsequent backward pass (weight transport problem) (Lillicrap et al. 2014 Akrout et al. 2019). For every forward pass, the backward pass is computed with the forward-pass pass updates frozen, which prevents online utilization of inputs (update locking problem) (Czarnecki et al. 2017 Jaderberg et al.) 20177. Furthermore, there are several differences between backpropagation and biological learning, as mentioned in Scction 2
The constraints of backpropagation lead significant. challenges, such as high memory requirements. These limitations render backpropagation computationally expensive and unsuitable for applications in resource- constrained scenarios (example: on-device machine learning (ODL) Cai et al. 2020 Zhu et. al. 2023). Efforts to mitigate these algorithmic constraints of backpropagation
lead to improved learning efficiency.
PM: joint first authors (order by toss); AB: PT. Work partly done IIT Hyderabad.

---

Published in Transactions on Machine Learning Research (06/2025)
We introduce and investigate a multi-layer neural network training algorithm - SPELA (Solo Pass Em- bedded Learning Algorithm) that uses embedded vectors as priors to preserve data structure as it passes through the network. Previous studies indicate that prior knowledge helps one learn faster and more easily (Goyal & Bengio 2022 Wang & Wu 2023). Although we do not claim complete biological plausibility, SPELA is built. on the premise that biological neural networks utilize local learning Miing el al. 2021 and neural priors to form representations. We introduce neural priors as symmetric vectors distributed on a high-dimensional sphere whose dimension equals the size of the corresponding neuron layer. Our back- propagation-fee learning algorithm demonstrates significant gain in computational efficiency, making it suitable for on-device learning (ODL) and brain-inspired learning features, such as early exit. (for layered cognitive reasoning (Scardapane et al. 2020) and local learning. In this paper, we make the following contributions:
We introduce SPELA. Next, we extend SPELA to convolutional neural networks. SPELA is a family of algorithms that use a single forward pass (with no backward pass) for training. During inference, output from any layer can be utilized for prediction. It makes an innovative use of embedded vectors neural priors for efficient learning.
Design:
Evaluate: Experiments conducted in this paper indicate that SPELA closcly matches backpropagation in performance, and due to its computational efficiency, maintains an edge over it in resource- consstriee scenarios. Moreover, SPELA can efficiently fine-tune models trained with backpropagation (transfer learn- ing). In addition, extending SPELA to convolutional neural networks (CNNs) allows for complex image classification.
Complexity Analysis: Theoretical bounds for peak memory usage show that SPELA can edge over backpropagation in the analyzed settings.
2
Related works
Recently, there has been significant interest in designing efficient training methods for multi-layer neural networks. Hinton 2022) presents the Forward-Forward (FF) algorithm for neural network learning. In FF, backpropagation replaced with two forward passes: one with positive (real) data and the other using generated negative data, Each layer aims to optimize a goodness metric for positive data and minimize it for negative data. Separating positive and negative passes in time enables offline processing, facilitating image pipelining without activity storage gradient propagation interruptions. These algorithms have garnered significant attention, and multiple modifications have been proposed in conjunction with applications in
image recognition (Lee & Song. 2023 Pau & Aymone 2023 Momeni et al. 2023 Dooms et al., 2024 Chen et al. 2024) and in graph neural networks (Park et al., 2024). In our experiments, SPELA can classify any layer without storing goodness memory, in contrast to the FF approach, which aggregates goodness values across layers. Furthermore, unlike FF, SPELA eliminates the need to generate separate negative data for training. Instead, it efficiently uses the available data without requiring additional processing.
Using forward and backward passes, Pehlevan 2019 introduced the concept of non-negative similarity matching cost function for spiking neural networks to exhibit local learning and enable effective use of neuromorphic hardware. Lansdell et al.] (2020) introduced a hybrid learning approach wherein each neuron learns to approximate the gradients. The feedback weights provide a biologically plausible way to achieve performance comparable to networks trained via backpropagation. Giampaolo et al. (2023) follows a similar strategy to our approach of dividing the entire network into sub-networks and training them locally using backpropagation (SPELA divides the network into sequential layers). Rather than backward propagation, Dellaferrera & Kreiman (2022) uses two forward passes and uses the global error to modulate the second forward-pass input. Recently, Li et al. (2025) proposed diffusion-inspired denoising-basee training of neural networks. We consider Dellaferrera & Kreiman (2022) as the closest match to our approach, but with one forward pass and layer-specific non-global error during training for SPELA. Inspired by pyramidal neurons, Lv et al. (2025) used distinct weights for forward and backward passes to train a multi-layer network in a biologically plausible manner. As detailed by Pau & Aymone (2023): Srinivasan et al. (0024), these algorithms still lack several desired characteristics of on-device learning.
2

---

Published in Transactions on Machine Learning Research (06/2025)
3 Methods
3.1 Network Initialization and Learning Methods
The network defined follows: there L layers, each containing 4 neurons followed by nonlinear activation function (e.g Leaky ReLU), The weights of the network initialized randomly, Each layer L (except the imput layer) has N (number of classes in the given datasct) mumber of symmetric vectors, cach of dimension 4,- These symmetric vectors are assigned unique class. As the activation vector is also in the 4 dimensional space, we can measure how close the activation vector points to a particular symmetric vector using a simple cosine similarity function (Momeni et al. 2023). Based on cosine similarity, the network predicts the class assigned the symmetric vector closest to the activation vector. These symmetric vectors remain fixed and not updated during training. We describe the symmetric vector generation method in
detail in Section
symmetric in 7 dimensions
N
Prediction
Output
Prediction
3
Input
Aativation
784 784
(a) Network Architecture
(b) Prediction Method
Figure (a) Network Architecture: Each layer possesses distinct set of symmetrie vectors. Here, the network is trained on MNIST-10, resulting in 10 symmetric vectors. (b) Prediction Method: Inference is performed using the closeness of activation and symmetric vectors. The activation is represented in blue, and the prediction is in green.
3.2 SPELA description
Algorithm 1 and 3 describe SPELA's training and testing methodologies, respectively. In this method, after passing a data point through layer i, we update layer i's weights and biases and then propagate the data point to layer i++1 Each layer updates its parameters as the data moves Uhrough the network using its local loss function. In this fashion, parallel training of n layers becomes possible. Table contrasts SPELA and other learning algorithms. Our algorithm exhibits the most favorable traits for the applications discussed in this work: a single forward pass for training, backward pass, and local loss function with storage of activations.
3.3 Complexity Analysis
Updating the weights of the final layer in backpropagation requires one matrix-vector multiplication. Every other layer requires two matrix-veclor multiplications to consider gradients from subsequent layers. SPELA can visualized as cascading blocks one-layer networks that perform classification at every junction. can be viewed a sequence of final layers from the backpropagation algorithm. Updating the weights of any layer using SPELA (as the weight updates are all analogous the final layer of backpropagation) requires only metrix-vector multiplication. For deep neural nets, the number of vector-matrix multiplications
3

---

Published in Transactions on Machine Learning Research (06/2025)
Learning Methods
BP
FF
PEP
MPE
SPELA
Forward Pass
1
2
2
Backward Pass
1
0
Weight Update
1
2
Loss function
global
local
global
global
local
Activations
all
current
all
current
current
Table 1: Different learning algorithms are compared and contrasted with SPELA. PEP stands for PEPITA
Dellaferrera & Kreiman 2022) and MPE. for MEMPEPITA (Pau & Aymone 2023
Algorithm Training MLP with SPELA
1: Givem: An input (X), label ()), number of layers (K), and number of epochs (E)
2: Define: cos sim((A, B) = ||||_|
A.B and normalize(X) = || > Dot product and normalization of
vector
Set: ho
for 4- 1 to E do
Tterate through epochs Iterate through layers
for k +- to K do
6:
h = normalizo(hx-1))
7:
hi ox ox lx)
8:
loss: = -cos_ -cos_simmhh: vecs#
b vecsE(-) is the set of symmetric vectors b. Weight update using local loss > Bias update using local loss
We +- Wv -a a Ww (losss)
10:
by bk a Vol (lossk)
11: end for
12: end for
needed for weight updates half of what backpropagation requires. Furthermore, this excludes other op- erations required by backpropagation and not by SPELA, such as transposing of weights. Both algorithms would need one voctor-matrix multiplication for forward pass. Considering this, SPELA requires about 0.67 many malrix-vector multiplications as backpropagation (where the relative MACC for training is twice that of inference (Cai al. 20200). Regarding memory (see Table 2). we describe the complexities involved in variables that must be saved to calculate the weight updates. We do not include overhead memory complexities from storing weights, optimizer states, temporary variables, or other sources. At each computational step, SPELA trains only a single layer; hence, only that layer's activation needs to be stored. SPELA offers superior computational and memory complexity compared to backpropagation.
Forward pass complexity
Weight update Memory complexity
Algorithm
complexity
SPELA
N
LN2
N
BP
N
2LN
LN
Table 2:
The computation and memory complexities are shown above. The NN considered to have L
layers, each having N neurons, The complexity a vector matrix multiplication is assumed to be O(N2).
Here, for SPELA, it is assumedd that activations are not stored.
3.4 The learning in SPELA
During SPELA learning, every layer can considered a classifier head. Instead moving the activation towards a one-hot encoded vector as in conventional training schemes, we focus on moving the activation vector towards prodefined symmotric vector. At each layer, output o; = ||l!|| where h; = a24), where
by
2; = Wih,-1 + b,, where W; and by are layer weights and bias, h411 is the previous layer activation output, a() an activation such Leaky ReLU, Also, assume that the symmetric vector assigned to the correct class v. Then the loss is defined as the negative of cosine similarity: = -oo = v, as ||o = ||a|| 1. Mathematically, compute
4

---

Published in Transactions on Machine Learning Research (06/2025)
0L = of U - = Jo and
0
ØL do;
Oh, ah, 9o = ||hi|| (I a,0,') = gh.
0o:
do,
0L
2 0,
02, = 9h 2 (zi) = yz, where d'(2)=
ØL ow. = geh and
ØT. Ib.
The same process is identically applicable to all layers L in the network. This finally results in W +- W -9x; where 7 is the learning rate and t is iteration step. In our implementation, the
79ghi-1 and by 4 b9 -9-1 symmetric vector embeddings represent the layer-wise classifier head weights, which emulate cosine similarity computation, without getting updated during training. The output of this classifier head is utilized to compute & cross-entropy loss or a cosine loss (log(2 cosine similarity). The derivation above assumes a simpler setting, allowing readers to understand weight updates.
3.5 SPELA for Convolutional Neural Network
For the Convolutional Neural Networks (CNNs) (Figure 3a use the interleaving of traditional CNN and lulti-layer perceptron (MLP) layers to incorporate SPELA. The modifications are as follows: each kernel in the convolutional layer is assigned a cortain number of groups. Each group has a nonzero number of classes. Each class belongs to exactly one group, with all classes evenly distributed across groups. Each CNN layer predicts the group assignment of an input class. For example, when the groups are (dog, cat, fish), (banana, boat, bug), and (football, airplane, phone), then the class 'dog' will belong to the first group. The number of groups and classes assigned to each group varies between kernels, with randomness facilitating our performance. Each class is given a score depending on what the kernel returns. Tf a kernel returns the first group, all the classes corresponding to the first group get a score of one. Similarly, if a kernel returns the second group, the corresponding classes are scored, and so on. After tallying the scores, the class with the highest cumulative score is selected as the CNN layer's output. This particular distribution of groups and classes in groups is random across kernels but consistent across layers, one of the restrictions of our method. After obtaining the output of a particular CNN kernel, this slice of 2D data is flattened and projected down to smaller dimension using simple MLP. The classification is performed in the MLP precisely in typical SPELA MLP setup. The data is pushed through the network after the classification. We keep the MLP as tiny as possible to mitigate learning in this perpendicular direction and focus more on training the CNN layer. Algorithm 2 details the layer-wise training procedure for CNNs.
4 Empirical Studies
4.1 How does SPELA work?
We perform an in-depth analysis of SPELA's capacity. We first understand SPELA's learning dynamics on the standard MNIST 10 dataset. Following the design described in Dellaferrera & Kreiman
(2022) we 784 - 1024 - 10 size SPELA network on MNIST 10. The network is trained by cosine loss defined as log(2- - Cosine Similarity). Figure 2a describes both layers of SPELA's 's epoch-specific decreasing training loss curves. The mean loss for layer 2 (0.26 after 200 epochs) is always lower than the corresponding layer loss (0.34 after 200 epochs). Figure 2b describes the rise of test accuracy with the number of training epochs with SPELA. It is observed that SPELA training is effective at improving the accuracy from 11.60% without learning to 94.49% after 200 epochs of training. By design, SPELA can perform predictions at all layers (except the input layer), wherein the amount of available resources can determine the mumber of layers. Figure [b] also shows that the accuracy increases with the layer counts on MNIST 10 191.18% accuracy for layer one vs. 94.49% accuracy for layer two after 200 epochs of training), thereby justifying the need for multiple layers to improve network performance. This also empowers SPELA with early exit feature (Scardapane al. 2020), enabling easy neural network distribution across hardware
evaluate the performance
5

---

Published in Transactions on Machine Learning Research (06/2025)
Algorithm Training and Inference from CNN ith layer with SPELA
1: Given: к number of classes, C' 2.., Ki,, n, kernels and B, is conv block from previous layer Define: S score for class i, Vi E
Define: groups such that:
G2 CC
m C
GG v
4: for jorje + to do
Define MLP for each kernel
5: MLP,, (0, 1)xxd
6: end for
7: for +- to ni do
Kernel + MLP operation
0f CNN(B, ky)
-y is the jth kernel
9:
; flatten(o))
10:
Oj (LP)
11:
Loss, = -cos_ )
12:
Say the closest predicted group is Gm via cos sim loss
13:
for in Gm do
14: S +
15: end for
16: end for
17: for
+ 1 to do
18: Wky f- Wk, a** Vky (loss:)
Kernel parameter update using local loss MLP parameter update using local loss
19: WMLP, + WMLP,
VMLP, (lOSsR)
20: end for
21: Prediction: arg max, S₁
platforms and improving inference. Moreover, it is observed that SPELA reaches near maximum performance very quickly. SPELA achieves 74.88% performance after only 1 epoch of training on MNIST 10. Next, we analyze the effect of SPELA training on network parameters in Figure 2c The Frobenius norm of the layer weights demonstrates increasing trend with learning, precisely 45.27, 4.49 (before lcarning) to 56.16, 57.86 (after 200 epochs) for layer 1, and layer 2, respectively.
Loger
012
0.5
Layer1 Tut
Layer1 3et
1ape72 Tan
Layt22
Thaniing Soodle
Ta nang Epocta
Thaning Epects
(a) Loss Curve w.r.t Epochs
(b) Accuracy w.r.t Epochs
(c) Norm of weights w.r.t Epochs
Figure 2: SPELA network behavior during learning to classify MNIST 10 digits. (a) Cosine training loss of
SPELA to the progression of epochs. (b) Train and Test accuracy of both the hidden layer(1024 neurons) and the output layer(10 neurons) over epochs.
(c) Evolution of Frobenius norm of the layer weights for SPELA with learning. The solid lines denote the mean, and the shades denote the standard deviation of five simulation runs.
Figure 3b establishes the reprosentation learning capabilitics of SPELA. A t-SNE embedding analysis of the output layer representation (10 neurons) before learning Figure 3b exhibits high degree of overlap of the MNIST 10 digit classes. However, after training with SPELA for 200 epochs of the network Figure 3c

---

Published in Transactions on Machine Learning Research (06/2025)
observe 10 distinct clusters corresponding to MNIST classes in the t-SNE embeddings of the output layer representation.
Date
Kemei
Kernel
MLP n
MLP n
Ciassfication fellowed by local loss using SPELA
(b) Before Learning
(c) After Learning
Figure 3: (a) Schematic diagram of SPELA Convolutional Neural Network. (b), (c) Two-dimensional t-SNE embeddings of the output layer(10 neurons) of a SPELA MLP network trained on MNIST 10 (b) Before any training, (c) After training for 200 epochs. The colors corresponded to different digits of MNIST 10.
SPELA can predict directly by comparing the layer output with the symmetric class embeddings. However, we parallelize this comparison using classifier head with weights determined by the class embeddings. This design can either a cosine loss (SPELA) or cross-entropy loss (SPELA CH) for layer training. We compare SPELA's performance wherever possible with existing work. Table 3 describes the key comparison results on MNIST 10, KMNIST 10, and FMNIST datasets (we selected these datasets as they are well suited for a multilayer perceptron classification task). Table 7 similarly describes the performance of SPELA on reasonably complex datasets such as CIFAR 10, CIFAR 100, and SVHN 10. Notably, across both tables, SPELA is the only training method that operates with single forward pass; in contrast, backpropagation (BP), feedback alignment. (FA), direct random target projection (DRTP), and PEPITA require either forward pass followed by backward pass, two consccutive forward passes.
To ensure a fair comparison, we evaluate SPELA using two configurations: (i) baseline matching the setup from previous work (Dellaferrera & Kreiman 2022 (SPELA CH A and SPELA A), and (ii) an optimized configuration (SPELA B and SPELA B). Full experimental details are provided in Tables 19 and 20
On the MNIST 10 dataset, SPELA achieves an accuracy of only 3.91% lower without any dropout than an equivalent backpropagationn-taainet network (94.49% vs. 98.40%). Interestingly, 784 + 1000 SPELA achieves 91.47% performance for the same training. Using the same network configuration, SPELA achieves 77.13% on KMNIST-10 and 85.08% Fashion MNIST. Additionally, we observe that using cross-entropy loss (SPELA CH) slightly improves performance on KMNIST-10 and Fashion MNIST compared to cosine loss (77.13% vs. 76.36%, and 85.08% vs. 85.00% respectively). Conversely, on MNIST-10, cosine loss yields marginally better results (94.4%% vs. 94.14%). Finally, incorporating dropout did not lead to performance improvements for SPELA across any of the datasets.
Keeping the input and output layer sizes fixed at 784 and 10, we varied the number of hidden layers (1024 neurons each) of SPELA from to 9. Table 4 describes the results of such multilayer SPELA. We observe that 4-hidden-layer SPELA performs best on MNIST (96.28%) Similarly, 7-hidden layer and a 4-hidden layer perform best on KMNIST 10 and Fashion MNIST 10 respectively (83.57%, and 87.16%).
By design, SPELA does not require a softmax layer whose dimensions are usually equal to the mumber of classes the dataset. Without this constraint, we explore the optimal network configuration, keeping the total number neurons constant (1024 + 10 1034). Figure 4 and Table clearly show that SPELA's per- formance depends output layer size. When the output layer size is (and the corresponding hidden layer
Tleaary

---

Published in Transactions on Machine Learning Research (06/2025)
size is 1029), MNIST 10, KMNIST 10, and Fashion MNIST 10 performances are merely 53.19%, 35.66% and 50.31% respectively. Conversely, 784 + 984 50 architecture, wherein the output layer size is five times that of the number of classes, achieves the best MNIST 10, KMNIST 10, Fashion MNIST 10 performances of 96.28%, 80.23% and 87% respectively. This work does not aim to compete with backpropagation (BP). Instead, our focus is on ANN training for applications where BP is computationally infeasible due to issues such as storage of forward pass activations (such as in the case of on-device learning with backpropagation Caa et al. 20201).
Model
Architecture
# Epochs
MNIST 10
KMNIST 10 92.73 + 0.11
FMNIST 10 88.72 + 0.21
BP A
784 - 1024 10
100
98.56 + 0.06
FA
784 + 1024 + 10
100
98.42 = 0.07
DRTP
784 > 1024 10
100
95.10 + 0.10
PEPITA
784 + 1024 + 10
100
98.01 + 0.09
FF
784 + 4 X (++ 2000)
60
98.6
SPELA CH A
784 + 1024
100
87.02 = 0.2
59.73 + 0.23
73.63 + 0.52
SPELA CH A
784 + 1024 10
100
82.41 = 5.16
72.52 + 3.13
77.17 + 6.11
SPELA A
784 + 1024
100
89.55 + 0.06
64.27 + 0.19
79.25 + 0.11
SPELA A
784 + 1024 10
100
90.27 + 4.17
64.71 + 6.96
72.58 + 8.23
BP CH B
784 + 1024 + 10
200
98.40 = 0.08
88,81 = 2.03
91.89 + 0.23
SPELA CH B
784 + 1024
200
91.47 + 0.06
68.7 + 0.24
83.68 + 0.04
SPELA CH B
784 + 1024 + 10
200
94.14 + 1.22
77.13 + 2.23
85.08 + 0.58
SPELA B
784 + 1024
200
91.18 + 0.09
68.7 + 0.17
84.01 + 0.10
SPELA B
784 + 1024 + 10
200
94.49 + 0.74
76.36 + 2.79
85 + 1.2
Table 3: Test accuracies (mean + standard deviation) comparison of different learning methods on MNIST
10, KMNIST 10, and FMNIST 10 (Fashion MNIST 10) datasets. The accuracies of FA, DRTP, and PEPITA
are presented in Dellaferrera & Kreiman [0022) FF results are reported in Hinton (222 for network of 4 hidden layers with 2000 neurons cach. We report both hidden layer and output mean accuracics (average
of five runs) of SPELA. Table 19 and Table 20 describe all the relevant experiment details.
Dataset
# 1
# 2
# 96.23
#
# 5
6
#
# 8
# 9
91.21
95.7
96.28
96.22
96.26
96.15
96.16
96.14
MNIST 10
+ 0.11
+ 0.08
+ 0.07
+ 0.09
+ 0.05
+ 0.08
= 0.06
- 0.06
+ 0.08
68.70
80.32
83.11
83.34
83.47
83.48
83.57
83.16
83.23
KMNIST 10
+ 0.11
+ 0.27
+ 0.13
+ 0.06
+ 0.21
+ 0.19
= 0.20
+ 0.2
+ 0.22
83.98
86.81
87.14
87.16
87.16
87.12
87.07
87
86.95
Fashion MNIST 10
+ 0.07
+ 0.09
+ 0.08
+ 0.06
+ 0.08
+ 0.07
- 0.13
+ 0.15
+ 0.06
Table 4: Test accuracies (mean + standard deviation) of SPELA for varying network depth. Columns indicate the number of 1024-neuron hidden layers in the network. For instance, column #1 denotes a 784 + 1024 + 10 network, #2 denotes a 784 > 1024 + 1024 + 10 network. Each net with SPELA B configuration is trained for 200 epochs and five runs.
4.2 Training Memory Comparison
Next, we analyzed and compared SPELA's peak training memory consumption with equivalent BP network. Training memory typically dominates model memory during learning; hence, we focus on peak training memory usage. Figure 5 presents the relative peak GPU memory consumption when the number of hidden layers varies from 1 to 9. For all batch sizes: 50, 100, 200, 400, 800, and 1000, the relative peak training memory consumed by SPELA training is almost the same, resulting in flat curve. We use the training memory consumed by 784 1024 - 10 base for computing the relative peak training memory. Meanwhile, for BP network, the relative peak training memory increases with the number of hidden layers
8

---

Published in Transactions on Machine Learning Research (06/2025)
3
Lear1
(a) MNIST 10
(b) KMNIST
(c) Fashion MNIST
Figure 4: Variation of SPELA's test performance for output layer sizes of 2, 5, 10, 50, 100, 200, 400, 800, and 1000. We keep the total number of network neurons fixed to 1034; hence, the corresponding hidden layer sizes are 1032, 1029, 984, 934, 634, 234, and 34. Throughout the experiments, the SPELA B configuration is used for 200 training epochs. The solid lines denote the mean, and the shades denote the standard deviation of five simulation runs.
and batch size. Notably, the training memory is the residue after subtracting the model memory from the total peak memory. Due to SPELA's design, such as a per-layer classifier head which uses the vector embeddings as weights, the model memory of SPELA should be very slightly higher than that BP, which we observe in Table (for instance, 38.20 MB vs. 37.83 MB for nine-hidden-layer layer network).
BP
Peak Training Memory(Relabbee
2.75
SPELA 50
BP 100
SPELA
2.25
BP 200
SPELA 200
2.00
BP 400
SPELA 400
1.50
BP 800
SPELA 800
1.25
BP 1000
SPELA 1000
No.of Hidden Layers
Figure 5: Variation of relative peak training camnorissaaverage of five runs) with layer depth for SPELA and BP. We use the peak training memory occupied by 784 - 1024 10 SPELA/BL network to report the relative peak training memory. The hidden layer sizes vary from 1-9 with 1024 neurons in each hidden layer, and for batch sizes of 50, 100, 200, 400, 800, and 1000.
4.3 Transfer Learning with SPELA
In real-world applications, on-device learning(ODL) helps mitigate the impact of phenomena such as data drift by enabling the on-device update of ML models. However, on-device learning must be performed under constraints such as memory and power for tiny ML applications. Consider a neural network (NN) with three layers: Top L1, and L.2 Previous work on on-device learning using backpropagation shows that forward pass activation storage of such layers consumes significantly higher memory than neural network parameters (Cai et al. 2020 This assumes network design requiring Lo + L, L2 synapses well as Lg < L, +- La synapses. SPELA, the other hand, would need only unidirectional synaptic connection Lo + L1 + Lg iimplying SPELA would need fewer connection wires. These traits should make SPELA useful in tinyML applications. In this section, we examine the behavior of SPELA in the framework of liny transfer learning, wherein the models are pre-trained with backpropagation and optimized using SPELA. We compute the top-1 and top-5 accuracics for varying degrees of training data(keeping the test dataset fixed). We follow the canonical transfer learning approach wherein the classifier head replaced by layer size equivalent to the number of classes. Moreover, observed in Figure 4 since SPELA performs better when the output layer

---

Published in Transactions on Machine Learning Research (06/2025)
is 5x the mumber of classes, also evaluate performance SPELA 5x. Figure 6 and Tables 10
describe the performance on the six datasets. Although the ResNet50 model is trained with backpropaggtton((B)) and should be the obvious training method, SPELA doesn't 't lag far behind on the six datascts. Morcover, similar to previous observations in Figure 4 SPELA 5x always outperforms SPELA. For instance, SPELA 5x achieves a performance 98. 28.20% (vs. 99.12% 12% for a backpropagation network) on CIFAR datasct, 85. (vs. 87.46% for a backpropagation network) on CIFAR 100 datasct, 98.48 (88...8vv. 98. 08888% for backpropagation network) on Pets 37 dataset, 82.96% 85.21% % for backpropagation network) on Food 101 dataset.
12
13 14 15
Backprop: Top
SPELA: Top
SPELA 5x: Top 1
Backprop: Top 5
SPELA: Top 5
SPELA 5x: Top
200
100
309
Accuracy (in %6)
Accuracy (in 9%)
Accuracy (In %)
1.5
12
10
Train Size (in %))
Train Size (in %)
Train Size (in %)
(b) CIFAR 10 dataset
(c) CIFAR 100 dataset
(d) Pets 37 datasct
Figure 6: Accuracy plots (after 200 epochs of fine tuning) of SPELA, SPELA 5x and Backpropagation trained networks for Irain dataset size pgrcentages of 1, 5, 10, 25, 50, 75, and 100 during transfer
the test dataset fixed). The solid lines denote the mean, and the shades denote the standard deviation of five simulation runs. SPELA 5x denotes a network with a classifier layer size 5x the number of classes.
4.4 Ablation Studies
Why not use alternate distance? Just we try to orient the vectors to their corresponding embedded direction for correct classification, another question arises: Instead of classifying data in terms of closeness concerning angle (cosine loss), why not classify data in terms of closeness concerning distance (Euclidean loss)? Here, we run experiments by replacing our Cosine loss function defined by log(2 - cosine similarity) with the Euclidean norm loss function. Table 5 shows that when lested on MNIST 10, KMNIST 10, and Fashion MNIST 10 datasets, SPELA with Euclidean distance performs significantly lower than SPELA with cosine distance after training of 200 epochs (75.37% vs. 94.41% for MNIST 10, 52.71% vs. 75.05% for KMNIST 10 and 67.83% vs. 84.12% for Fashion MNIST 10).
Cer et al. 2018)) justified that small angles have very similar cosines and proposed using angular similarity: arccos(m) for classification. Table shows that angular loss defined by log(2 angular similarity) performs slightly better than cosine loss for KMNIST 10 (75.44% vs. 75.05%) and Fashion MNIST 10 (85.1% vs. 84.12)). On MNIST 10, angular loss performs almost at par with cosine loss (94.39% vs. 94.41%).
т
Randomizing the vector embeddings Symmetrically distributing the vectors is intuitive and axiomatic, as explained in Section A.1 However, the question remains: How much would the performance drop if we randomly choose these vectors? Accordingly, we rid ourselves of the complexity of finding a symmetric distribution and run experiments by drawing the embedded vectors from Gaussian distribution A 1) as well as from a uniform distribution ((-1,11), 1). (aable]jindiee for all three datascts when vectors are drawn at random. However, this performance drop is significantly lower than Euclidean distance in SPELA (MNIST 10: 4% vs. 19.54%, KMNIST 10: 6.65% vs. 22.34%, Fashion MNIST 10: 0.09% vs. 16.29%). In Appendix B.3.2, we discuss this observation further.
that classification performance slightly drops
Except for cosine and angular similarity in all of these studics, for SPELA, the first layer classification is better than the subsequent layer in the early exit set-up.
Binarization (+1) of SPELA Anderson & Berg 2018 shows that binarization does not significantly change the directions of the hime-diin vectors. As our algorithm tries to orient the activations
10

---

Published in Transactions on Machine Learning Research (06/2025)
to particular direction in high dimensions, it is safe to assume that the weight binarization should not significantly affect the performance. In Table 5 we show the results of our experiments involving the binarization of weights (here, we do not binarize the bias involved at each layer, only the weights to =1). Table 5 shows that our assumption is experimentally proven accurate. Binarization of weights while dealing with vectors in high dimensions does not. change the relative angular positions significantly; hence, the accuracy does not drop significantly cither. This modification could lead a far more efficient algorithm that cuts down on memory and energy consumption, as well as the area occupied by a chip, while not KMNIST 10 and Fashion MNIST 10 datasets, SPELA (with
sacrificing performance. Noteworthily, +1 weights) performs better than an equivalent backpropagation trained network (MNIST 10:91.04% vs. 93.57%, KMNIST 10: 68.69% vs. 67.98%, Fashion MNIST 10: 84.06% vs. 76.38%).
Learning rate dependence of SPELA Next, establish the learning rate dependence of SPELA's performance after 200 training epochs. Accordingly, Figure 7 and Table 17 describe the variation of test accuracies on MNIST 10, KMNIST 10, and Fashion MNIST 10 for learning rates of 0.01, 0.1, 1, 1.5, 2.5, and 3. We observe that MNIST 10 performs best (94.25%) for learning rate of 2.5, KMNIST 10 (75.69%) for learning rate of 3, and Fashion MNIST 10 (85.34%) for a learning rate of 0.1.
Table 22 provides the relevant experiment details for the above ablation studies.
Model
Architecture
MNIST 10
KMNIST 10
Fashion MNIST 10
SPELA-Coo
784 + 1024
91.09 + 0.12
68.74 0.18
84 + 0.11
SPELA-Cos
784 + 1024 + 10
94.41 + 0.49
75.05 3.47
84.12 + 3.30
SPELA-Areoce
784 -> 1024
91.03 + 0.06
68.39
0.12
83.85 + 0.08
APPLAA-Arcoo
784 + 1024 + 10
94.39 + 0.91
75.44
2.01
85.1 + 0.71
SPELA-EA Euc
784 -> 1024
75.37 + 0.34
52.71
= 0.28
67.83 = 0.77
SPELA-Fau
784 + 1024 + 10
48.16 + 8.76
28.16
= 8.14
51.82 + 7.18
OPELA-RandNor
784 + 1024
90.91 + 0.08
68.29
0.21
84.03 + 0.09
SPELA-RandNor
784 + 1024 - 10
86.7 + 9.76
68.4 + 6.97
79.89 + 6.25
OPELA-RandUni
784 -> 1024
90.27 + 0.16
67.37 = 0.34
83.68 + 0.04
PFELA-RandUnff
784 + 1024 + 10
84.12 + 5.17
65.45 = 4.69
73.08 = 7.28
SPELA-Bin
784 -> 1024
91.04 + 0.09
68.69 = 0.17
84.06 + 0.04
SPELA-Bin
784 + 1024 + 10
79.39 + 12.67
61.80 = 4.02
74.97 + 4.09
BP-Bin
784 + 1024 + 10
93.57 + 0.83
67.98 = 4.80
76.38 + 4.09
Table 5: Ablation study results of SPELA MNIST 10, KMNIST 10, and FashionMNIST 10 datasets after 200 training epochs (and for five runs). SPELA-Cos implies SPELA with cosine distance, SPELA- Arccos implics SPELA with Are cosine distance, SPELA-Euc implics SPELA with Euclidean distance, SPELA-RandNorm implies SPELA with random vectors from a normal distribution, SPELA-RandUnit implies SPELA with random vectors drawn from a uniform distribution, SPELA-Bin Sin/B--Bit implies with #1 weights.
100
100
200
Layor L
Layer
000
50
3.1
04
15
25
Goanerg
Loamnig Nate
Loarning
(a) MNIST 10
(b) KMNIST 10
(c) Fashion MNIST 10
Figure 7: Variation of SPELA test accuracies (after 200 epochs of training) for learning rates of 0.01, 0.1, 1, 1.5, 2.5, and Solid lines denote mean accuracies, and shades denote standard deviation over five runs.
11

---

Published in Transactions on Machine Learning Research (06/2025)
4.5 SPELA Convolutional Neural Network
We evaluate the performance of our SPELA convolutional neural network (CNN) on image classification tasks. Accordingly, we here enlist the performance of SPELA CNN on complex datasets such as CIFAR 10,
CIFAR 100, and SVHN 10. Table 23 summarizes the details of our experiments. Similar to Dellaferrora & Kreiman (022): : Liao et al. 2016 we compare the relative performance of SPELA CNN to reported results on previously proposed backpropagation alternatives (Table 23). Table 7 aimss to showcase that our SPELA CNN can match these performances with only local learning and early exit capabilities. Of course, the performance of the CNN will improve by adding global error akin to backpropagation. Although SPELA doesn't 't require a global error signal (either through backpropagation or as second forward pass), a one- layer SPELA CNN attains a best performance of 45.21%. This is 5.21% lower than DRTP. Adding another layer to SPELA CNN improves performance to 56.33%, similar to a one-layer PEPITA. On the SVHN 10 dataset, a two-layer SPELA CNN archives a mean accuracy of 79.02%, which is a significant improvement over vanilla SPELA performance on SVHN 10 (66.89%; Table 7
Model
CIFAR 10
CIFAR 100
SVHN 10
BP
64.99 + 0.32
34.20 + 0.20
FA
57.51 + 0.57
27.15 + 0.53
DRTP
50.53 + 0.81
20.14 + 0.68
PEPITA
56.33 + 1.35
27.56 + 0.60
SPELA CH B CNN(1)
45.21 + 12.67
19.62 + 5.19
70.10 + 16.68
SPELA CH B CNN(2)
52.4 + 6.98
21.43 + 5.94
79.02 + 1.01
SPELA_ B CNN(1)
44.35 - 13.48
22.08 + 4.53
78.14 + 0.71
SPELA B CNN(2)
56.59 - 0.97
26.71 Σ 0.79
76.28 + 6.45
Table 6: Test accuracies (mean + standard deviation) comparison of different CNN architectures on CIFAR 10, CIFAR 100, and SVHN 10 datasets. The accuracies of BP, FA, DRTP, and PEPITA are represented from Dellaferrera & Kreiman (2022). We report both hidden layer and output mean accuracies (average of five runs) of a SPELA convolutional neural network (CNN). SPELA CNN(1) and SPELA CNN(() imply a network with a 1-layer and 2-layer CNN, respectively. BP, FA, DRTP, and PEPITA results are presented the best-case scenario, which SPELA attempts to match under the constraint of equivalent CNN layers.
5 Discussion
We propose SPELA as a computationally efficient learning algorithm to train neural networks. For the experiments conducted, SPELA has a feature set comprising symmetric embedded vectors, local learning, early exit, and a single forward pass for training with no storage of activations, no weight transport, and no updated weight locking. As part of it, we perform detailed experiments to benchmark SPELA against existing works. In addition, we analyzed its suitability for transfer learning backpropagatioon-taainn image recognition networks. We perform ablation studies on SPELA to justify the design choices. Finally, we extend SPELA to convolutional neural networks (CNN) and benchmark it on equivalent image recognition tasks. Analyzing theoretical complexity (lower bound) and on-device implementation shows that SPELA is more efficient than backpropagation regarding memory utilization. Hence, this work can help guide the implementation of on-device learning in tiny microcontrollers such as ARM Cortex-M devices. Overall, we believe SPELA is helpful for wide range of ML applications, wherein we care about training and testing efficiency regarding accuracy and memory. To conclude, we view this work as starting point for developing efficient learning algorithms that can aid in applications where backpropagation presently has limitations. In the future, we will extend SPELA to the training and inference of advanced architectures such as deep convolutional neural networks and transformers. Although SPELA is biologically inspired, it lacks some key features biology, such as the spiking behavior of neurons. Further work is necessary to integrate such features with SPELA.
12

---

Published in Transactions on Machine Learning Research (06/2025)
References
Mohamed Akrout, Collin Wilson, Peter Humphreys, Timothy Lillicrap, and Douglas B Twoed. Deep learning without weight transport. In H. Wallach, Н. Larochelle, A. Beygelzimer, F. d' B'Acche-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL https://proceeding a Paper.
Alexander G. Anderson and Cory P. Berg. The high-dimensional geometry of binary neural networks. International Conference on Learning Representations, 2018. URL https://openreviier net/forummid- B1IDRdeCW
Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not paramcters for cf- ficient on-device learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 11285- 11297. Curran Associates, Inc., 2020. URL https://proceeddnng neurips wwtpseeecrrppppppppeeeeeee femper_ffleeee fil.es//pppeee//00//illl BLf/co004412444c666c000006668684
Paper pdff
Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Ghajardo-Cespeddes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Universal sentence encoder, 2018. URL https: http:////aarii.oo/aaa////ee//// wwwg/bs////////ooo 11175
Xing Chen, Dongshu Liu, Jeremie Laydevant, and Julie Grollier. Self-contrastive hawaadd-foraate algorithm, 2024. URL
Henry Cohn and Abhinav Kumar. Universally optimal distribution of points on spheres. Journal of the American Mathematicol Society, 20(1):99-148, 2007.
Wojeiech Marian Czarnecki, Grzegorz Swirszez, Max Jaderberg, Simon Osindero, Oriol Vinyals, and Koray Kavukcuoglu. Understanding synthetic gradients and decoupled neural interfaces. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML'17, pp. 904 912. JMLR.org, 2017.
Giorgia Dellaferrera and Gabriel Kreiman. Error-driven input modulation: Solving the credit assignment problem woth backward pass. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 4937-4955. PMLR, 17-23 Jul 2022. URL mlr. mrl.
html
Thomas Dooms, Ing Jyh Tsang, and Jose Oramas. The trifecta: Three simple techniques for training deeper forward-forward networks. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL
https://openreevier htt.:///ooruiiie///////oo
Fabio Giampaolo, Stefano Izzo, Edoardo Prezioso, and Francesco Piccialli. Investigating random variations of the forward-forward algorithm for training neural networks. In 2023 International Joint Conference Newral Networks (IJCNN), pp. 7, 2023. doi: 10. 11110//JCNNN555..2222.....
Anirudh Goyal and Yoshua Bengio. Inductive biases for deep learning of higher-level cognition. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 478822666: 2022. doi: 1000//9p...20011 0068
URL http:://royallocceetvbbbbbbbb httpshhwwg..gg./////////.......
org/ abtp//bb//o.oeee wwwp:100.////0000000 2021.
Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations, 2022.
Bernd Illing, Jean Robin Ventura, Guillaume Bellec, and Wulfram Gerstner. Local plasticity rules can learn deep representations using self-supervised contrastive prodictions. In A. Beygekimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021.
13

---

Published in Transactions on Machine Learning Research (06/2025)
Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Reseorch, pp. 1627-1635. PMLR, 06-11 Aug 2017. URL https: //proceedings.oo www.engsmmmm.....mmmmmm........m pptps////o///dddebbbr.ll.aa....
Benjamin James Lansdell, Prashanth Ravi Prakash, and Konrad Paul Kording, Learning to solve the credit assignment problem. In International Conference on Learning Representations, 2020. URL https:
//openreview.nen wtwp:eeviwiieeeeouuuuuuueaeeeeee wttt:/foruumidd-yyeeB........
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Back- propagation applied to handwritten zip code recognition. Neural Computation, 144):511-55) 1989. doi: 10.1162/neco.19h 1989....9999......11.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521 444, May 2015.
ISSN 1476-4687. doi: 10.1038 10.1038/naturel /natureee5539 URL http:s//ooai.oogg/00111///tttert
Broug--hhang Lee and Jeonggeun Song. Symba: Symmetric backpropaggation-fice contrastive learning with forward-forward algorithm for optimizing convergence, 2023.
Qinyu Li, Yee Whye Teh, and Razvan Pascanu. Noprop: Training neural networks without back precppropagation or forward-propagation, 2025. URL http:p://aaxxi..og/abb//////////
24322
Qianli Liao, Joel Z. Leibo, and Tomaso Poggio. How important is weight symmetry in backpropagation? In Proceedings of the Thirticth AAAI Conference Artificial Intelligence, AAAI16, pp. 1837-1844. AAAI Press, 2016.
Timothy P. Lillicrap, Daniel Cownden, Douglas B. Tweed, and Colin J. Akerman. Random feedback weights support learning in deep neural networks, 2014.
Changze Lv, Jingwen Xu, Yiyang Lu, Xiaohua Wang, Zhenghua Wang, Zhibo Xu, Di Yu, Xin Du, Xiaoqing Zheng, and Xuanjing Huang. Dendritic localized learning: Toward biologically plausible algorithm, 2025. URL
Ali Momeni, Babak Rahmani, Matthieu Mallejac, Philipp del Hougne, and Romain Fleury. Backpropagation- free training of deep physical neural networks. Science, (8766))) 1303, 2023. doi: 10.1 10.1126/science. science. adi8474. URL https::/www: www.ssciiccc............
ad18474
Namyong Park, Xing Wang, Antoine Simoulin, Shuai Yang, Grey Yang, Ryan A. Rossi, Puja Trivedi, and Nesreen K. Ahmed. Forward learning of graph neural networks. In The Twelfth International Conference on Learning Representations, 2024. URL https:://openrevviw http://wwwww..eee..ooooireeeedre
Danilo Pietro Pau and Fabrizio Maria Aymone. Suitability of forward-forward and pepita learning micommons-tiny benchmarks. In 2023 IEFE International Conference on Omni-layer Intelligent Systems (COINS), pp. 1-6, 2023. doi: 10.1109/COINS57S55 202.22001..2.223
Congiz Pehlevan. A spiking neural network with local learning rules derived from nonnegative similar- ity matching. In ICASSP 2019 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 0958-7962, 2019. doi:
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Willians. Learning representations by back- propagating errors. Nature, 323 (3088):3333 536, Oct 1986. ISSN 1476-4687. doi: 10.1038/3235332 URL
Edward B Saff and Amo BJ Kuijlaars. Distributing many points on a sphere. The mathematioal intelligencer, 19:5 11, 1997.
Simone Scardapane, Michele Scarpiniti, Enzo Baccarelli, and Aurelio Uncini. Why should we add early exits neural networks? Cognitive Computation, 12(5) 966, June 2020. ISSN 1866-9964. doi: 10.1007//122559-000 URL http::tttp.pddee
14

---

Published in Transactions on Machine Learning Research (06/2025)
Ravi Srinivasan, Francesca Mignacco, Martino Sorbaro, Maria Refinetti, Avi Cooper, Gabriel Kreiman, and Giorgia Dellaferrera. Forward learning with top-down feedback: Empirical and analytical characterization, 2024.
Zihao Wang and Tei Wu. Theoretical analysis of the inductive biases in deep convolutional networks. Tn Terrth- Conference on Neural Information Processing Systems, 2023. URL https://openrevieer
net//forumm?dd=oloWWaaaaal
P.J. Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78 (10):155--1600 1990. doi: 10.1109/5.58337.
James C.R. Whittington and Rafal Bogacz. Theories of error back-propagation in the brain. Trends in Cognitive Sciences, 23(3):233 250, 2019. ISSN 1364-6613. doi: http:://tbi.org/110.006// tic..............00000 URL httpps//ww..sseeecccrree... htt.///cww..........c..cccaaaaaa
Shuai Zhu, Thiemo Voigt, JeongGil Ko, and Fatemeh Rahimian. On-device training: A first overview on existing systems, 2023.
15

---

Published in Transactions on Machine Learning Research (06/2025)
A Additional Methods
A.1 Electron simulation description
evenly distributed circle is simple: dividing the circle into equal arcs and placing the points at their endpoints. For higher-<limnsioonll spheres (4; 2 3), the problem is non-trivial and does not admit a closed-form solution. We use a well-known method of electron approximation to simulate N electrons in any D-dimensional space. We constrain each electron to a unit-radius ball and simulate the relative forces among electrons (Saff & Kuijlaars, 1997 Cohn & Kumar 2007]. At each iteration, we move each electron according to the direction and magnitude of the net force, respecting the unit-ball constraint. We define stability in terms of relative change in electrostatic potential. As the electrons shift around the ball, the elect rostatic energy changes. As the electrons redistribute, their movements occur over progressively smaller distances. The relative change in electrostatic energy tends to zero with iterations. Once this relative change is less than defined threshold, we claim to have stable configuration and obtain an 4-ddimensional ball with a symmetric distribution of N electrons (see Fig. la
Figuring out N points
A.2 Algorithms
Algorithm 3 Inference on MLP trained with SPELA
1: Given: An input (X) and number of layers K
2: Define: cos sim( A, B)
A.B [А
Set: ho
4: for & +- to K do
b Passing data through all the layers
5: lig = o W lix-1111 bk)
6: end for
7: for - to N do
is the number of classes Similarity between activation vector and symmetric vectors
S. cos cos__sssm___ sim(hg, vi))(i))
end for 10: Prediction: arg max, S;
D Class corresponding to the maximum score is prediction
Additional Results
B.1 How does SPELA work?
Model BP
Architecture
# Epochs 100
CIFAR 10
CIFAR 100
SVHN 10
3072 + 1024 10
53.48 0.36
27 0.21
77.83 0.48
3072 1024 + 10/
FA
100
53.82 0.24
24.61 + 0.28
3072 1024 + 100
3072 1024 10/
DRTP
100
45.89 + 0.16
18.32 + 0.18
3072 1024 + 100
3072 1024 + 10/
PEPITA
100
45.89 + 0.16
18.32 + 0.18
3072 + 1024 + 100
SPELA B
3072 -> 1024
200
43.2 + 0.18
19.77 + 0.19
48.33 + 5.
3072 + 1024 + 10/
SPELA B
200
43.41 + 1.03
21.24 + 0.12
66.89 + 2.09
3072 + 1024 + 100
Table 7: Test accuracies(mean (mean standard deviation) comparison different learning methods on CIFAR 10, CIFAR 100, and SVHN 10 datascts. The accuracics of FA, DRTP, and PEPITA are presented Dellaferrera & Kreiman (2022). We report both hidden layer and output mean accuracies (average of five runs) of SPELA. For CIFAR 10 and SVHN 10, we 3072 1000 10 network, and for CIFAR 100, use 3072-1024-11 100 network.
16

---

Published in Transactions on Machine Learning Research (06/2025)
Datasct
a
b
C
f
g
i
53.19
85.31
92.09
95.56
95.73
95.55
95.38
94.46
91.74
MNIST
- 4.73
4.05
+ 3.87
+ 0.11
0.13
+ 0.08
= 0.07
= 0.11
0.22
35.66
69.15
73.67
80.23
79.98
79.92
78.96
75.63
68.44
KMNIST 10
+ 6.39
+ 3.38
+ 0.18
- 0.3
+ 0.18
= 0.36
= 0.42
0.89
50.31
75.23
84.88
86.82
86.8
86.66
85.92
84.33
Fashion MNIST 10
+ 6.45
+ 9.1
+ 1.37
+ 0.05
0.09
0.06
- 0.06
0.12
0.11
Table 8: Test accuracies (mean + standard deviation) of SPELA for varying output layer depth, keeping the total neurons in the network fixed to 1034. The columns denote network configurations a: 1032-2, 1029-5, 1024-10, d: 984-50, e: 934-100, f: 834-200, g: 634-400, h: 234-800, i: 34-1000. The SPELA B configuration is used throughout the experiments for 200 training epochs and five runs.
Model
# 1
#
#
3
#
#
# 6
#
#
# 9
BP
3.26
7.46
11.65
15.85
21.03
25.23
29.43
33.63
37.83
SPELA
3.30
7.54
11.78
16.02
21.24
25.48
29.72
33.96
38
Table 9: Model mimory(in MB) occupied by a BP model and an equivalent SPELA model for the number of hidden layers varying from 1-9 (cach of size 1024 neurons). The input and output layer sizes are 784 and 10, respectively.
B.2 Transfer Learning with SPELA
Backprop: Top
SPELA: Top
SPELA 5x: Top
Backprop: Top
SPELA: Top 5
SPELA Top
Accuracy (in %6
Accuracy (In %)
Accuracy (in %))
40
(b) Aircraft 100 dataset
(c) Food 101 dataset
(d) Flowers 102 datasct
Figure 8: Accuracy plots (after 200 epochs fine tuning) of SPELA, SPELA 5x and Backpropagation trained networks for train dataset size percentages of 1, 5, 10, 25, 50, 75, and 100 during transfer dearning(keeping the test dataset fixed). The solid lines denote the mean, and the shades denote the standard deviation of five simulation runs.
In these experiments, the networks are trained for 200 epochs with learning rate of 0.1, and analysis is done on six datasets: Aircraft 100, CIFAR 10, CIFAR 100, Flowers 102, Food 101, and Pets 37 datasets (the numbers denote the number of classes in that dataset). Note that most of these datasets have a large the experimental details. ResNet50 model pro-trained on ImageNet-
number of classes. (ablee2]dcscrt downlooddd from PyTorch Hub) extracts features from the layer before the classifier head. These features then train classifier head using backpropagation SPELA.
17

---

Published in Transactions on Machine Learning Research (06/2025)
Test size:
10
25
100
SPELA topl
58.79
19.33
67.28
4.50
64.03
6.47
60.67 + 5.10
72.38 1: 2.80
73.35 4: 3.17
69.32
8.29
SPELA topá
90.23
10.89
95.01 + 0.69
93.84
1.67
95.14
+ 1.55
95.68 + 0.52
96.53 + 0.20
94.78
1.70
SPELA 5x topl
75.64
0.54
78.64
0.53
75.30
4.97
81.21
0.16
80.88 1 0.73
82.11
0.32
82.04
0.50
SPELA 5x top5
97.81 + 0.28
98.13 + 0.24
97.84 + 0.33
98.36 + 0.16
98.22 + 0.34
98.43 + 0.05
98.20
0.24
BP topl
75.93
2.13
76.58
2.18
67.22
8.64
81.18
1: 0.80
80.47
1 1.87
81.57
0.27
81.18
0.77
BP top5
08.25
0.77
97
1.70
2.40
99.15
1: 0.07
99.06
0.18
0.12 99.12
99.12
0.11
Table 10: Test accuracy for CIFAR 10 dataset.
Test size:
10
50
75
100
SPELA topl
26.35 + 1.27
43.22
0.91
42.97
+ 2.25
51.14
1.37
51.97 + 0.92
54.52 + 0.54
54.35 + 1.03
SPELA top5
53.14 +
69.50
1.41
68.71
2.34
75.53
1.15
75.40
0.91
76.88 1 0.85
76.62
1.40
SPELA 5x topl
26.33 + 0.96
47.26 = 0.30
51.58
+ 0.26
55.81 = 0.09
57.63
+ 0.19
59.07 + 0.30
59.87
+ 0.26
SPELA 5x top5
55.74 + 0.97
77.97
0.31
81.04
+ 0.27
83.69
0.12
84.80
+ 0.18
85.57 + 0.06
85.95
0.30
BP topl
55.00 + 0.43
59.19
0.40
46.16
+ 2.86
61.18
0.09
60.79
0.48
61.42 + 0.20
61.08
0.13
BP topō
84.05
+ 0.32
86.77 = 0.25
81.79
+ 1.24
87.55
0.09
87.40 E 0.20
87.61
+ 0.07
87.46 + 0.04
Table 11: Tost accuracy for CIFAR 100 datasct.
Test size:
5
10
25
50
75
100
SPELA topl
66.54 1
76.86
1.57
78.60
2.61
82.93
1.99
83.48
1.33
84.94 1
84.60
1.46
SPELA top5
89.89 + 1.34
94.28
1.41
94.33 + 0.98
95.55 = 0.57
95.16 + 0.34
95.37 + 1.21
95.34 + 0.91
SPELA 5x topl
75.96 + 1.34
83.73
0.37
85.71 1 0.66
88,08
0.42
88.93
0.11
89.19 1 0.29
89.34
0.20
SPELA 5x top5
95.97 + 0.45
97.85
0.23
98.34 + 0.21
98.53 0.07
98.57
0.06
98.61 + 0.09
98.48
0.17
BP topl
86.71 + 0.82
88.64
0.44
88.99 + 0.16
89.28
0.23
89.26 + 0.08
89.44 + 0.25
89.29
0.10
BP top5
98.79 1 0.11
0.06
99.02 1 0.08
99.01
,9
0.04
98.90 1
0.04
Table 12: Test accuracy for Pets 37 dataset.
Test size:
50
100
SPELA topl
3.32 + 0.43
6.68 + 0.54
9.53 + 0.92
11.94 = 0.70
2.18 + 0.64
14.74 + 0.60
5.38
+ 1.67
SPELA top5
11.76 + 1.13
19.77
1.63
24.70
+ 1.60
27.55
1.15
8.51 + 1.87
31.08 - 0.94
16.72
3.14
SPELA 5x topl
1.97
0.41
5.47 = 0.53
9.39 + 0.49
17.10 = 0.72
20.06 + 0.87
24.11 1 0.45
24.09
0.64
SPELA 5x top5
9.00
0.57
17.83
0.67
25.31 + 0.33
38.72
0.98
42.92 0.94
47.39 + 1.01
46.61
0.83
BP topl
14.94 + 2.46
24.29
0.56
30.29
1 0.72
33.01 = 0.25
29.82
3.41
33.89 1
33.32
0.49
BP top5
37.38
+ 1.94
51.80 0.74
58.93
+ 0.43
62.26 0.20
61 22
+ 1.28
62.87 + 0.35
62.59
0.41
Table 13: Test accuracy for Aircraft 100 dataset.
Test size:
1
5
10
25
50
75
100
SPELA topl
26.09 i
39.43
42.28
1 1.24
49.93
0.73
52.48
1
52.65
0.96
SPELA top5
49.77 + 1.61
63.19 - 1.15
65.53
+ 0.93
71.99 = 1.25
72.06
+ 0.82
73.38
+ 0.83
73.32 + 0.91
SPELA 5x topl
27.77 + 0.45
44.89
0.36
49.50
1 0.23
53.41 0.23
56.39
1 0.05
58.10
+ 0.11
59.08
0.21
SPELA 5x top5
52.58 + 0.84
72.18
0.22
76.02
+ 0.30
79.15
0.21
80.98
0.22
82.32 - 0.13
82.96
0.16
BP topl
55.34 1 0.36
58.97 = 0.34
56.93 1 1.30
62.16
0.29
62.61
1 0.08
62.40
1 0.16
62.09 - 0.10
BP top5
80.95 + 0.26
83.66 0.24
82.96 + 0.72
85.49
0.07
85.71
0.09
85.51 - 0.16
85.21
0.09
Table 14: Test accuracy for Food 101 dataset.
18

---

Published in Transactions on Machine Learning Research (06/2025)
Test size:
50
SPELA topl
13.07 + 2.23
23.58 3.15
25.56 + 1.81
33.48 0.86
40.65 + 1.92
41.75 + 1.58
42.95 + 1.62
SPELA top5
32.28 +: 2.12
45.18 3.42
48.23 1: 1.62
56.06
1.22
61.84
1.38
61.68 1 2.01
62.51
1.29
SPELA 5x top1
12.52 + 1.24
16.65 1.38
16.82 + 1.20
38.65
0.88
55.37
1.08
61.29 + 0.80
64.82
0.82
SPELA 5x top5
28.89 + 2.37
36,27
1.77
36.65 1: 1.52
62.67 = 0.87
76.39
1 0.43
80.31
1 0.85
82.40
+ 1.31
BP topl
58.61
+ 4.33
75.62
0.73
77.24
+ 0.68
79.53
0.32
80.12
0.10
80.39
+ 0.07
80.34
0.27
BP top5
82.72 1 2,03
91.64 0.33
92.21
1 0.30
93.16 = 0.26
93.39 1 0.28
93.52 1 0.05
93.61
0.10
Table 15: Test accuracy for Flowers 102 dataset.
B.3
Extension of Ablation Studies
B.3.1 Relevance of Euclidean distance
The goal of learning in SPELA to learn to orient the activation vector h2 towards the correct class symmetric vector Ve. This akin reducing the angle between h; and ve, which turn increases the cosine similarity angular similarity. Hence, cosine is an appropriate similarity.
If, for instance, h4 100ve, then even though the cosine loss would be zero due to perfectly aligned vectors, the Euclidean distance would be high, and hence the corresponding loss would be high. We notice that in such a scenario, SPELA's 's performance drops significantly, SPELA-Euc in Table 16 But if we use normalized
Euclidean distance wherein the vectors are h and 8e that of cosine, HELA-Norm-EEE in Table
then the performance of SPELA closely matches 1
16
Model
Architecture
MNIST 10
KMNIST 10
Fashion MNIST 10
SPELA-Cor
784 + 1024
91.09 = 0.12
68.74 + 0.18
84
0.11
SPELA-Cos
784 + 1024 + 10
94.41 + 0.49
75.05 + 3.47
84.12 3.30
SPELA-Euc
784 > 1024
75.37 + 0.34
52.71 + 0.28
67.83 0.77
SPELA-ERS
784 + 1024 + 10
48.16 + 8.76
28.16 + 8.14
51.82 7.18
SPELA-Norm-He
784 + 1024
90.24 + 0.11
67.11 + 0.13
83.53 0.09
SPELA-Norm-Euc
784 + 1024 - 10
90.0 + 3.64
71.57 = 2.2
84.60 1.14
Table 16: Ablation study results of SPELA on MNIST 10, KMNIST 10, and FashionMNIST 10 datasets
after 200 training epochs (and for five runs). SPELA-Cos implies SPELA with cosine distance, SPELA- Euc implies SPELA with Euclidean distance, SPELA-Norm implies SPELA with Euclidean distance on normalized activation and symmetric vectors.
B.3.2 Remarks on randomizing the Vector Embeddings
Remark As we operate dimensions much higher than the number of embodded vectors (number of classes), a non-symmetric distribution should perform equivalent to symmetric structure. The performance gap between symmetric and non-symmetric structures would be noticeable when the number of dimensions exceeds the number of classes.
Remark 2:
We use the energy of the system as structured metric:
X(V) Σ Σ DEV VEV.
u
Of all possible vectors x e Rd, if V is fixed, the symmetric structure has the minimum energy. Comparing vectors drawn from the Gaussian distribution N0, 1) and the symmctric structure, we get high energy difference. Despite this, the network learns from the incoming data. A symmetric structure is necessary for lower dimensions (where the number of dimensions is comparable to the number of embedded vectors).
19

---

Published in Transactions on Machine Learning Research (06/2025)
We learn that although having vectors embedded in symmetric structure is optimal, it is unnecessary. The model will mold the weights according to the relative positions of the vectors and classify the data accordingly, which ascertains the model's flexibility.
Datasct
Architecture
0.01
0.1
1.5
2.5
3
90.27
90.97
89.72
91.26
91.01
90.97
MNIST 10
784 1024
+ 1.1
+ 0.12
+ 1.01
+ 0.05
+ 0.12
+ 0.06
91.86
92.92
86.93
93.31
94.25
92.31
MNIST 10
784 - 1024 + 10
+ 3.22
+ 1.63
+ 11.06
+ 1
+ 1.12
+ 3.28
65.74
68.54
64.82
67.9
68.48
68.6
KMNIST 10
784 + 1024
+ 2.77
+ 0.18
= 3.23
+ 0.5
0.11
+ 0.17
75.06
74.30
73.86
66.11
74.12
75.69
KMNIST 10
784 > 1024 + 10
2.47
+ 3.38
#3.81
+3.57
2.41
2.03
81.57
83.97
81.10
83.34
83.99
83.96
Fashion MNIST
784 + 1024
2.11
#0.0
= 2.35
+ 0.57
0.06
+ 0.05
83.12
85.34
81.32
83.65
84.52
81.6
Fashion MNIST 10
784 1024 10
+ 4.51
+ 0.21
+ 3.28
E 1.61
1.66
+ 4.99
Table 17: SPELA test accuracies (after 200 epochs of training and over five runs) for learning rates of 0.01,
0.1, 1, 1.5, 2.5, and 3.
B.4 Extension of SPELA Convolutional Neural Network
We next varied the channels in SPELA B CNN from 32 to 128 in SPELA C. CNN and observed that SPELA improves performance on CIFAR 10, CIFAR 100, well as SVHN 10 (Table 18
Model
CIFAR 10
CIFAR 100
SVHN
SPELA B CNN(2)
56.59 = 0.97
26.71 + 0.79
76.28 1 6.45
SPELA C CNN(2)
64.76 0.49
27.59 + 4.7
84.93 + 0.21
Table 18: Test accuracies (mean + standard deviation) comparison of different SPELA CNN architectures CIFAR 10, CIFAR 100, and SVHN 10 datasets. SPELA B and SPELA C indicate networks with 32 and 128 channels, respectively(PPee. (Please refer to Section C.4 for experimental details).
C Experimental Details
We used NVIDIA RTX 4500 Ada generation GPU for all our studies.
20

---

Published in Transactions on Machine Learning Research (06/2025)
C.1
How does SPELA work?
BP
CH A
SPELA CH A
SPELA A
Layer
1024
1024
1024
Layer 2
10
10
10
Learning rate
0.1
0.1
0.1
Decay rale
0.1
X 0.1
X 0.1
Decay epoch
60
60
60
Batch size
50
50
50
# Epochs
100
100
100
Dropout
10%
10%
10%
Weight inil
He Unilorm
He Uniform
He Unilorm
Bias
False
False
False
Optimizer
SGD
SGD
SGD
(momentum=0.9)
Activation
ReLU
ReLU
ReLU
Loss
Cross-Entropy
Cross- rotss-
Positive Cosine
Table 19: Experiment Details for SPELA MLP
BP CH B
SPELA CH
SPELA B
Layer
1024
1024
1024
Layer
10
10
10
Learning rate
2.5
2.5
2.5
Decay rate
0.1
0.1
0.1
Decay epoch
Every 10
Every 10
Every 10
Batch size
50
50
50
# Epochs
200
200
200
Dropout
0
0
Weight init
He Uniform
He Uniform
He Uniform
Bias
True
True
True
Optimizer
SGD
SGD
SGD
Activation
Leaky 0S00p)
Leaky ReLU(Slope=0.00)
Leaky ReLL(SSope=0.00
Loss
Cross- Pross-Entropy
Cross-Fatropy
Positive Cosine
Table 20: Experiment Details for SPELA MLP
21

---

Published in Transactions on Machine Learning Research (06/2025)
C.2 Transfer Learning with SPELA
BP
SPELA
SPELA 5x
Layer
#Classes
#Classos
# Classsss
Learning rale
0.1
0.1
0.1
Decay rate
0
0
0
Batch size
128
128
128
# Epochs
200
200
200
Dropout
0
0
0
Weight init
He Uniform
He Uniform
He Uniform
Bias
True
True
True
Optimizer
SGD
SGD
SGD
Loss
Cross-Entropy
Cross-Entropy
Table 21: Experiment Details of SPELA on Transfer Learning
C.3
Ablation Studies
BP
SPELA
Layer 1
1024
1024
Layer 2
10
10
Learning rate
2.5
2.5
Decay rate
0.1
0.1
Decay epoch
Every 10
Every 10
Batch size
50
50
# Epochs
200
200
Dropout
0
Weight init
He Uniform
He Uniform
Bias
True
True
Optimizer
SGD
SGD
Activation
Leaky ReloU(Slope00001)
Leaky ReLU(Slope=O.001)
Loss
Cross-Fntropy
Positive Cosine
Table 22: Experiment Details of SPELA MLP for Ablation Studies
22

---

Published in Transactions on Machine Learning Research (06/2025)
C.4 SPELA Convolutional Neural Network
SPELA CH B
SPELA B
Input size
32x32x3
32x32x3
Conv
(2)5,1
(2)5,1
MLP
10/100
10/100
Learning rate
0.1, 0.1
0.1, 0.1
Docay rate
Batch size
64
64
# Epochs
15+10
15+10
Dropout
0
0
Weight init.
Kaiming Uniform
Kaiming Uniform True
Bias
True
Optimizer
Adam
Adam
Activation
Leaky R.L0(S))pe=0.001
Leaky (eLo(
Loss
Cross-Entroph
Positive Cosine
Table 23: Experimental details of SPELA convolutional neural network.
23


Reviews:
[{"review_id": "VCB2reuPCy", "reviewer": "Reviewer_7RVP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nThe SPELA method does require less memory than traditional backpropagation and the authors demonstrate that it converges quickly. Overall, the paper is clearly written with clear explanations of how the method works and what experiments were performed to validate it.\n\n# Weaknesses\n\n## Major\n\nThe SPELA method mainly consists of two insights. First, that by attaching a classifier to every layer of the network, backpropagation can be replaced with the more memory-efficient individual finetuning of each layer. Second, that any layer can function as a classifier if the $d$-dimensional output space is partitioned for each class; the spectral partition that the authors propose seems like the most logical way to do so. Both of these insights have their own drawbacks, which they trade off for efficiency. The fact that every layer is trained for classification  means that at each layer the network must learn the most class-differentiating features, rather than possibly learning more important features that are not confined to one class (like the textures of fur or paint). Conversely, the spectral partitioning comes with several substantial drawbacks that should be discussed in the paper: \n\n* In cases where $d << N $ (the number of classes), the class vectors will have a spatial relationship, exactly as illustrated in Fig. 1(b), which biases the classifier toward certain kinds of errors.\n* The SPELA approach cannot be extended to multilabel classification, while the linear classifier approach trivially can.\n* Ordinary linear classification heads allow the outputs to be interpreted in terms of probabilities/confidences, and also allow for the application of interpretability methods; the SPELA method does not seem to allow this.\n\nIn cases where the number of classes is large relative to the number of dimensions, the first limitation disappears and this method may have applicability. However, in cases where the number of classes is small (e.g., CIFAR-10, MNIST), a linear classifier may instead be appended to each layer and used to train that layer (the output of the layers would be passed through to the next layer without going through this classifier). This would fix all of the objections above and still be fairly cheap and efficient to implement. I would encourage the authors to also compare against this approach for their target datasets.\n\n## Relatively minor\n* In section 3.1, I assume the fully-connected model also has nonlinearities, but this is not mentioned in the paper.\n* The authors mention that classification can be done at any layer of the network, but don't really address or use this fact, except in the case of convolutions.\n* In Table 3, why no results for KMNIST and Fashion MNIST for the baselines? Why is the architecture used for SPELA different from the one used for the baselines?\n* In Figure 6, it is more standard to keep the test set fixed even as the training data size increases, which would have given an easier-to-interpret comparison. If the authors wanted a larger test set, perhaps k-fold validation could have been used.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method, SPELA, meant for efficient learning for multiclass classification problems in resource-constrained environments. In effect, the method attaches a classifier to each layer of a fully connected neural network(*) that chooses the class with the lowest cosine distance to a fixed unit vector in $d$-dimensional space that represents this class; here, $d$ refers to the layer output dimension, and the class unit vectors are chosen so that they are maximally separated in the space (within that constraint, the class-vector assignments are random.) During finetuning, each layer can be optimized independently (fixing all previous layers), or jointly - in either case, the gradient is not propagated across layers, so there are substantial memory savings over conventional backpropagation.\n\nThe paper then extends this framework to convolutional layers. This is done by randomly splitting the classes between the convolutional kernels (so each kernel distinguishes between only some of the possible classes), and then a point system is used to pick the 'winning' class for the layer.\n\nFinally, the authors extend their method to transfer learning, where backpropagation is used to train the base model, and then SPELA is used to adapt the model to a new dataset.\n\nThe authors perform experiments and ablations to validate their method.\n\n(*) I assume that this is what the authors mean, but they omit any mention of a nonlinearity when describing the network architecture.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Please address the weaknesses described above. In particular, I suggest studying the two components of SPELA (early exit vs. spectral partitioning) separately by using a linear classifier at each layer, as described above, to better understand SPELA's design decisions and tradeoffs.", "broader_impact_concerns": "A broader impact statement is not required, but please address the method's limitations w/r/t interpretability.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "U7UL4hCNT9", "reviewer": "Reviewer_jjPs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. **Interesting Learning Paradigm**: The introduction of a forward-pass-only algorithm that bypasses backpropagation is interesting, particularly relevant for resource-constrained application. \n\n2. **Local Loss Functions**: The use of local loss functions for weight updates is an interesting method that can potentially address the inherent inefficiencies of global gradient backpropagation. \n\n3. **Symmetric Embedded Vectors**: The concept of generating symmetric vectors via a physics-inspired process to serve as class priors is interesting, as it can ensure uniform angular separation when using cosine similarity.\n\n4. **Potential for Edge AI**: The proposed method is designed for applications in on-device machine learning and edge AI, where computational and memory resources are limited.\n\nWeaknesses:\n\n1. **Lack of Detail in the Symmetric Vector**: One of the paper’s most important claims is that symmetric vectors are generated by simulating electron dynamics within an n-dimensional unit ball until electrostatic energy converges to a minimum. Especially considering that the definitions of these symmetric vectors are important to ensure that cosine similarity measurements can lead to well-defined decision boundaries and consistent classification performance. However, the authors provide no details regarding how this simulation is achieved to generate the symmetric vectors. The readers would like to know, for example, which computational method (e.g., iterative repulsion dynamics, or a Monte Carlo) is used to simulate the electrons’ movements; how convergence is determined or what thresholds are used.\n\n2. **Initialization and Iterative Process**: The implementation details, such as the initialization of the electron positions, step sizes, and the number of iterations required, are completely absent.  \n\n3. **Weight Update Strategy**: SPELA’s reliance on local loss functions for weight updates, which can bypass the backward pass, raises several concerns: (1) without a mechanism to ensure that local updates are coordinated across layers, there is a significant risk that the network’s overall objective is not properly optimized, which can potentially lead to suboptimal performance; (2) the paper does not address whether the forward-pass-only update method might lead to issues with stability or convergence, especially in deeper networks or more complex tasks. \n\n4. **Theoretical Justification**: The paper lacks a theoretical study to justify why these local updates should approximate or even match the performance of global backpropagation-based updates. \n\n5. **Insufficient Empirical Validation**: Although the authors provide experimental results on datasets such as CIFAR-10 and CIFAR-100, the presentation of these results is superficial. The lack of detailed ablation studies and sensitivity analyses regarding the effects of non-symmetric versus symmetric vector embeddings leaves a gap in understanding the robustness and scalability of SPELA. \n\n6. **Overall Clarity and Reproducibility**: The paper fails to provide sufficient implementation details. Without a clear description of the simulation process for generating symmetric vectors and a robust justification for the local update strategy, the work remains more of a conceptual idea than a concrete and deployable algorithm.\n\n7. The authors emphasize the benefits of SPELA for resource-limited applications, however, the experimental evaluation falls short in directly supporting these claims. Although the authors compare classification performance on datasets such as CIFAR-10, CIFAR-100, and SVHN 10, there is a lack of direct measurements or analyses regarding resource utilization. Specifically, the paper does not provide empirical evidence for reductions in memory footprint, computational time, or energy consumption, which are important metrics for validating claims of efficiency in edge AI and on-device learning scenarios. Without such direct evaluations, it is difficult to say whether SPELA can truly offer a resource-efficient alternative to backpropagation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Solo Pass Embedded Learning Algorithm (SPELA), a resource-efficient learning method that updates weights solely during the forward pass using local loss functions, thus can avoid the costly backward propagation of gradients and reducing memory and computational demands. SPELA uses the symmetric embedded vectors, which are generated by simulating electron dynamics in an task-oriented dimensional unit ball until the system’s electrostatic energy converges to a minimum, to serve as priors that ensure uniform angular separation and balanced decision boundaries when cosine similarity is used for classification. The authors demonstrates competitive performance on benchmark datasets (CIFAR-10, CIFAR-100, SVHN 10) and shows promise for deployment in edge AI and on-device learning scenarios by significantly lowering data, computation, and energy requirements while remaining adaptable to various network architectures. However, there is no experimental validation on how the proposed learning algorithm can lower data, computation, and energy requirements.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Detailed Description of Symmetric Vector Generation**: Please include a comprehensive discussion on how the symmetric vectors are generated. Clarify the simulation process for electron dynamics, the update mechanism, and the convergence criteria for minimizing the energy function.\n\n2. **Empirical Evaluation of Resource Efficiency**: Please include an experimental section with direct measurements of memory usage, computational time, and energy consumption, especially under resource-constrained conditions (e.g., on-device experiments).\n\n3. **In-Depth Theoretical Analysis of Local Loss Convergence**: Please provide a theoretical analysis of the convergence properties of the local loss functions. Discuss potential issues regarding alignment with the global objective and include any relevant mathematical derivations or proofs. \n\n4. **Expanded Ablation Studies**: Please include additional ablation studies to evaluate the impact of different vector initializations (e.g., symmetric versus Gaussian), learning rates, and network depths. This should help to understand the robustness of the SPELA algorithm. \n\n5. **Clarification of Implementation Details**: Please provide a detailed discussion on the implementation parameters (e.g., hyperparameters, batch sizes, training protocols) for all experiments. Supplementary material or an appendix could be used if space is a concern. \n\n6. **More Comparative Analysis**. Please include comparisons with additional baseline methods, particularly those designed for resource-constrained application or edge computing, to contextualize SPELA’s performance. Comparisons should consider both performance and computational analysis.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4OeyVZyLJu", "reviewer": "Reviewer_UQGC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed algorithm addresses a significant need for efficient learning in resource-constrained environments.\n\n2. SPELA's ability to learn with only a single forward pass represents a notable departure from traditional methods.\n\n3. The empirical results showing strong performance in few-shot learning scenarios are impressive.\n \nWeaknesses:\n1. The paper presents a confusing narrative by introducing both SPELA and SPELA(O), with the introduction primarily focusing on SPELA while the experiments exclusively use SPELA(O).\n2. There is a lack of theoretical analysis explaining why cosine similarity as a loss function works effectively for this learning paradigm.\n3. The experimental comparisons use inconsistent network architectures between SPELA(O) and backpropagation (e.g., 784-1000-34 vs. 784-1024-10), undermining the reliability of direct performance comparisons.\n4. The paper does not adequately compare the number of training epochs required across different methods, making it difficult to assess true computational efficiency.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces SPELA (Solo Pass Embedded Learning Algorithm), a backpropagation-free learning algorithm designed for resource-constrained environments. SPELA operates using a single forward pass and local loss functions, eliminating the need for backward passes and the storage of computational graphs. The authors present two variants: the original SPELA and an optimized version called SPELA(O) that reduces computational complexity from O(n²) to O(n).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Streamline the presentation of the algorithm by focusing on SPELA(O) throughout the paper. The current distinction between SPELA and SPELA(O) is confusing, especially since experiments only use SPELA(O). Consider removing references to the original SPELA or clearly explaining the historical development and why SPELA(O) supersedes it.\n2. Provide theoretical analysis or justification for why cosine similarity loss is effective for this learning paradigm. This should include mathematical derivation of how this approach relates to traditional gradient-based learning.\n3. Use consistent network architectures when comparing SPELA(O) with backpropagation and other methods. The current use of different architectures (e.g., 784-1000-34 vs. 784-1024-10) undermines the validity of performance comparisons.\n4. Add epoch count information to Tables 1, 3, and 4 to properly compare the computational efficiency of different methods. Without this information, it's difficult to assess the true advantage of SPELA(O) in terms of training time.", "broader_impact_concerns": "No Broader Impact Concerns Section presented in the paper", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript is still not consistently making proper use of the \citep and \citet commands. For example, the text block on the first page that currently reads

Previous studies indicate that prior knowledge helps one learn faster and more easily Goyal & Bengio (2022); Wang & Wu (2023). Although we do not claim complete biological plausibility, SPELA is built on the premise that biological neural networks utilize local learning Illing et al. (2021) and neural priors to form representations.

should read

Previous studies indicate that prior knowledge helps one learn faster and more easily (Goyal & Bengio (2022); Wang & Wu (2023)). Although we do not claim complete biological plausibility, SPELA is built on the premise that biological neural networks utilize local learning (Illing et al. (2021)) and neural priors to form representations.

The authors need to go back through the LaTeX source to ensure correct use of \citet and \citep
Decision: Accept (poster)