Metamorphic Forward Adaptation Network: Dynamically Adaptive and Modular Multi-layer Learning
Authors: Yu Sun
Vijja Wichitwechkarn
Ronald Clark
Mirko Kovac
Basaran Bahadir Kocer
Venue: Accepted by TMLR ()

Abstract: Back-propagation is a widely used algorithm for training neural networks by adjusting weights based on error gradients. However, back-propagation is biologically implausible with global derivative computation and lacks robustness in long-term dynamic learning. A previously proposed alternative to back-propagation is the Forward-Forward algorithm, which bypasses global gradient dependency and localises computations, making it a more biologically plausible approach. However, Forward-Forward has been evaluated in limited environments, does not yet match back-propagation's performance, and only supports classification, not regression. This research introduces the Metamorphic Forward Adaptation Network (MFAN), using a contrastive learning property as its core, and retaining the layer-wise architecture of the Forward-Forward algorithm. Compared to the Forward-Forward model being limited to discrete classification, MFAN can process discrete and continuous data, showing stability, adaptability, and the ability to handle evolving data. MFAN performs well in continuous data stream scenarios, demonstrating superior adaptability and robustness compared to back-propagation, particularly in tasks requiring dynamic, long-term learning.

Published in Transactions on Machine Learning Research (4/2025)
Metamorphic Forward Adaptation Network: Dynamically Adaptive and Modular Multi-layer Learning
Yu Sun
sofia.sun@berreley..dd
Deportment of Computing, Imperial College London
Vijja Wichitwechkarn
vw273 uk
Department of Engineering, University of Cambridge
Ronald Clark
ronald.ctark@es.orr uk
Department of Computer Science, University of Oxford
Mirko Kovac1.2.3
mirko, KarkkkokoaaBeempa..
Laboratory of Sustainability Robolics, Swiss Federal Laboratories for Materials Science and Technology (Empa) Laboratory of Sustainability Robolics, École Polgtechnique Fédérale de Lausannne (EPPL) Aerial Robotics Laboratory, Department of Aeronautics, Imperial College London
Basaran Bahadir Kocer
koccer@rristol.acccc uk
School of Civil, Aerospace and Design Engineering, University of Bristol
Reviewed on OpenReview: https: // openreview. net/ forum? id= htt.:///lllllccgggoooooseseseeee
Abstract
pack-propaggtion is a widely used algorithm for training neural networks by adjusting weights based on error gradients. However, prok-propag biologically implausible with global derivative computation and lacks robustness in long-term dynamic learning. A previously proposed alternative to back-propagation the Forward-Forward algorithm, which bypasses global gradient dependency and localises computations, making it more biologically plausible approach. However, Forward-FFrwwer has been evaluated in limited environments, docs not yet match back-propagation's 's performance, and only supports clas- sification, not regression. This research introduces the Metamorphic Forward Adaptation Network (MFAN), using a contrastive learning property as its core, and retaining the layer- wise architocture of the forward-Forwand algorithm. Compared model being limited discrete classification, MFAN can process discrete and continuous data, showing stability, adaptability, and the ability to handle evolving data. MFAN per- forms well in continuous data stream scenarios, demonstrating superior adaptability and robustness compared to back-prop,,,tion, particularly in tasks requiring dynamic, long- term learning.
the Forward-Forward
1 Introduction
Deep learning has achieved remarkable success across numerous applications, primarily due the efficiency of the back-propagation (BP) algorithm in training deep neural networks. BP has been the foundation for advancements in fields ranging from image recognition to natural language processing. However, as these models are increasingly deployed in dynamic environments, such as adaptive systems and continuous data stream scenarios, the limitations of BP become more apparent.
One notable limitation of BP the weight symmetry requirement (the weight transport problem), which necessitates that the weights for forward and backward propagation remain identical throughout the training process [Bengio et al. 2015). Additionally, BP requires distinct inference and learning phases, with error sig-

---

Published in Transactions on Machine Learning Research (4/2025)
nals propagated backward layer by layer. This process depends on globally synchronised updates and precise knowledge of activation function derivatives, both of which are biologically implausible. Alternatives such as Feedback Alignment (FA) and Direct Feedback Alignment (DFA) address these constraints by replacing exact weight symmetry with fixed random feedback weights or direct error signals by offering mechanisms that align more closely with biological plausibility (Nokland 2016). Furthermore, recent. studies of wide neural networks and learning rules in the Neural Tangent Kernel (NTK) regime show that simpler methods can achieve comparable performance by leveraging local error-weighted input correlations instead of relying global feedback synchronisation (Boopathy & Fiete 2022). For more detailed discussion of biologically plausible methods, readers are referred to (Lx et al. 2024).
Another significant shortcoming of BP is its limitations in real-time applications requiring sustained plasticity (Dohare et al. 2024). BP relies on precise knowledge of forward-pass computations to calculate accurate gradients, which can result in ineffective gradient updates in scenarios where underlying data distributions are constantly changing. This dependence on static computations hinders BP's ability to adapt to dynamic environments, as gradient updates derived from batch of training data may fail to generalize effectively to evolving tasks. Consequently, BP-ttriined models often struggle to adapt or generalize in such settings.
As a solution, adaptive neural networks (Ashfahani & Pratama 2019/, which dynamically modify their architecture by expanding or contracting based on task complexity, have demonstrated superior performance in real-time settings (Kocer et al. 202Ip. For example, tasks involving continuously streaming data demand immediate adaptation and stable predictions, where adaptive models have proven to be more effective. While there is some evidence of the backward flow of error signals in biological systems, which is central to BP (Francioni et al. 2023), , neurons in the brain are believed to operate primarily through local updates rather than the global error propagation characteristic of BP. This disconnect has driven interest in exploring alternative learning methods (Karimi et al. 2024 Lv et al. 2024), such as the Forward-Forward (FF)
algorithm (Hinton 2022).
The FF algorithm is a greedy multi-layer learning procedure inspired by Boltzmann machines (Hinton et al. 1986) and Noise Contrastive Estimation (Gutmann & Hyvärinen 2010).. FF replaces BP's forward-backward passes with two forward passes one with positive (real) data and one with negative (generated or corrupted) data. Each layer of FF has its local objective function aiming to have high goodness for positive data and low goodness for negative data through weight adjustments in every hidden layer, as illustrated in Fig. D.
An example of a goodness measure is the sum of the squares of the activities of the rectified linear neurons in each layer, aiming to learn to make the positive data goodness above a threshold @ and the negative data below. Thus FF can classify input vectors by computing the probability that an input vector is positive. This is achieved by applying the logistic function, a, to the difference of the goodness and 0,
p(positive) = a ( EEE -
where by is the activity of hidden unit j before layer normalization.
While BP requires fully differentiable forward process to propagate learning signals throughout the network, FF can free this constraint by having each layer learn with locally generated loss signals.
Even though FF provides a more modular, layer-wise learning approach, it tends to be slower, particularly when applied to large-scale tasks, and its reliance on local optimisation can result in sub-optimal coordination across layers, potentially reducing overall performance in larger networks.
Researchers have explored several enhancements to the FF algorithm, including hyper- paper-paraeeter optimisa- tion and hybrid approaches Gandhi et al., 2023 Izzo et al. 2024 Kumar et al.) Reyes-Angulo & Pahoding 2024 Wang et al. 2024). One study [Gandhi et al. 2023) introduced a pyra-
2023 Ahamed et al.) 2023
midal strategy for adjusting the loss threshold progressively across layers, enhancing performance on tasks beyond MNIST classification, including sentiment analysis on the IMDb dataset. Additionally, integrating the FF algorithm with BP (Tzzo et al.) (0244) has shown promising results, combining the robustness of BP with the biological plausibility of FF. These hybrid models have demonstrated improved noise resilience and adaptability, especially in environments where task complexity evolves.
2

---

Published in Transactions on Machine Learning Research (4/2025)
In light of the efforts and limitations outlined above, research identifies additional constraint of the FF model that has yet to be addressed. The FF model's design combines both the feature vector (x) and the target variable (y) as inputs. During inference, this requires knowledge of all possible discrete values to combine with the test I to create valid test inputs. The model then computes goodness values for these inputs and selects the y prediction corresponding to the highest goodness value. While this design facilitates classification, poses challenges for regression tasks.
To address these limitations, this paper proposes the Mctamorphic Forward Adaptation Network (MFAN), novel architecture using contrastive learning property as its core, and utilising layer-wise architecture similar to FF. MFAN extends FF's algorithm to the continuous data domain, supporting both regression and classification.
2
The Proposed Model
(a) Forward- Algorithm
(b) Per-layer Latent Space T-SNE Plots
Layero
Layer; Layer;
2
+ Yypos = T pos
$ Zy
+ Ynne = I' neg
x
0
2 1+1
2
Zzng 0
Pane
nog
= "rpos
.i41
2xpos
"I pos
Learning the difference between Tyas and Tong
(c) MFAN: Metamorphic Forward Adaptation Network
Encoder
Trained Encoder
E
fx
fx (trained)
.
20
c
R By i+1 pos
ypos
2 "y pos
2x
ypos "g.
fy
Decoder
Layer Selector
Yneg
Classification
Regression
0 y pos
21 y pos
2 pos
++1
fy
Vpred (best)
i+1 Zy neg
D neg
Vnng
ppred
pred
pred
Learning Upas, Yneg relationship
Learning reverse mapping
Figure 1: a) Schematic for the Forward-Forwald (FF) model. The FF model combines x data and y labels into "good' and "bad' data inputs. Each layer's 's loss function maximises/ maximisess/iiiiisss. minimises goodness for 'good' aall'/"bal /bad' data. At inference, all labels are used, and the label with the highest goodness is taken as the output. Hence, FF can not be applied to continuons data, such as regression. (b) Latent space embedding for each MFAN layer in MNIST classification tasks. c) Schematic for the proposed approach: MFAN consists of input encoder fay output encoder fy and docoder fz' The encoder maps the input data into the same latent space, and the Cosine Similarity contrastive loss is used to pull positive samples of and y logether and push negative samples apart. The decoder maps from the latent space to the predicted output. Cross-entron entropy or Root Mean Square loss can be used to learn the reverse mapping. The effective layer selector selects the layer that makes the final prediction.
3

---

Published in Transactions on Machine Learning Research (4/2025)
The first improvement in the design of MFAN and the FF algorithm is data processing. Negative y (Yneg) is generated from positive y (Upos) by either choosing the wrong label (classification) introducing random noise offisess (regression). MFAN no longer uses combined Ipos Fneg as input, and instead separately encodes data and its positive and negative labels into the same d-dimensional latent space with an input encoder fx: and an putput encoder Jy
Zx. (x(
(2)
Opon fg (ppos)
(3)
Yong fy (Yueg)
(4)
obtaining Eas Ayos Fynex E Rdd
Then, replacing FF's goodness measure, each MFAN layer independently optimises a contrastive loss using cosine similarity (Lahitani et al. 2016). Cosine similarity is metric used to mcasure the similarity between two vectors by calculating the cosine of the angle between them,
sim( 02)
01
(5)
1 1/2
As cosine similarity measures the alignment between vectors, the contrastive loss function uses this property to encourage MFAN to learn to pull 2n and zypoe together and push 22 and Eynes apart in latent space Rd
Zz Ypon
2z Yonn
Leoe sim
Zz
+ Cr1
||
Zyyne
022
|Zypoo
[2yong
(6)
where dn and 02 are tunable parameters, the last term of this loss aims to retain the distance between positive and negative data mappings.
Using this loss function, MFAN encoders first trained together obtain mappings from the input/ coout//uuput space to the latent space, pulling 2 Aype The encoder parameters then frozen and used to obtain (v, Zy) pairs. These are used to train a decoder f which learns the reverse mapping of 2pper back to ypos- As the loss minimisation of encoder maps žn and Eyp.. together through training as shown in Equation then this condition also expected to hold in the test set at inference.
0test (fa gy (Zuren)
(7)
The input encoder and decoder are used together to make predictions as detailed in Algorithm 1 Thus, MFAN can predict without knowing all possible y, hence being able process continuous data and do regression tasks that FF is unable to perform.
Each layer in MFAN makes prediction, and effective layer selector determines which layer to use for the final output as detailed in Algorithm 2 For classification tasks, the layer with the best training performance is used during inference, or multiple top-performing layers can be averaged for smoother and more stable predictions. For regression tasks, the selector identifies the best-performing layer during training and uses deoper layer at inference to address task complexity and extrapolation. With a continuous data stream, the model dynamically adjusts depth by finding the berf-performing layer from the previous time step and using that layer at the subsequent inference time. The proposed approach is illustrated in Fig.

---

Published in Transactions on Machine Learning Research (4/2025)
Algorithm MFAN Pseudo-code (Encoder & Decoder)
1: Input: digits 2, correct labels Upas: wrong labels Uneg
2: for Encoder layer Li do
fz (2
Youu -- (2)-1
5: Yoog t ( (2--1
Yueg
6:
Lios Limm_sim sim
1/ I1
=
Spon
+ 011
22 2
kneg Smeg
+ 012
[1
Pppos
- Upon
Voos
yneg
>> Compute layer-wise cosine-similarity loss
Local weight update Apply normalization and ReLU activation to 2 2//ow Xymex pass to the next layer end for
w¹ + w¹ 17 V wi (ooss
sim
10: for Decoder layer Li do
11:
1 g+ g (z+)
b Map intermediate digit representation back to label space
12: LCE(U, ypos) = Ec Upos.c log (0c)
Compute cross-entropy loss Local weight update
13: w¹ +- w² 17 Vioi (COE)
14: end for
15: Output: Predicted labels
Algorithm 2 MEAN Pseudo-code: Layer Selector
Input:
(1, Lo,.,,,.,,, LN): Trained MEAN layers
Dirain: Training (or valiclaiion) dataset
M: Performance metric (e.g., accuracy or MSE)
2: Initialize: perffi] +- 0, i 1,...,,
3: for each epoch do
4: for i= to N do
perfi += M Derain) end for end for
it f- arg (prrf])
Output: 1*
3 Experiments and Evaluations
3.1 Classification Evaluations
This section evaluates the ability of MEAN to handle standard classification tasks and compares its perfor- mance the FF and BP models. While FF and BP have been extensively tested on classification tasks prior research, with BP consistently outperforming FF, MFAN was primarily designed to address different limitation of FF: its unsuitability for regression tasks. No prior work has applied FF to regression problems to the bost of our knowledge, as it has only been evaluated on classification tasks. Here, we aim to demon- strate that MFAN can also perform classification tasks, doing better than FF but still falling short of BP's performance. These results help establish a baseline for MFAN's classification capabilities before focusing on its primary advantages in regression tasks in dynamic environments.
3.1.1 MNIST Classification: Performance on Digit Classification
The MNIST dataset was used evaluate MFAN on simpler classification task and compare perfor- mance FF and BP. The configurations of all models were designed for comparability rather than peak optimisation. The model setup configurations given in Appendix A.1.I
5

---

Published in Transactions on Machine Learning Research (4/2025)
MFAN, FF, and BP employ fundamentally different learning mechanisms. BP learns the relationship between MNIST digits and their labels, FF distinguishes positive and negative data within each layer based on a unique processing structure, while MFAN not only learns the digit-label relationship but also captures why a digit is not associated with other labels through its contrastive learning framework. These differences influence computational demands and the resulting test accuracy: BP achieved the best performance in terms of both accuracy and training efficiency, with an average test accuracy of 94. +4.36%=00.44 and a training time of 5 seconds. MFAN, while trailing BP by 2.71% in accuracy (91.65% + 0.20%), demonstrated more consistent performance, albeit with a longer training time of 33 seconds due to its more complex computations. FF struggled significantly in its default setup, achieving only 45% test accuracy after 42 seconds of training. To reach 90.5% + 1.42%, FF required much wider networks with nearly 10 times more iterations, extending its training time to 375 seconds. These results highlight. MFAN's ability to outperform FF in terms of both accuracy and stability, while still falling short of BP in this classification task.
These results highlight MFAN's ability to learn meaningful representations despite its lower accuracy com- pared to BP. This is further supported by I-SNE visualisation of the latent space learned by MFAN (Fig, [p), which revealed well-defined clusters of digit labels, demonstrating each of its layers can capture scmantically meaningful features.
3.1.2 CIFAR-10 Classification: Performance on Image Classification
The CIFAR-10 dataset was used to evaluate MFAN, BP, and FF on more challenging image classification task. Fully connected architoctures were employed rather than convolutional layers for comparison purposes and to maintain consistency with other tests in the paper. Previous research ((Lin et al.) 2015 has shown that pure BP models achieve around 51 51% test accuracy on CIFAR-10 with fully connected layers, providing a key reference point for our comparisons. The model setup configurations are given in Appendix A.1.2
In the short training-time setup, BP again demonstrated superior performance, achieving an average test accuracy of 50.72% + 0.21% with a training time of 21 seconds. MFAN reached 37.47% + 1.38% test accuracy in 25 seconds, while FF achieved only 13.41% + 2.92% after 30 seconds of training. However, with wider network architectures keeping the same 3-layer, short-training selup but scaling each layer to 10 times the number of nodes, from hundreds to thousands and longer training times, both FF and MFAN improved, achieving test accuracies reaching 46%. Nonetheless, their performance is still inferior to that of the BP model, with also 10 times more nodes, which achieved 49% test accuracy. In all these conditions, we compared them without additional tuning and only increased the network width by adding 10 x more nodes to each layer. In such a case, we observed overfitting in BP and some performance increases as discussed for FF and MFAN.
It is worth noting that while MFAN and FF models improved their accuracy with 10 times wider layers, they came at the cost of extended training times, leading to total of approximately 30 minutes for both MFAN and FF models. This made further optimisation impractical and not worthwhile. In contrast, the BP model with 10 times more nodes required only around 2 minutes for training, underscoring the relative time inefficiency of the FF and MFAN models in vision tasks like CIFAR-10 classification as compared to BP. Overall, BP remained the most efficient and accurate model for this classification task.
Table 1: Classification results: Test accuracy.
Dataset
MFAN
BP
FF1
MNIST
91.65% + 0.20%
94.36% + 0.46%
90.50% + 1.42%
CIFAR-10
37.47% + 1.38%
50.72% + 0.21%
13.41% + 2.92%
'The best performance with short training time.
Discussion: Moving Beyond Classification
The results from the MNIST and CIFAR-10 evaluations confirm that MFAN is capable of performing classi- fication tasks and outperforms FF in this domain. However, it falls short of BP, which remains more efficient and better suited for standard classification benchmarks. This outcome consistent, with MFAN's design
6

---

Published in Transactions on Machine Learning Research (4/2025)
focus, it was developed primarily to address regression and dynamic data challenges where FF and BP have limitations.
Unlike BP and MFAN, FF cannot perform regression tasks or handle continuous data. Its design, cen- tred on measuring goodness for each labelled data, inherently limits its ability to model smooth functional relationships required for regression. These design constraints make FF unsuitable for tasks involving contin- uous data streams or dynamic environments, where adaptability and precise representation of input-output relationships are critical.
MFAN, on the other hand, is specifically designed to address these challenges. Its layer-wise contrastive learning framework, dynamic depth adjustment, and robustness to noise make it better suited for handling short, dynamic data streams and evolving data distributions. The following sections will focus on MFAN's primary strength: its ability to perform regression tasks in dynamic environments and its advantages over BP and FF in these scenarios.
3.2 Function Regression: Continuous Data Learning
3.2.1
Evaluation Setup
MFAN Per-Layer
Expected Extrapolation
Outcome Comparisons
Effective 12
Predictions
Elfective
Eifecttvv
Offectiive 15
f (Xire
Train
MFAN
Predict
f (x.)
100 Ti
200 210
Positive
Negative
Expected Extrapelation
Expected
100 110
200 210
MEAN
BP Model
Figure 2: Schematic for function regression extrapolation task procedure. Positive data pairs, represented by the thin blue line, consist of x Rd yposs fx) E Rd where f the target function and d 100. Negative data pairs, represented by the thin orange line, consist of x E Rdd ynog = Upos Rd where F random offset. The expected extrapolation data (Irett Stest) E (IRd Rd) in shown by the thick blue line. Some MFAN and BP performance comparison plots are shown.
We evaluate MFAN on continuous data using series of trigonometric functions as regression targets (Hay Sharon 2024 Xu et al., 2021). We use a thrco-layer BP model as the bascline for comparison, and we detail the model configuration in Appendix [..2] To explore the adaptive layer selection mechanism of MFAN, we implement six- six-layer MFAN architecture while also evaluating both models with the for fairness, as detailed in Appendix A..21 The six-layer setup allows MFAN to demonstrate dynamic depth adjustment, which is not fully observable in the three-layer configuration, as it can only shrink the depth but cannot expand it. In fact, MFAN uses three layers most frequently during inference, similar to BP, but dynamic depth adjustment enables more flexible predictions.
number of layers
The experimental setup involves constructing datasets using sliding window approach. Specifically, the f(x) RRd dd 100, and
training dataset spans the domain 110,, with data points IRd Upos

---

Published in Transactions on Machine Learning Research (4/2025)
21+1 2; 0.1i with To = Ō, Z, and i [1, 100). Negative data Yueg are generated by adding random offsets to Upos: creating Vneg = Upos + F. The test dataset spans the domain [100, 210] and evaluates extrapo- lation performance. The training and evaluation processes are repeated to obtain average results, which are analysed to assess both accuracy and stability across all evaluations. Figure 2 visualises the first and last sets of data pairs for the training and lesting datasets.
3.2.2 Results & Analysis
Observations show that MFAN dynamically chooses output layers during inference, adjusting the network depth based on task requirements. On average, MFAN uses three layers during inference, matching BP's fixed-depth setup. However, when configured with six layers, MFAN demonstrated the ability to expand or shrink its effective depth, providing more prediction choices than BP. This flexibility is particularly evident in more complex regression tasks, where deeper layers are utilised for extrapolation to achieve smoother and more accurate predictions. Conversely, BP's fixed architecture limits its adaptability, resulting in less accu- rate and less stable prediction plots. Both models exhibited comparable inference times, averaging between 0.03 and 0.06 seconds per evaluation, indicating that MFAN's dynamic architecture does not compromise prediction efficiency.
Sensitivity analysis further highlights MFAN's robustness. We introduced Gaussian noise with various stan- dard deviation levels (ranging from 0.5 to 20) during training. With small standard deviations (0.5 to 5), MFAN's error rate is observed to increase by around 2%, while BP's error rate rises by 8%. With high noise levels (standard deviation range from 5 to 20), MFAN maintained relatively steady performance, with a 6% error rate increase, whereas BP's error rate rose rapidly, leading to over 400% error rate increase and poor predictions overtaken by the added noise. This superior noise tolerance is likely due to MFAN's embedded noise desigu, where the model intentionally introduces noise in yneg during training and learns to distinguish it from (z, ypos) pairs. This mechanism enhances MFAN's ability handle noisy data more effectively than BP.
Table 2ileepsee summarises the regression performance metrics, presenting the average percentage errors and stan- dard deviations for both MFAN and BP across these function regression tasks. MFAN has much lower standard deviation values and has not been observed to fail any regression tasks in this experiment. The average percentage root mean square errors (RMSE) of MFAN are also lower than BP. These results demon- strated that MFAN consistently outperformed BP stability and accuracy, emphasising its ability to handle complex, nonlinear functional relationships and extrapolate effectively to unseen domains.
These findings highlight MFAN's robustness in handling continuous data, particularly in scenarios where the relationship between input and output variables is complex and nonlinear. Its combination of dynamic depth selection, contrastive learning principles, and layer-wise weight updates makes MFAN a powerful tool for function regression (asks. Furthermore, MFAN's demonstrated resilience to noise and adaptive archilecture further validates its suitability for scenarios requiring high stability, accuracy, and the ability to handle variability effectively.
Table 2: Regression results: Percentage mean + standard deviation.
Regression f(x)
MFAN² error
BP error
f(x) = sin
06.43% + 0.22%
10.59% 04.02%
f(x) cos(0.2z)
06.41% 0.35%
21.16% + 10.11%
f(x) sin (0.4 cos (0.2x))
07.09% = 1.09%
19.63% 12.13%
f(x) $0.4 cos (0.2r) c0.(0.2 + 30)
10.39% + 0.07%
13.48% 00.56%
f(x) sin (0.8 cos (0.2z) + 20)
09.03% + 0.11%
18.51% = 11.68%!
f(x) = cos sin (0.2x) + 3)
12.26% + 2.12%
13.23% = 02.30%
'High standard deviation values in BP results indicate unstable predictions, leading significant performance fluctuations. 2MFAN's stability and lower error rates consistently outperform BP across all evaluations.
8

---

Published in Transactions on Machine Learning Research (4/2025)
3.3
Continuous Data Stream: Adapting to Dynamic Environments
3.3.1
Evaluation setup
a
MFAN
Train
MFAN
MFAN
Evaluate
BP
BP
Train data segment data segment
Ti (x)
Task stream
T1
T2
T3
T4
T5
T6
W
b
100
200
300
400
5000
600
BP prediction
Expected
BP
Predictions
MFAN prediction
Expocted
MFAN 4AA
Figure 3: a) The evaluation setup for continuous data stream: training and evaluation process involves six sinusoidal task variations (T)), each spanning domain of length 30q. These tasks are concatenated continuously to form smooth task stream, divided evenly into 3000 points along the x-axis. Data segments are constructed using a sliding window approach, with each segment representing a vector in R100 The model trained on five consecutive data segments and tested on the subsequent segment. This procedure is repeated iteratively until all points in the task stream have been processed. b) Prediction plots: the predictions from BP and MFAN models during the evaluation phases are plotted against the expected values for the three task streams.
This set of experiments evaluates MFAN's performance in continuous data stream tasks with model config- uration details in Appendix A.3 where the model is required to learn and adapt to a sequence of evolving trigonometric functions (Fig. 3 These tasks simulate dynamic environments where the data distribution changes over time, forcing the model to continuously update its knowledge and adapt to new, unseen data. We use six tasks, each represented by function defined in domain of length 30x. These tasks con- catenated continuously to form smooth task stream (TS), which is then divided into 3000 evenly spaced points (p:). These points are grouped into 100-dimensional data segments using sliding window approach: dala; = P100] datag = [P2 Pro,)), and so on until the final segment. datag291 "1e901...,,PP000
The training and evaluation process structured such that the model trained on five consecutive segments and then testod on the next segment in the sequence. This procedure repeated iteratively until all data points in the task stream have been processed. For the schematic of the continuous data stream evaluation process, refer to Fig. 3. Three task streams (TS1, TS2, and TS3) were designed to evaluate MFAN's ability to handle task transitions, adapt to unseen data, process complex patterns, and reorganise after interruptions.
3.3.2 Results & Analysis
Sinusoidal Function Series (TS1)
9

---

Published in Transactions on Machine Learning Research (4/2025)
TS1 involved sinusoidal functions with varying frequency and amplitude, testing the model's adaptation ability to the smooth and nonlinear changes (Fig. -ST).. MFAN initially learns the first task slowly but adapts well to task transitions and new data, producing stable and accurate predictions. In contrast, BP learns the first task quickly but struggles to adapt to subsequent tasks. The error rate plots (Fig. indicate that MFAN Trequently adjusted its layers at the beginning of TS1. Per-layer error plots ii... + suggest that at the start, its shallow layer (I1) had random predictions with very high errors, while its deeper layers (L2, L3) produced conservative predictions due to the lack of meaningful directional information from shallow layers. As L1 learned discriminating features, it makes better predictions than deeper layers, which seems to overfit. The effective layer selector in MFAN is adjusted to select the best-performing layer during training, thereby reducing error rates and stabilising performance. In contrast, BP showed sharp increases in error and fluctuations, indicating poor adaptation.
4 TST)
In terms of computational performance, both MFAN and BP demonstrated impressive efficiency. The train- ing and inference times for both models were in the millisccond range (Table A5). with MFAN's training taking slightly longer than BP's, but still within very fast timeframe. Despite this, MFAN's superior abil- ity to adapt to evolving tasks and maintain stability across long-term learning cycles presents significant advantage in dynamic environments.
Trigonometric Function Series (TS2)
TS2 featured a broader range of trigonometric functions, requiring the model to capture more complex relationships and more intricate patterns (Fig. 4 4- TS2).. MFAN maintained a stable error rate throughout, efficiently handling the increased pattern complexity and variety. In contrast, BP's error rates became increasingly unstable during task transactions (Fig. D-TS2).
Linear and Trigonometric Function Series (TS3)
TS3 introduced a linear function amid a scries of trigonometric tasks to test the models' reaction to sudden drastic changes in data distribution (Fig. [S3TS)). This 'injury" task, where a linear function interrupts the periodicity of trigonometric functions, disrupted the learning process. MFAN struggled during "injury', attempting to maintain periodical predictions. Multiple layer adjustments were observed as MFAN actively attempts to recognise the "injury' (Fig. [p3).... This also proves that the MFAN learning procedure has been hindered in this period. Once the lask switched back to trigonometric functions, MFAN quickly adapted and demonstrated high accuracy. This ability to recover and re-establish stable predictions after 'injury" illustrates MFAN's capability to recover in continuous data stream scenarios. In contrast, the BP quickly picked up the linear behaviour of 'injury'. Its reliance on gradient- optimisation allowed it to learn the linear relationship and reduce error rates quickly. However, BP suffered from deteriorated performance when shifting back to nonlinear tasks.
These results highlight MFAN's robustness in handling dynamic, real-world environments where the ability to adapt to evolving tasks is critical. MFAN's layered learning approach allows it to selectively focus on different aspects of the data, maintaining stability and performance when faced with evolving tasks. MFAN also demonstrated recovery when the data stream evolved from a new phase to an older pattern, which is worth further investigation. This investigation would involve the biological plausibility of the proposed approach.
During the 'injury' period in TS3, MFAN appears to struggle as it altempts to maintain periodical predic- tions. Multiple layer adjustments were observed, reflecting the model's active efforts to recognise the 'injury'' Despite this, MFAN quickly adapted and demonstrated high accuracy once the task returned to trigono- metric functions. Additional analysis reveals that MFAN decreases its error more effectively and becomes more stable during longer periods of consistent patterns. With 30% shorter data points, MFAN receives less consistent patterns each time, having 2% to 3% larger error rate amplitudes with oscillatory behaviour. Then, with 20% longer training time for each task, MFAN is observed have about 10% less error rate. These observations call for further theoretical exploration to understand the number of required data points during transitions between different data streams for accurate predictions.
10

---

Published in Transactions on Machine Learning Research (4/2025)
Prediction Comparison
BP prediction
prodiction
Eippectdd
TS1
BP
MFAN
BP prediction
NCRAN prediction
Unppected
TS2
BP
MFAN
ESP prediction
MIFAN pradicttion
Expacted
Exgrected
TS3
BP
MFAN
Error Rates Comparison
TS1
MFAN L1
TS2
MFAN L2
MFAN
BP error
TS3
MFAN error
x MFAN effective laver
Figure 4: a) Predictions for all task streams: TS1 tests the models' ability to handle task transitions and adapt to unseen data. TS2 introduces more complex trigonometric functions, testing the models' ability to deal with increased functional complexity. TS3 introduces a linear function amid trigonometric tasks, testing the models' adaptability during and after a significant change in task distribution. b) The percentage error rates corresponding to the predictions (a), numerically compare the performance of BP and MFAN for each task stream. c) Per-layer error rates: The per-layer error rate for MPAN and BP in TS1, illustrates how MFAN dynamically selects layers during training to stabilise error and improve prediction accuracy.
4 Discussion
MFAN presents advancements in neural network training, particularly for applications that require con- tinuous data streams and adaptability in dynamic environments. Through integrating contrastive and a multi-layer learning algorithm with dynamic layer adjustment, MFAN offers a promising alternative to tra- ditional BP, demonstrating good stability, performance, and flexibility, especially in handling continuous data given tasks.
MFAN excels function regression, where consistently outperformed BP by capturing complex, nonlincar relationships and generalising effectively unseen data. The contrastive learning properties (Chen et al., 2020 Du et al.) 2024) at the core of MFAN could be one of the primary reasons behind generalizability, while the layer adjustment and independent layer learning characteristics inherited from FF also contribute to better performance.
MFAN's performance in continuous data stream tasks further highlights its adaptability and stability. the long-term scenario with evolving data, MFAN generated more accurate and stable prodictions comparod BP. Moreover, MFAN effectively adapts varying task complexity and exhibits recovery the ability to regain performance when transitioning between distinct data phases, including reverting to previously
11

---

Published in Transactions on Machine Learning Research (4/2025)
encountered patterns. This aspect needs further investigation and discussion to potentially support the biological plausibility of MFAN.
4.1 Optimisation and Computation: Balancing Complexity and Efficiency
MFAN's flexibility in learning rate adjustment and layer-wise optimisation allows independent tuning of components fa. fy. and fy making it responsive to varying data complexities. This adaptability enhances its performance on diverse tasks, though it introduces additional hyper-parameters that require tuning.
MFAN requires about 1.5 times more computational resources than BP due to its complex architecture and (akes about 1.2 to 5 times more training time than a single BP, depending on the task. Yet MFAN can achieve half or third of the percentage error rate in regression tasks than BP, and achieve stability and high accuracy in continuous data stream tasks where BP fails to deliver accurate prediction. Thus, this investment is justified by these substantial advantages in adaptability and predictive accuracy in dynamic settings.
4.2 Biological Plausibility
We would like to discuss biological plausibility with the following aspects (Lv et al. 2024 : (i) asymmetry of weights; (ii) local error representation; (iii) non-parallel training, and (iv) the role of recurrent dynamics.
Asymmetry of weightss: In traditional artificial neural networks, synaptic weights the forward path are mirrored exactly along the foedback path through a mochanism called weight transport, es- sential to pack-propaggtton. This symmetry enables precise error correction but is considered biolog- ically implausible because real neurons unlikely to share exact synaptic weights. Instead, studies like [Nokland 2016) propose methods such as feedback alignment and direct feedback alignment, which replace symmetric weights with random fixed weights, allowing networks to learn effective error-driven representations without strict symmetry.
The MFAN model addresses this limitation by incorporating localised feedback mechanisms. Instead of relying on global feedback paths, MFAN uses per-layer feedback updates to adjust weights based on local input-output corrclations. This design evokes findings in wide neural networks, where weight updates can align with local statistics of inputs (Boopathy & Fiete 2022). Furthermore, localised feedback reflects the role of recurrent dynamics in the brain, where neural activity depends on both feedforward and recurrent interactions to adapt to complex tasks (Mante et al.] 2013).
Local error representation: Biological synapses are thought to adjust their strength using only local information, contrasting with the global error signal used in back-propagation. This local adap- lation mirrors how neurons in the brain rely on context-dependenl recurrent dynamics to selectively process and integrate sensory inputs (Mante et al. 2013)
Empirical studies, such as (Ito et al.) 2022), demonstrate that task-specific neural circuits dynami- cally adjust activations using intrinsic connectivity patterns. MFAN's emphasis on localised feedback aligns with these findings. By utilising layerwise adaptation, MFAN enhances its robustness and ability to generalise across tasks.
Conventional neural network training often involves distinct forward and backward propagation phases. Instead, natural learning processes are more continuous, with over- lapping phases for input integration and weight adaptation. Recent studies propose that biological networks achieve efficient learning through mechanisms like contrastive Hebbian learning and recur- rent altractor dynamics (Kymn el al. 2024h.
Non-parallel training:
MFAN excels in handling continuous data streams, demonstrating improvements over traditional BP. Its architecture supports real-time adaptation by updating weights as new data arrives, instead of waiting for batch processing. This dynamic also aligns with findings in the hippocampal-entorhnaa system, where modular attractor networks enable compositional representations of spatial and con- textual information (ymn al. 2024). MFAN also employs contrastive learning to enhance feature
12

---

Published in Transactions on Machine Learning Research (4/2025)
differentiation, which strengthens relevant connections while weakening less significant ones. This mechanism can help to manage the trade-off between retaining prior knowledge and adapting to new inputs.
While the forward path is often emphasised in biologically plausible models, recent studics highlight the importance of recurrent dynamics for implementing complex behaviours (Manle et al. 2013). Recurrent circuits in the prefrontal cortex enable dynamic selection and integration of task-relevant inputs, as demonstrated by (Mante et al., 2013 Similarly, (Kim et al. 2017) reveals hierarchical cortical organization driven by interneuron distributions, suggesting that recurrent interactions are essential for efficient neural computation. By integrating these principles, MFAN could be extended to recurrent-like processes to enhance learning efficiency and biological plausibility,
Recurrent dynamics:
MFAN approach lacks some other biologically plausible features, like the use of spikes and unsigned error signals. Spiking neurons enable temporal dynamics and energy efficiency in natural learning processes, while unsigned error signals offer direction-neutral approach to synaptic adjustments. Fortunately, MFAN's layer- rise-independent design provides flexibility for re-architecture. Each layer acts its own input and feedback loops, making it adaptable to integrating additional components and performing localised adjust such as incorporating spiking neural nelworks (Subbulakshmi Radhakrishman el al., 2021 Yin et al., 2023). or configuring specific layers for magnitude-only gradient updates to enhance biological realism.
celf-contained unit with
We acknowledge that deriving formal theoretical guarantees for the aligmment of 2n and Typo under all possible conditions is challenging endeavour. However, this work establishes robust empirical foundation for the assumption. We examined the assumed cosine similarity value after MFAN training and observed high-dimensional vectors with the average element value of 0.9981, close to 1, showing that the alignment between 2z and Zygee empirically holds. Future work could explore tighter bounds or additional theoretical frameworks to further strengthen this connection.
5 Conclusion
In this work, we introduced the Metamorphic Forward Adaptation Network (MFAN), an innovative neural architecture designed to overcome the limitations of traditional back-propagation and Forward- Forward al- gorithms. MFAN demonstrates flexibility and robustness in tasks requiring adaptation in streaming data, particularly through its dynamic layer depth adjustment and contrastive learning mechanisms. It outper- forms baseline models in function regression and continuous data stream by delivering stable and accurate predictions even in complex, evolving environments.
Beyond its technical advancements, MFAN represents a step toward more biologically plausible noural net- works. Unlike conventional models relying on global error signals, MFAN employs a localised, layer-wise learning approach akin to how biological systems may process and adapt to information. This allows MFAN to dynamically adjust to changing data streams, aiming reflect biological learning reorganise its neural connections in response to new experiences.
Looking ahead, future rescarch could further explore the integration of additional biologically inspired fea- tures, such as spiking neurons or more advanced forms of neural plasticily, to further align MFAN with the adaptive learning processes observed in natural systems.
Acknowledgments
We thank Mohamad Abdul Hady for the discussions during the development phase of this work. This rescarch is partially supported by seedcorn funds by Civil, Aerospace and Design Engineering, Isambard Al and Bristol Digital Futures Institute at the University of Bristol. This work was also partially supported by funding from the Engincering and Physical Sciences Research Council (award no. EP EP/NO18494/1. EP EP/R026173// EP/R RPP/R00953// 1, EP PPP/S31144/L EP EP/,000 /W001136/11, the SERI funded ERC Proteus consolidator grant (grant no: MB22.00066), the EU H2020 AeroTwin project (grant ID 810321), the Empdefunded DroneHub project.
13

---

Published in Transactions on Machine Learning Research (4/2025)
and the Empa-Imperial College London research partnership. Mirko Kovac was supported by the Royal
Society Wolfssn fellowship (RSWF/R1/1800))
References
Md Atik Ahamed, Jin Chen, and Al-Zubar Imran. Forward-forward contrastive learning. arXiv preprint ar orx::22305.0 02922, 2023. doi: 2w.45550/arXiv.235...mm
Andri Ashfahani and Mahardhika Pratama. Autonomous deep learning: Continual learning approach for dynamic environments. In Proceedings of the 2019 SIAM international conference on data mining, pp. 666-674. SIAM, 2019. doi: 10.1137/1.778161197766
Yoshua Bengio, Dong-Hyun Lee, Jorg Bornschein, Thomas Mesnard, and Zhouhan Lin. Towards biologically plausible deep learning. arXiv preprinl arXie: 04 156, 2015. doi: 10.48550/arXiv.55022000 1502044166.
Akhilan Boopathy and Tla. Fiete. How to train your wide neural network without backprop: An input-weight aligmment perspective. In International Conference on Machine Learning, pp- 2178 2205. PMLR, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020. URL https://arxiv.
05709
Shibhansh Dohare, J Fernando Hernandez-Gareia, Qingfeng Lan, Parash Rahman, A Rupam Mahmood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 632(6826)::68 774, 2024. doi: 0111888/555860224-077777
Jun Du, Jianhang Jin, Jian Zhuang, and Cheng Zhang. Hierarchical graph contrastive learning of local and global presentation for multimodal sentiment analysis. Scientific Reports, 14(1):5335, 2024. doi: 00.1038/s11588-024-555772
Valerio Francioni, Vincent D Tang, Norma J Brown, Enrique HS Toloza, and Mark Harnett. Vectorized instructive signals in cortical dendrites during brain-computer interface task. bioRain, 2023.
Saumya Gandhi, Ritu Gala, Jonah Kornberg, and Advaith Sridhar. Extending the forward forward algorithm. arXiv preprint arXie:2307.04205, 2023. doi: 10.48550/ wttp:///aXii......0.220
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnor- malized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297 304. JMLR Workshop and Conference Proceedings, 2010.
Guy Hay and Nir Sharon. Function extrapolation with neural networks and its application for manifolds. arXiv preprint 17550:2405.1056, 2024. doi: 10.48550 /ae///aXivvv.000 200.1...005.0.63
Geoffrey Hinton. The fowarr-ffrwart algorithm: Some preliminary investigations.
arXi preprint
arXiv:2212. 18345, 2022. doi: 10.48550/arXiv 2212.13355..
Geoffrey E Hinton, Terrence J Sejnowski, et al. Learning and relearning in boltzmann machines. Parallel distribuled processing: Explorations in the microstructure of cogmilion, 1(282-317):2, 1986. doi: 10.7551 3334s./3333333333333
Takuya Ito, Guangyu Robert Yang. Patryk Laurent, Douglas H Schultz, and Michael W Cole. Construct- ing neural network models from brain data reveals representational transformations linked to adaptive behavior. Nature communications, 13(1):673, 2022.
Stefano Izzo, Fabio Giampaolo, Diletta. Chiaro, and Francesco Piccialli. Shelob-fft ffl: addressing systems heterogeneity with locally backpropagated forward-forward learning. In 2024 IEEE 44th International Conference on Distributed Computing Systems Workshops (ICDCSW), pp. 23 28. IEEE, 2024. doi: 10. 1100/TCDCSWc336.2222.000
14

---

Published in Transactions on Machine Learning Research (4/2025)
Ali Karimi, Ahmad Kalhor, and Mclika Sadeghi Tabrizi. Forward layer-wise learning of convolutional neural networks through separation index maximizing. Scientific: Reports, 14(1):8576, 2024. doi:
10.0033//415996-24455777673
Yongsoo Kim, Guangyu Robert Yang, Kith Pradhan, Kannan Umadevi Venkataraju, Mihail Bota, Luis Carlos García Del Molino, Greg Fitzgerald, Keerthi Ram, Miao He, Jesse Maurica Levine, et al. Brain- wide maps reveal stereotyped cell-type-based cortical architecture and subcortical sexual dimorphism. Cell,
469, 2017.
Basaran Bahadir Kocer, Mohamad Abdul Hady, Harikumar Kandath, Mahardhika Pratama, and Mirko Kovac. Deep neuromorphic controller with dynamic topology for aerial robots, In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp- 110-116. IEEE, 2021.
Deepak Kumar, Umang Goswami, Hariprasad Kodamana, Manojkumar Ramteke, and Prakash Kumar Tam- boli. Variance-capturingg forward-forward autoencoder (vffae): A forward learning neural network for fault detection and isolation of process data. Process Safety and Environmental Protection, 178:176-194, 2023.
doi: 10. http://jsspp.2......
Christopher J Kymn, Sonia Mazelet, Anthony Thomas, Denis Kleyko, E Paxon Frady, Friedrich Sommer, and Bruno A Olshausen. Binding in hippocampal-entorhinaal circuits enables compositionality in cognitive maps. ArXiv, pp. arviv 2406, 2024.
Alfirna Rizqi Lahitani, Adhistya Erna Permanasari, and Noor Akhmad Setiawan. Cosine similarity to determine similarity mcasure: Study case in online essay assessment. In 2016 4th International conference on cyber and IT service management, pp. 6. IEEE, 2016. doi: 10.1109/ 00110//CCTSM...1..77777777
Zhouhan Lin, Roland Memisevic, and Kishore Konda. How far can we go without convolution: Improving fully-connected networks, 2013. URL https://arxiv org/ abs
1511 abs/1511.....1//11//11/11/11//11
02580
Changze Lv, Yufci Gu, Zhengkang Guo, Zhibo Xu, Yixin Wu, Feiran Zhang, Tianyuan Shi, Zhenghua Wang, Ruicheng Yin, Yu Shang, et al. Towards biologically plausible computing: A comprehensive comparison. arXiv preprint 2rXiv:2406.1600 2024. doi: 10.48550/ 40.48550/arXiv. 2ar.ivv2200..666.
Valerio Mante, David Sussillo, Krishna V Shenoy, and William T Newsome, Context-dependent computation by recurrent dynamics in prefrontal cortex. nature, 57377)7)):78- 84, 2013.
Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. Aduances in neural information processing systems, 29, 2016.
Abel A Reyys-Angulo and Sidike Paheding. forward algorithm for hyperspectral image classification. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3153 3161, 2024.
Shiva Subbulakshmi Radhakrishnan, Amritanand Sebastian, Aaryan Oberoi, Sarbashis Das, and Saptarshi Das. A biomimetic neural encoder for spiking neural network. Nature commaniicatinns, 12(1):2143, 2021. doi:
Yanshuo Wang, Ali Cheraghian, Zeeshan Hayder, Jie Hong, Sameera Ramasinghe, Shafin Rahman, David Ahmedt-Aristizaauu Xuesong Li, Lars Petersson, and Mehrtash Harandi. Backpropagation-fiee network for 3d test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23231 122211, 2024.
Keyulu Xu, Mozhi Zhang, Jingling Li, Simon S Du, Ken-ichi Kawarabayashi, and Stefanie Jegelka. How neural networks extrapolate: from feedforward to graph neural networks. In International Conference on Learning Representations (ICLR), 2021. doi: 10.48550/arxiv.2000 www.xXiv..00111...0
Bojian Yin, Federico Corradi, and Sander M Bohté. Accurate ouline training of dynamical spiking neural networks through forward propagation through time. Nature Machine Intelligence, 5(5):518 527, 2023. doi: 10.1038 10.1038/s422585
15

---

Published in Transactions on Machine Learning Research (4/2025)
A
Appendix
A.1
Classification
A.1.1
MNIST Classification
Models were trained on 60,000 MNIST images and tested on 10,000 images. Model configurations are outlined in Table A1 below. Accuracy and latent space distribution were analysed.
Table Al: Model setups for classification comparisons.
MNIST Model Setup
MFAN
BP
FF
Network depth
3-layer
2-layer
3-layer
Network width
[500, 300, 300]
(00,
[500, 300, 300]
Learning rate
5e
5e
5e
5e-1
5e-1
[a1, 02)
[0.6, 0.6]
Number of train data
60000
60000
60000
Number of test data
10000
10000
10000
Encoder loss
Cosine Similarity
Decoder loss
Cross Eutropy
Cross Entropy
Cross Entropy
A.1.2 CIFAR-10 Classification
Models were trained on 50,000 CIFAR-10 images and tested 10,000 images. Model configurations are outlined in Table A2 below. Accuracy and latent space distribution were analysed.
Table A2: Model setups for classification comparisons.
CIFAR Model Sctup
MFAN
BP
FF
Network depth
3-layer
3-layer
3-layer
Network width
[256, 128, 64]
[256, 128, 64)
[256, 128, 64)
Learning rate [a1, az]
5e 2
.e 3
55
5e
5e 3
[0.6, 1]
Number of train data
50000
50000
50000
Number of test data
10000
10000
10000
Encoder loss
Cosine Similarity
Decoder loss
Cross Entropy
Cross Entropy
Cross Entropy
16

---

Published in Transactions on Machine Learning Research (4/2025)
A.2 Function Regression
Models were trained and lested on self-generated function regression datasets. Models were assessed on generalizability, the understanding of underlying relationships, and the ability to handle functions with various complexitios. Model setups are listed in Table A3
Table A3: Model sctups for function regression comparisons.
FR Model Setup
MFAN
BP Model
Network depth
6-layer
3-layer
Network width
[128, 128, 64, 64, 32]
128, 64, 32]
Learning rate
1e 1e-5 le 11e=
1e-5
[d1,0)]
[0.6, 1]
Number of data
100
100
Length of data
100
100
examples: Ø1, 22
[0, I..,, 100), [0.1, 1.1, 100.1]
[0, 1, 100),
[0.1, 1.1, .... 100.1]
Train domain
[0, 110]
[0, 110
Test x domain
1000, 210
1100, 210]
Uposuin 0.1, Mpossin
0.03 0.03, Vpossaxx
Range Yneg
Upositt
0.1
Encoder loss
Cosine Similarity
Decoder loss
Root Mean Square
Root Mcan Square
17

---

Published in Transactions on Machine Learning Research (4/2025)
A.2.1 Additional evaluations
Table A4: Additional evaluations.
Evaluation Aspect
MFAN
BP Model
Training Time (3 layers)
2.7s
0.5s
Training Time (6 layers)
5s
1s
Slight drop
Performance Change (3 - 6 layers)
+2% error rate increasc
Negligible
Performance Change (6 + 3 layers)
+0.12% error rate increase
Inference Time
0.03 0.06s
0.02 0.04s
A.3 Continuous Data Stream
Models were trained and lested self-generated function regression datasets. Models were assessed on generalisability, the understanding of underlying relationships, and the ability to handle functions with various complexities. The model setup is listed in Table A3 Models were evaluated by performing learning for a continuous data stream of three tasks in dynamic environments. The models' predictions and error rates were monitored. Models' long-term dynamic adaptability and stability were evaluated. Detailed model configurations are shown in Table A5
Table A5: Model setups for function regression with continuous data stream comparisons.
Model Sctup
MFAN
BP Model
Network depth
3-layer
3-layer
Network width
(64, 16)
644 32, 16
Learning rate
le
1e-5 le
le-1
[a1, a2)
[0.6, 1]
Number of data
2901
2901
Length of data
100
100
I examples: 31, 22
[0,0.2 ..., 18.7], (0.2, 0.4, 18.9]
[0, 0, 18.7], [0.2, 0.4, 18.9
Uparmin 0.1, pposmin 0.03]
Range Yneg
[yposami + 0.03, Upos
+ 0.1]
Encoder loss
Cosine Similarity
Decoder loss
Root Mean Square
Root Mean Square
Training time
0.0034s
0.0006s
Inference time
0.0024s
0.0004s
18


Reviews:
[{"review_id": "EGdMkmwxb8", "reviewer": "Reviewer_P38z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main downside of the paper, as far as I can see, is the extreme simplicity of the experiments. This, in general, would not be a problem, but the authors write in their introduction:\n\n\"Even though FF provides a more modular, layer-wise learning approach, it tends to be slower, particularly when applied to large-scale tasks...\"\n\nThis suggests they aim to improve upon this property. However, the most complicated task considered is MNIST. It would therefore be beneficial to run MFAN on more challenging tasks, such as CIFAR-100.\n\nAdditionally, unless I am misunderstanding the authors' intent, I believe this sentence in the introduction is incorrect: \"There is little evidence for the backward flow of error signals, which is central to BP, in biological neural systems.\"\n\nThere is ample evidence for the backward flow of error signals in biology—see [here](https://pubmed.ncbi.nlm.nih.gov/37961227/) for a very recent example.\n\nFinally, this sentence: \"As BP relies on global gradient updates, BP-trained models often struggle to adapt to dynamic environments or generalize to evolving tasks\" is missing a period at the end.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is a good paper that introduces a simple change to the Forward-Forward (FF) algorithm to improve it. The authors show that on certain simple tasks, such as regression, their method performs better than backpropagation (Table 1). The paper is well-written, and I felt I could understand everything.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "This is a good paper, but I think the authors should run at least one more experiment with a task more challenging than MNIST. This would give readers more confidence that MFAN can scale to harder problems. \n\nIf the authors include such an experiment, I will vote to accept the paper. MFAN does not need to outperform BP on this more challenging experiment; however, if MFAN performs worse than BP, the authors should explain why they think this is so, and how they think the algorithm can be improved going forward.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Az49d5ZVzd", "reviewer": "Reviewer_uh3q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strength]\n\nThe paper proposes a biologically plausible network using a forward-forward model with a contrastive loss function. The proposed method, MFAN, demonstrates adaptive prediction capabilities and outperforms the baseline in certain metrics.\n\n\n[Weakness]\n\n**Theoretical Guarantees:**\nA key assumption of the model is that $z_x \\approx z_{y_{\\text{pos}}}​​$. However, there appears to be no theoretical guarantee for this assumption. This lack of justification weakens the model's foundation.\n\n**Vague Details:**\nSeveral crucial aspects of the model lack sufficient explanation. For example, the effective layer selector is a critical component, yet the authors only mention that \"the layer with the best performance on the training set is used at inference time\" and \"the model records the best-performing layer at the previous time step.\"\n- During the online continual learning tasks, how is the performance of the previous time step quantified during inference?\n- Does this imply the model references evaluation results for layer selection?\n- What criteria define \"best performance\"? Is it loss, accuracy, or another metric?\n\n**Implementation Disparities:**\nIt is unfair to compare MFAN (6 layers) with a baseline model that uses only a 3-layer MLP. Additionally, the paper does not provide details about the training and inference times for MFAN relative to standard backpropagation models of similar size.\n\n**Layer Selection and Adaptive Prediction:**\nAdaptive prediction is one of advantages of the proposed method, as argued by the authors although it requires to predict from all layers to choose the target. However, further details about the layer selection mechanism are necessary. I strongly recommend measuring inference time and demonstrating the proposed model's superiority compared to plausible baselines, such as a classifier head at each layer in backpropagation models.\n\n**Implementation Details:**\nThe paper lacks details about the implementation, such as whether non-linear activation functions are used between layers.\n\n**Related Works:**\nThe discussion on related works is insufficient. Many biologically plausible methods address the limitations of backpropagation, such as Feedback Alignment [6] and Neural Tangent Kernel [1]. These should be acknowledged and compared.\n\n**Sensitivity Analysis:**\nHow sensitive is the model's performance to the choice of noise degree in experiments? A sensitivity analysis is highly recommended.\n\n**Task Definition and Misalignment:**\nDo you have any intuition as to why the BP model in Figure 2 (center) is slightly misaligned with the ground truth? Could this be due to the task definition?\n\n\n**Continual Learning Claims:**\nAlthough the authors claim the experiments were conducted in an online continual learning setting, the setup appears to involve streaming data rather than true continual learning. To substantiate the claim, the paper should measure forward and backward transfers across tasks. Furthermore, Figure 3 suggests that the test data closely resembles the training data (e.g., periodic functions with slight shifts). This raises questions about the model's generalizability, especially considering its worse performance than the BP model in Section 2.1.\n\n**Analysis of the ‘Injury’ Period:**\nThe authors should analyze why MFAN struggles during the \"injury\" period in TS3.\n\n**Neural Plasticity:**\nThe claim that \"MFAN also demonstrated neural plasticity to some extent\" lacks supporting evidence. In standard continual learning, balancing plasticity and stability is crucial. Referring to the task as \"online continual learning\" adds to the confusion.\n\n**Forward Path Emphasis:**\nThe authors argue that only the forward path is important for biologically plausible networks. However, several studies emphasize the role of recurrent dynamics in the brain [3,5] and provide methods for implementing such dynamics [2,4].\n\n[1] Boopathy, Akhilan, and Ila Fiete. \"How to train your wide neural network without backprop: An input-weight alignment perspective.\" International Conference on Machine Learning. PMLR, 2022.\n\n[2] Ito, Takuya, et al. \"Constructing neural network models from brain data reveals representational transformations linked to adaptive behavior.\" Nature communications 13.1 (2022): 673.\n\n[3] Kim, Yongsoo, et al. \"Brain-wide maps reveal stereotyped cell-type-based cortical architecture and subcortical sexual dimorphism.\" Cell 171.2 (2017): 456-469.\n\n[4] Kymn, Christopher J., et al. \"Binding in hippocampal-entorhinal circuits enables compositionality in cognitive maps.\" NeurIPS. 2024.\n\n[5] Mante, Valerio, et al. \"Context-dependent computation by recurrent dynamics in prefrontal cortex.\" nature 503.7474 (2013): 78-84.\n\n[6] Nøkland, Arild. \"Direct feedback alignment provides learning in deep neural networks.\" Advances in neural information processing systems 29 (2016).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new forward-forward algorithm, called Metamorphic Forawrd Adaptation Network (MFAN). The authors use contrastive loss by adding noise to make negative samples. The paper shows the superiority of the proposed method compared to 3-layer MLP trained with the standard back-propagation on MNIST and toy trigonometric functions.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- In Section 1, zy_pos -> $z_{y_\\text{pos}}$\n- In Eq. 8, $f_y^{-1}f_x$ -> $f_y^{-1}(f_x$\n\n- Move Figure 3 on Page 7 as it is referred to on page 5.\n\n- All citation is citet, please change them to citep.\n- In Figure 3 $segment_1tosegment_5$ -> $segment_1$ to $segment_5$.", "broader_impact_concerns": "I do not have any concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ssOaR2jXfH", "reviewer": "Reviewer_1Acp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed algorithm seems to be novel and has comparable performance in the presented experiments. However, I cannot fully evaluate the validity of this work until the algorithm and experimental set-ups are clarified. Your proposed algorithm is never actually specified. Are you using gradient descent with these loss functions to update weights for each of the three networks? If so, how does it avoid the multiple apparent downfalls that you have provided for standard BP methods, and how is it still \"layer-wise\" as you claim in Section 3?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces the Metamorphic Forward Adaptive Network (MFAN), a novel neural network architecture and algorithm inspired by the Forward-Forward (FF) algorithm and contrastive learning. MFAN aims to address limitations of both back-propagation (BP) and FF, particularly in continual learning scenarios. MFAN encodes both inputs and outputs to a latent space, encouraging training input samples to be close to correct outputs and far from incorrect outputs. Then, a decoding network is learned from the latent space to classification or regression outputs to predict on new inputs. Experiments comparing to BP in classification, regression, and continual learning are provided.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Upon making the following changes (as well as answering the questions above), I will re-evaluate this work to determine whether I can recommend it for acceptance.\n\nYour paper should explain your proposed algorithm clearly enough so that a reader could reimplement it themselves. So far, you have only described the loss function without explaining what \"training\" actually looks like, whether it's back-propagation or not. I recommend including a pseudocode algorithm as well as a much more thorough textual description. This is critical to support your claims in the rest of the paper.\n\nYou discuss the FF model quite in-depth in the abstract, introduction, and Figure 1, but only experimentally compare to BP. You should include the original FF algorithm as well as any relevant improvements in your experiments for proper empirical comparison.\n\nHow did you select hyperparameters? Improperly selected hyperparameters can confound experimental results by putting one algorithm at an unfair advantage or disadvantage to others. Even further, you should discuss the computational cost of hyperparameter optimization for each approach. Also, why is it fair in your Function Regression experiments that the MFAN model has twice as many layers as the BP model?\n\nPlease use `\\citep{}` for parenthetical citations. The vast majority of your citations, as currently written, should be parenthetical rather than textual. For example, \"...maintaining the plasticity Dohare et al. (2024).\" -> \"\"...maintaining the plasticity (Dohare et al., 2024).\"\n\nGenerally, the figures and tables appeared quite late compared to when they were referenced in the text. The top of the following page is a good default location. \n\nAlso, most figures contain text that is much too small to read at a standard level of zoom: all text should be legible when printed and no smaller than subscripted text in an equation (such as Equation 6).\n\nThe captions of figures and tables should only include clarifying information relevant within the figure/table. Any supporting information, such as how the experiments are designed under Figures 2-3, and analysis or discussion, such as the notes under Table 1, should be in the main text.\n\nSpecific corrections and changes: \n\nPage 1, abstract: \"However, back-propagation is gradient-dependent...\": this is redundant after the previous sentence. Rather, why is it bad to be gradient-dependent?\n\nPage 1, abstract: \"...and lacks robustness...\" -> \"...and lacking robustness...\"\n\nPage 1, abstract: \"A promising alternative to back-propagation is the Forward-Forward algorithm...\": either cite this and/or say that it was \"previously proposed\".\n\nPage 1, abstract: \"...not yet matched...\" -> \"...doesn't yet match...\"\n\nPage 1, abstract: \"...using contrasting learning property...\" -> \"...using a contrasting learning property...\"\n\nPage 1, introduction paragraph 1: \"...advancements in the fields...\" -> \"...advancements in fields...\"\n\nPage 1, introduction paragraph 2: \"BP often requires ... continuously changing.\": This sentence is not very clear, particularly how the first part is related to the second part. Are you referring to catastrophic forgetting, and/or how the gradient change from a batch of training data may not be a useful step for test data?\n\nPage 1, introduction paragraph 2: missing period at end of last sentence.\n\nPage 2, paragraph 2: \"logistic function, $\\sigma$ to the goodness minus $\\theta$,\" -> \"logistic function, $\\sigma$, to the difference of the goodness and $\\theta$\"\n\nPage 2, paragraph 6: \"both x and y as inputs\": what are $x$ and $y$? Always define variables, even if they seem obvious.\n\nPage 3, last paragraph: please also provide these results in a table for clarity. \n\nPage 4, Figure 1 caption: letter labeling is inconsistent.\n\nPage 5, Table 1 captions: \"The corresponding... Fig. 2)\" is a sentence fragment.\n\nPage 6, Figure 2 caption: Two closing parentheses are missing, one in the mathematical expression in the 2nd sentence and the other in the 3rd. Also, I don't understand this \"sliding window\" approach: it seems different to that described under Figure 3. Particularly, I think these expressions are not correct: \"training dataset $x: x_{i+1} = x_i + 0.1i$ with $x_0 = 0$ $i \\in \\mathbb{Z}$, $i \\in [1, 100]$, training dataset $y_{pos} = f(x)$\", as they imply $x_0=0, x_1=x_0+0.1\\cdot1=0.1, x_2=x_1+0.1\\cdot2=0.3, x_3=x_2+0.1\\cdot3=0.6, ...$.\n\nPage 8, Figure 3 caption: \"Evenly divide the task stream into...\" -> \"The task stream is evenly divided into...\". Also, use `\\text{}` to write out words within mathematical expressions: i.e. `\"on $\\text{segment}_1$ to $\\text{segment}_5$,\"` yields \"on $\\text{segment}_1$ to $\\text{segment}_5$,\".\n\nPage 9, paragraph 1: \"While MFAN training demands more computational resources than BP due to its complex architecture, ...\": how much more? Please report this for at least one of your experiments. \n\nPage 9, Section 3.2 header: \"Biological Plausability\" -> \"Biological Plausibility\"\n\nPages 12-14: Ensure the appendix tables are labeled and captioned, like \"Table A.1: Hyperparameter configurations for MNIST Classification experiments.\".", "broader_impact_concerns": "No broader impact statement included: I don't think one is necessary.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is technically sound overall. 
However, some of the concerns remain. 
Particularly, regarding the sensitivity analysis question, the claim about MFAN’s robustness is not explained with enough detail. There might be pattens in the performance results that require additional attention. 
"When Gaussian noise is introduced during training, with small standard deviations, MFAN’s error rate increases by only 2%, while BP’s error rate rises by 8%. With a high noise level, MFAN maintained a relatively steady performance." 
Please specify the exact standard deviation and what "high noise" means, and what "relatively steady" is.

Please cross-check for possible inconsistencies of presenting additional experiments and results in the rebuttal and the revised manuscript. 

MFAN performs substantially worse than BP on CIFAR-10 (Section 3.1.2, newly added in revision). The provided explanation "However, with much wider networks (over 10 times wider than the short-training setup) and longer training times, both FF and MFAN improved significantly, achieving test accuracies exceeding 46%." needs further elaboration. Can we actually see the performance of BP with a 10x wider network? Efficiency aspects need to be discussed too as MFAN with a wider network requires much more time compared to BP.
Decision: Accept (poster)