Enabling Fine-Tuning of Direct Feedback Alignment via Feedback-Weight Matching
Authors: Yunseok Lee
Seulki Lee
Venue: ICLR 2026 Poster (2026)
Keywords: direct feedback alignment
deep learning
fine tuning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: First study on effective fine-tuning with DFA (Direct Feedback Alignment)

Abstract: Although Direct Feedback Alignment (DFA) has demonstrated potential by enabling efficient and parallel updates of weight parameters through direct propagation of the network's output error, its usage has been primarily restricted to training networks from scratch. In this paper, we introduce feedback-weight matching, a first method that enables reliable fine-tuning of fully connected neural networks using DFA. We provide an analysis showing that existing standard DFA struggles to fine-tune networks that are pre-trained via back-propagation. Through a thorough analysis of weight alignment (WA) and gradient alignment (GA), we demonstrate that the proposed feedback-weight matching enhances DFA's ability and stability in fine-tuning, which provides useful insights into DFA's behavior and characteristics when applied to fine-tuning. In addition, we prove that feedback-weight matching, when combined with weight decay, not only mitigates over-fitting but also further reduces the network output error, leading to improved learning performance during DFA-based fine-tuning. Experimental results show that feedback-weight matching, for the first time, enables reliable fine-tuning across various fine-tuning tasks, compared to existing standard DFA, e.g., achieving 7.97% accuracy improvement on image classification tasks (82.67% vs. 74.70%) and 0.66 higher correlation score on NLP tasks (0.76 vs. 0.10). The code is available on an anonymous GitHub.

Published as a conference paper at ICLR 2026
ENABLING FINE-TUNING OF DIRECT FEEDBACK ALIGNMENT VIA FEEDBACK-WEIE MATCHING
Yunscok Lee
Seulki Lee
UNIST walk10098umist.ac.
KAIST
ABSTRACT
Although Direct Feedback Alignment (DFA) has demonstrated potential by en- abling efficient and parallel updates of weight parameters through direct propaga- tion of the network's output error, its usage has been primarily restricted to training networks from scratch. In this paper, we introduce bodbbck-weiigt matching, a first method that enables reliable fine-tuning of fully connected neural networks using DFA. We provide an analysis showing that existing standard DFA struggles to fine-tune networks pre-trained via pack-propagation. Through thorough anal- ysis of weight alignment (WA) and gradient alignment (GA), we demonstrate that the proposed feedback-weight matching enhances DFA's ability and stability fine-tuning, which provides useful insights into DFA's behavior and characteris- ties in fine-tuning. In addition, we prove that feedback weight matching, when combined with weight decay, not only mitigates over-fittinn, but also further re- duces the network output error, leading to improved learning performance during DFA-based fine-tuning. Experimental results show that feedback- match- ing. for the first time, enables reliable fine-tuning across various fine-tuning tasks, compared to existing standard DFA, e.g., achieving 7.97% accuracy improvement on image classification tasks (82.67% vs. 74.70%) and 0.66 higher correlation score NLP tasks (0.76 vs. 0.10). The code is available on an GitHut
INTRODUCTION
Recently, alternative training mechanism called Direct Feedback Alignment (DFA) (Nokland 2016) has been proposed. Based on the concept of Feedback Alignment (FA) (Lillicrap et al. 2016), DFA passes the error of the output layer directly each layer of the network to update the weight parameters without compute-intensivy back- back-propagation (Rumelhart et al., 1986). By using random feedback matrices, the weight gradient of each layer is independently approximated from the directly passed error, enabling efficient training of fully connected networks through the parallel update of multiple layers. This contrasts with the conventional back-propagation that propagates the network етror sequentially from the last to the first layer.
Although Direct Feedback Alignment (DFA) (Nokland 2016) has shown its potential in training primarily for fully connected networks (Garg & Vempala 2022 Launay et al. (020). its application to fine-tuning (Devlin al. 2018), i.e., adapting a pre-trained network to a new task, has been less studied until today despite its practical usefulness. In fact, it has been known that fine-tuning net- works with DFA is challenging (Chu & Bacho 2024); the performance of networks fine-tuned with DFA is generally unreliable compared to that of those fine-tuned with pack-propagation (Rumelhart ct al. 1986). Given that fine-tuning has become one of the practical and also effective ways re- utilizing pre-trained networks for various downstream tasks (Church et al., 2021), investigating how DFA can be applied to the fine-tuming mechanism both theoretically and empirically is necessary.
Enabling fine-tuning with Direct Feedback Alignment (DFA) (Nøkland 2016) not only broaden DFA's usability but also introduce an alternative approach to current back pooodk-ropggatoo--baae tuning (Rumelhart al.) 1986) Church et al.] (021). Currently, DFA has not yet been estab- lished as a reliable stand-alone training method that can provide comparable performance to back- propagation (Launay et al. 2019 Crafton et al. 2019). Thus, taking wide range of well-pre-trained
fine-
https://gtthub.commma www..co/oaaaaaaaoooaooaaao Sab/Feedbsckes hatp:eeedbickeee.hhhahtthhh chiM

---

Published as a conference paper at ICLR 2026
Feedback matrix
Feedback Weight Matching (Ours)
Fine-tuning process
Fully connected layer
Feedback Matching
Weight Matching Output
using standard DFA
Output
Output
Output
Win1
W
P1+1
Weel
P
Decompose
F
Reconstrnet
W
W
Fi
W
F
W
W -
Fi-1
Wi-1
F
Input
Imput
Input
Input
Pre-trained Network
Feedback Network Weight Matched Network
Figure I: An overview of the Feedback-Weiee Weight Matching process.
models, such Transformer-baset foundation models (Kenton & Toutanova 2019), as the starting point would be practical strategy that can complement DFA's unstable and limited learning capa- bilities. Additionally, by incorporating DFA's unique advantages, such back- sock-ropagattion free and parallel training, into the widely used fine-tuning, we can explore new possibilities for re-utilizing pre-trained models with DFA more agile, efficient, and biologically plausible manner, in contrast to conventional puck-propagation requiring much more resources and time in model training.
In this paper, we introduce -FAed fine-tuning method, which investigates the feasibility of Di- rect Feedback Alignment (DFA) (Nokland (016) for fine-tuning deep neural networks, with the aim of extending the scope of DIA to embrace various pre-trained networks. We first analyze the reasons why the existing standard DFA, which updates the pre- pre- trained weights using random feedback ma- trices. does not perform well in fine-tuning, This analysis is based on the weight aligmment (WA) and gradient alignment (GA) (Refinetti et al. 2021), which are two measures proposed to estimate the state and learning performance of DFA. From this analysis, we propose feedback- weeihhh matching as illustrated in Fig. I which first reconstructs the feedback matrices by decomposing the pre- tre-trained weights (feedback matching) and then re-initializes the weights based on the reconstructed feedback matrices before starting fine-tuning (weight matching). Additionally,
prove that applying weight decay (Krogh & Hertz 1991) on top of the proposed feedback-weight matching considerably im- proves and stabilizes the fine-tuning performance of DFA, beyond the general regularization effect on weight parameters. Together with the simple yet effective feedback-weight matching, weight decay acts key facilitator for fine-tuning fully connected networks with DFA. To the best of our knowledge, this work is the first to explore the possibility of applying DFA to fine-tuning of fully connected networks with in-depth study.
The experiments provide evaluation results consistent with our theoretical analysis; applying bodbbakk-eiigh matching enables more effective and reliable fine-tuning of fully connected net- works with DFA over various fine-tuning tasks, when compared to existing standard DFA (Nøkland) 2016). For instance, the image classification accuracy of fully connected networks fine-tuned with feedback-weigh matching reaches 82.67%, while the standard DFA achieves 74.70% 70%. Also, it suc- cessfully fine-tunes Transformer models (BERT) (Devlin et al. 2018) on NLP tasks, e.g., achieving 0.76 correlation score, while the standard DFA barely conducts fine- -t--ittingg at all. i.c., achieving mere 10 correlation score. These results demonstrate the potential for extending DFA to the widely used pre-training and fine-tuning strategy, moving beyond its limited usage in from-scratch training.
2 BACKGROUND AND RELATED WORK
DEA. is common to train neural network using the Sack-propagation algorithm (Rumelhart al. 1986). Given fully connected network, we denote W, as the weight of (-th layer of the network, ((y.)) as the loss function, where ý the ground-truth output, and y the network output, and W,h To update the weight parameter with the gradient descent algorithm (Ruder, 2016), the gradient of the loss L w.r.t the weight W, is obtained using back pack-propagation (BP) as:
h = g(ar) as the output of the 1-th layer, where g(-) is the activation function, and
dL
SWBr_
oW, [(W], 1)) c o (a:) hi1:; Ōay=aC/öa
where @ is the Hadamard product. However, back propaaattonnnn poses challenges, specifically the weight transport (Grossherg 1987. Crick 1989) and backward locking problems (Lillicrap al.)
2

---

Published as a conference paper at ICLR 2026
2020 Launay et al. 2019). Direct Feedback Alignment (DFA) (Nokland 2016) addresses the weight transport problem by employing random feedback and mitigates the backward locking problem by delivering the network's output error to each layer independently. Specifically, 1) the global error e = y y is transmitted to each layer, and 2) the weight Wi+1 at the I-th layer is replaced with a random feedback matrix Fj, leading to the following weight gradient:
Oc
SWPPA
((Fee O g'((a))) (((a)))
(2)
Ow,
where X is the weight-decay hyperparameter. Eg. (2) eliminates the necessity of sequential layer- wise gradient computations of back proppaappaatioon (Rumelhart et 1986).
GA and WA. To better elucidate the dynamics of DFA (Nokland 2016), the concept of gradient alignment (GA) is introduced (Lillicrap et al.) 2016). GA quantifies the similarity between the weight gradients obtained through DFA and those derived via back-propagation (Rumelhart et al. 1986). This achieved by comparing the weight updates generated from the identically initialized weights by both methods. It has been hypothesized that a stronger (higher) GA corresponds to enhanced learning performance in DFA. In addition, the weight alignment (WA) (Refinetti et al. 2021) has been introduced to evaluate the relationship between the weight and the feedback matrix in DFA, suggesting that strong WA is associated with strong GA. Although GA and WA have been instrumental in analyzing the learning efficacy of DFA, prior research has not explored their utility in the context of fine-tuning. In contrast, this paper pioneers the application of GA and WA concepts systematically investigate the fine-tuning process in DFA.
Applying DFA to Transformers and CNNs. Some studies (Launay et al. 2020) explore the appli-
cability of DFA (Nøkland (2016) to various fully connected networks, including NeRF (Mildenhall | Sitzmann et al. 2019), recommender systems (Guo et al. 2017). and NLP (Vaswani Merity et al.) 2016). While they show that DFA can train range of deep architectures, they also reveal a significant performance gap between DFA and back-propagation
ct al.
2017
(Rumelhart jet al. 1986), particularly in Transformer models (Vaswani 2017). When applied to models not based on fully connected networks, such as CNNs, the performance gap between DFA and back- propagation is even more pronounced. For instance, VGG- (Simonyan & Zisserman, 2014) on CIFAR- 100 (Krizhevsky et al. 2009) trained with DFA achieves 1% top-1 accuracy (Launay al. 2019). while back- boeck-ropagattio achieves 60%. Similarly, in ImageNet (Deng et al.) (009). it is 6.2%
53% (Crafton et al., 2019). Given that applying DFA to from-scratch training scenarios 1) consistently underperforms relative to back pock-propagation, 2) takes a much longer training time than fine-tuning, and 3) is limited to a narrower range of architectures, we argue that utilizing fine-tuning for DFA would be a more effective, efficient, practical, and expedient approach. Thus, in this study, we investigate the potential of employing DFA in fine-tuning which is conducive to the widely-used and--ine-tuune- strategy (Devlin et al. 2018).
Applying DFA to prok-propagattin weights As described above, in CNNs. DFA encounters challenges in effectively learning the necessary spatial information (Crafton et al. 2019). Similarly, in fully connected networks, DFA known to produce feature representation that deviate from those learned via pack-propagation (Nokland 2016). Moreover, although stable training can be achieved when transitioning from weights learned through DFA to back- poek-propagation, the reverse not true; switching from pack-propagation to DFA results in unstable training, and DFA fails to fully recover its performance even after large train epochs (Chu & Bacho 2024). These imply inherent difficulties in fine-tuning with DFA using pre- pre-trained weights.
DFA with weight decay. In the prior study Song et al. (021). it is analyzed that weight de- cay (Krogh & Hertz 1991) can reduce the output error in fully connected networks when used with Feedback Alignment (FA) (Lillicrap et al. 2016). Nevertheless, the analysis predominantly focuses on training of networks from seratch using FA, rather than on the fine-tuning process with DFA. This work, for the first time, examines the impact of weight decay in the context of fine-tuning with DFA. Our findings show that weight decay can be beneficial in fine-tuning with DFA, as reduces the network output error, enhancing learning performance.
3 FEEDBACK- WEIGHT MATCHING
We first discuss why the existing standard DFA (Nøkland 2016) does not behave stably in fine- tuning, based on weight aligoment (WA) and gradient alignment (GA) (Refinetti et al.; 2021). Then,
3

---

Published as a conference paper at ICLR 2026
introduce feedback-weit weight matching, which enables effective and reliable fine-tuning of DFA with two phases: 1) feedback matching and 2) weight matching. In the first phase, feedback ma- trices are reconstructed from the pre-trained weights. In the second phase, network weights re-initialized to align with the reconstructed feedback matrices. Then, fine-tuning is performed us- ing the standard DFA method.
3.1 WHY DOES DFA PERFORM UNRELIABLY IN FINE- LINETUNNIG?
Definition 3.1. (Weak Weight Alignment) Given L--aaee fully connected linear network updated (trained) with DFA (Nokland 2016), the weight parameter of the 1-th laver at the t-th training step, which is denoted as Wi Wisign becomes follows &Refinetti et al. 2021):
W = FIAÍ, WiekL= =FAAFF and W = A4,
(3)
where A = -1= Ex e (x')), and A12 17
(B) x" )BB x" x"""""""" (e"
(e
Here, F, is the feedback matrix of the 1-th layer, A and A/z2 are the alignment matrices, and B, = 2 Ag E R XNL is defined recursively using the feedback matrices only, with Ao = (Refinetti al. 2. Eq. (3) referred to as weak weight alignment (WA) (Refinetti al. 2021
representing the state where no particular relationship exists between W and F,F 1-1 and
between W J. and FT-11 At the early stage of DFA training, weak WA is naturally induced since Af22 in Eq. (3) starts with arbitrary values. However, the training proceeds, Af22 becomes proportional the identity matrix (Refinetti al., 2021), i.e., Afz2 x I, leading another state called strong weight aligmment (WA), which is defined as follows.
Definition 3.2. (Strong Weight Alignment) If Af22 OC Eq. (3) becomes the state called strong weight alignment (WA), which is defined as follows.
WicieL x F,F
W1 x FI
(4)
is known that the strong WA in Eq. (4) given F. F2 = I, implies strong gradient align- causing the gradient direction of the DFA weight, W W aligned to that of back- pookppropagttion (Rumelhart et al. 1986). Hence, strong WA leads the learning trajectory DFA to be comparable to that of back- prokkgatioonnnn with strong GA.
ment (GA) (Refinetti et al. 2021) defined Eq.
However, if the pre-trained weights are fine-tuned via existing standard DFA using arbitrary random feedback matrix F,, it becomes difficult to achieve strong WA in Eq. (4) as shown below, likely to result in sub-optimal fine-tuning performance by inducing weak GA from weak WA.
Proposition 3.3. If the pre-trained weight, W is updated via DFA with arbitrary randomfeedbacc matrices Fú, the strong WA condition Eg. (4) is unlikely to be satisfied as:
W Wi<kL F,FL1, WL
(5)
where W denotes the weight after t steps of training, starting from the pre-trained weight W
Proof. The proof is provided in Sec. A
Thus, Eq. (5) immpliss that simply applying the standard DFA to the pre-trained weight parameters is less likely induce the strong WA condition in Eq. (4)
3.2 INDUCING STRONG WEIGHT ALIGNMENT
To enable fine-tuning with DEA by deriving strong GA from strong WA defined in Eq. (4) propose the feedback-weight matching method, which induces both strong WA and GA as follows. Definition 3.4. (Feedback Matching) From the pre-trained weight W we set the feedback matrix F, such that:
we
F.F R W Wi IIKL and F FT W b-
Eq. (6) requires decomposing the pre-trained weight W Ti into F, and FT 1- It can be achieved either through traditional methods, such SVD (Singular Value Decomposition) (Klema & Laub, 1980), or alternatively. by optimizing EE. (23) as in Sec. B Once the feedback matrix F, is recon- structed as [Eq. (6) proceed to the weight matching process to induce strong WA, as follows.
4

---

Published as a conference paper at ICLR 2026
Definition 3.5. (Weight Matching) Given the reconstructed F, derived by feedback matching, as in Eq. (6) we re-initialize the pre-trained weight W into W so that it matches Fi such that:
W Wi F1F,11 and W FL
The following proposition shows that
Eq. (6) and together lead strong WA condition in Eq. (4)
Proposition 3.6. If the re-initialized weight Wi in Eq. (7) is updated using DFA with the feedback matrix F, derived by Eg. (6) the strong WA condition in Eg. (4) is induced as:
W WEKL F,F1-11 W F L-1
where Wi is the weight at step t, initialized from W.. Proof. The proof is provided in Sec. A
Thus, Eq. (8) indicates that applying bredback-weiggee matching to the weight updated from the re-
initialized weight induces the strong WA condition in in[Eq. (4) contrast to standard DFA (Eg. (5))
Subsequently, strong WA, achieved through F.q. (6) and Eq. (7) leads to strong GA (Refinetti et al.,
202T]. By matching the feedback matrix to the pre-trained weights, as in Eq. . (6) becomes possible to preserve the knowledge embedded in the pre-trained weights. Additionally, by re-initializing the pre-trained weights from the matched feedback matrices, as in Eq. (7) it becomes possible to facilitate the attainment of strong WA through DFA in fine-tuning.
3.3 INDUCING STRONG GRADIENT ALIGNMENT
While the previous section (Sec. 3.2) shows that the proposed beelback-weighh matching in Eq. (6) and (7) promotes strong weight alignment (WA), naturally leading to strong gradient alignment (GA), we show that weddback-weight matching also directly induces strong GA. We begin by formally defining gradient alignment (GA) as follows.
Definition 3.7. (Gradient Alignment) The gradient alignment (GA) is defined as the cosine simi- larity between the weight gradient obtained using DFA (Nokland 2016), denoted as GDF and the weight gradient of back- pack-propagation (Rumelhart et al. 1986), denoted as GBP, which is given by:
cos DFG GBP) = GDPA GBP/ Gap/|GDaal G
(9)
We show that feedback- weight matching, i.e., Eq. (6) and 77) also directly induce strong GA when fine-tuning the first layer of the two-layer fully connected linear network.
Proposition 3.8. Feedbuck-wetd weight matching given in Eq. (6) and ( induces strong GA, iee, a higher GA. in the first layer of a fully connected linear network, as follows:
C0SFW M ((1 W)) 2 COSDFA (F1, W!))
(10)
where COS Ms L(F1, 1, W!) refers GA in the first layer using bredback-weigh matching, while DF (F,, W is GA in the first layer using standard DFA without feehoulee feedback- matching.
Proof. The proof is provided in Sec. A
4 WEIGHT DECAY
Similar to conventional trains using back pocpppropagation (Nøkland 2016), weight decay (Krogh & Hertz 1991) has been shown to mitigate over- over-fitting of DFA, though its effect in fine-tuning has not been studied. We discuss how the proposed boelback-weightt matching helps weight decay to reduce the network enror (i.e., improving learning performance) in fine-tuning when applied to DFA.
Lemma 4.1. Given the re-initialized weight W in Eq. (7) and the pre-trained weight WO
IIIE the following terms, Fi<KL and TL, become non-negative with high probability.
PICK<L | W | W [. W - lof FI|F,FTTII /l 2 0,
(11)
PL = WW W V W W [[ W - 1ct 11[[ 0,
(12)
implying W W N W w for all <
Proof. The proof is provided in Sec.
Based on Lem. 4.1, we show that feedback- weight matching reduces the network output error e+1 over the step when combined with weight decay (Krogh & Hertz 1991).
5

---

Published as a conference paper at ICLR 2026
Proposition 4.2. Let e' denote the output error of a two-layer fully connected non-linear network (i.e., L=2) at the t-th training step. 1, is the learning rate, Y S Amin (G) is a positive constant, where G= G=E ,, ( xx)²( w Xj) with the number of neuron p and a non-linear function (-),, X is the weight-decay hyperparameter the step i, and y is the output of the network. By applying feedback-weight matching in Eg. (6) and 77. the following holds:
||o || S (1 77 7)7) + + -02T2
(13)
for all t 2 and constant Ozg, with r2 defined in Eq. (1I)
Proof. The proof is provided in [Scc. A]
Conjecture 4.3. Given an L-llver fully connected non-linear network, suppose that the right-hand
side of the inequality in Eg. (13) i.e., (1- 4 m)a) ||e + X |yll, contains Wil as linear
components for some I S L. Then, based on Prop. 4.2 and Lem. 4.1, it is conjectured that
Eq. (13) can be generalized into:
L Σ I
||e S
"7
X |yll
am
(14)
4
with constants on, and 1i defined in Eq. (11) and 72) for some
< < L and all 0.
From Eq. (13) and subsequently F.q. (14) it be seen that the proposed feedback-weigh matching
preserves the weight decay effect by decreasing the network error ||e || by the quantity 17*
X||v|l It is achieved by Σ/=2 arr, which effectively counteracts the adverse impact of weight decay, namely, the increase in error ||e/+1|| when ll S lf arri 2 X||y||| - [lell.
Due to the dependence of the term || [ Wlll the reconstructed weights, deriving theoretical analysis is challenging. Nonetheless, the effect of weight decay on error reduction is empirically validated through our controlled experiments, as detailed in the experimental section.
5 EXPERIMENT
We evaluate the proposed feedback feeigack-weight matching on wide range of fine-tuning tasks. To the best of our knowledge, this is the first extensive experiments to investigate fine-tuning of DFA across diverse tasks and models, which is enabled by the proposed feedhack-weigh matching.
First, feed-weight matching is applied to image classification tasks using two fully connected networks with and layers, respectively. These networks are pre-trained with CIFAR- 100 (Krizhevsky et al., (2009) and Tiny (Le & Yang. 2015) using back- propppropagttion (Rumel- hart et al.] 1986), and then fine-tuned on CIFAR- (Krizhevsky et al. 2009). SVHN (Netzer et a 2011). and STL-10 (Coates et al., 2011) through DFA applying feedback-
matching. Next, apply it to NLP tasks using Transformers, i.c., BERT-Tiny and Small (Kenton & Toutanova] 2019 BookCorpus (Zhu al. 2015) & Wikipedia, and then fine-tuned with GLUE tasks (Wang, 2018). Lastly, we apply feedback weightt matching to Vision Transformer (ViT) models, i.e., ViT-Tiny and ViT-Small (Wu et al. 2022). Each model utilizes pre-trained weights learned from ImageNet (Deng et al. 2009) and is fine-tuned on CIFAR-10 (Krizhevsky) al. 2009), STL-10 (Coates al. 2011). and ImageNette (Howard, 2019). It is important to note that even standard DFA has rarely been applied to Transformer models for scrathh training due to its inherent challenges and difficulties (Launay et al., 2020). Our experiment is the first attempt to apply DFA fine-tuning to Transformers (i.e., BERT, ViT), which is considered more challenging than from-scratch DFA training. The detailed experimental setups are provided in Sec.
Ture al. 2019), pre-trained
5.1 FINE-TUNING PERFORMANCE
Tab. summarizes the fine-tuning performance on image classification tasks (i.e., test accuracy) of feedback-weight matching compared against the standard DFA fine-tuning that does not apply feedback-weight matching. As shown in the table, the proposed feedback-weight matching enables reliable fine-tuning for various network architectures and tasks, which consistently outperforms stan- matching achieves
dard DFA with an average of 2. 16% accuracy gap. For instance, feedback- the 6-layer network from CIFAR- 100 to SVHN which is 7.97%
82.67% accuracy when higher than standard DFA that achieves 74. 74.70% It also indicates that the proposed feedback- weight matching maintains more robust performance over network depths, whereas the performance of standard DFA deteriorates with deeper networks. For instance, in the case of fine-tuning from

---

Published as a conference paper at ICLR 2026
Table I: Image classification tasks. The fine-tuning performance of feedback-weight matching (DFAous) on the 4- and 6-layer fully connected networks, compared with standard DFA fine-tuning (DFAine), and from-scratch-trainigg of DFA (DFAscratc)) The pre-trained weights are obtained through back proppagaoogatonn The bold indicates the best performance in DFA fine-tuning.
4-layer
6-aayer
Target Data
Scratch
CIFAR-100
TiyyInageNet
Scratch
CIFAR-100
TinyImageNet
DFAwsath
DFAGs
DFA..
DFAi
DFAu.
DFAurach
DFA
DFA.us
DFA
DFA..
CIFAR-10
52.78
53.79
55.38
56.75
55.51
51,94
53.04
55.39
51.08
55.54
SVHN
82,93
79.55
82.87
80.31
83.16
81.89
74.70
82.67
76.03
81.39
STL-110
42.20
44.83
45.30
50.62
45.61
40.48
43.42
45.28
43.33
45.21
CIFAR- 100 to SVHN, the accuracy drop between 4-layer and 6-layer networks is only 0.20% with weedback-weight matching, which is 24x smaller than the case not applying it (i.e., 4.85% drop).
Tab. presents the evaluation results of feedback-weight matching applied to BERT-Tiny and BERT- Small, fine-tuned for NLP tasks, using the same experimental setup in image classification tasks Tab. D). Similar to image classification tasks, feedback -eedback-weight matching enables DFA to fine-tune BERT for various tasks of the GLUE dataset in more robust manner compared to standard DFA. In particular, substantial performance gains are observed on datasets with limited samples, where fine- tuning primarily depends on pre-trained weights. For example, CoLA,, Bodbackk-eiigh matching achieves a Matthews correlation of 0.53 in BERT-Small, compared to 0.06 with standard DFA. Simi- larly, on STSB, BERT-Small achieves Pearson correlation of 0.76 with feedback-weight matching, while standard DFA yields only 00, demonstrating significant gap in both learning performance and reliability. In the worst case, standard DFA fails learn from the fine-tuning data entirely, achieving 0.00 Matthews correlation for CoLA with BERT-Tiny, whereas feedback- weight match- ing achieves 0.29. Unlike architectures with stacked fully connected layers, transformers incorporate attention mechanisms, which interfere with alignment. While alignment cannot be directly adjusted for projection layers such as key, query, and value, aligning the subsequent dense layers alone sig- nificantly improves weight alignment, gradient alignment, and overall performance. Experiments related to this are presented in Sec. E
Table NLP tasks. The fine-tuning performance of feedback- feedback-weiihht matching (DFAwurs) on Trans- former architectures (i.e., BERT-Timy and BERT-Small), compared with standard DFA fine-tuning (DFAnne). The pre-trained weights are obtained via baok-propagation (BP). For reference, we also present the from-scratch--rainne results of DFA (DFAwcrtth). The bold indicates the best perfor-
mance in DFA fine-tuning.
CoLA
SST-2
MRPC
QQP
MNLI
QNLI
STSB
RTE
WNLI
Model
Training
(mat)
(ace)
(acc)
(acc)
(ace)
(acc)
(pear)
(acc)
(ace)
DFAsensch
0.00
95.2
67.4
81.2
59.2
84.2
-0.11
50.2
50,0
BERT Tiny
DFA
0.00
92.4
67.4
80.6
60.0
80.2
-0.
51.2
51.0
DFAvurs
0.29
95.9
69.7
82.3
60.2
84.3
0.36
55,5
52.6
DFAaemtchh
0.19
96.5
75.2
86.7
67.4
80.9
0.05
60.0
50.3
BERT
Small
DFA
0.06
95.6
70.9
86.0
67.0
85.3
0.10
59.0
49.3
DFAcurs
0.53
97.3
92.5
86.9
65.8
87.2
0.76
59.0
51.0
shows fine-tuning results for image classification tasks using Vision Transformers (ViT'). Consistent with previous evaluations, the experiment results confirm that beadback-weighht matching effective not only for simple fully connected networks but also for complex Transformer architec- tures, e.g., the classification accuracy improves from 0.210 to 0.319 for ViT-Small on ImageNette.
Tab. 3
Table 3: Image classification tasks with Vision Transformers. The fine-tuning performance of feedback-weit weight matching (DFAgus) on ViT-Small and ViT-Tiny compared with standard DFA fine- tuning (DFA and DFA training from scratch (DFAscratch) Bold indicates the best performance.
ViT-Tiny
ViT-Small
Target Data
DFAggratt
DFAge
DFAges
DFAggacch
DFAgn:
DFAgus
CIFAR-10
0.281
0.332
0.397
0.322
0.378
0.392
STL-11
0.197
0.164
0.247
0.221
0.111
0.247
Iette:Nettee
0.168
0.209
0.294
0.230
0.210
0.319
5.2 WEIGHT ALIGNMENT AND GRADIENT ALIGNMENT
Fig. 2a and 2b plot the weight alignment (WA) and the gradient alignment (GA), along with the train and test accuracy, for some fine-tuning setups. As shown in the figures, the proposed feedback-
7

---

Published as a conference paper at ICLR 2026
Weight Matching (DFA
DFApre
DF Aggnschh
Weight Aligmment (WA)
Gradient Aligament (GA)
Train Accuracy
Test Accuracy
100
85
0.9
0.9
0.5
0.5
85
80
75
70
OK
Epoch
Epoch
Epoch
Epoch
(a) Fine- mun-ttuning the 4-layer network from TinylmageNet SVHN.
Weight Alignment (WA)
Grudient Aligament (GA)
Train Accuracy
Test Accuracy
0.9
0.9
95
95
0.6
0.6
85
85
0.3
75
75
0.0
0.0
65
65
Epuch
Epuch
Epoch
Epach
(b) Fine-tuning BERT-Sman from BookCorpus & Wikipedia to MRPC (GLUE).
Figure 2: WA, GA, train accuracy, and test accuracy. The green graph denotes DFA fine-tuning with feedback-weight matching (ours), yellow denotes DFA fine-tuning without feedback-weight matching, blue DFA trained from scratch.
weight matching (green) induces strong weight alignment (WA) from the outset, subsequently strong gradient alignment (GA) as analyzed in Sec. 3.2 and 3.3 leading to both enhanced train and test contrast, standard DFA (yel-
accuracy across all experiments with faster and stable convergence. low), not applying feedback- weight matching, achieves significantly lower WA and GA. While they gradually increase over fine-tuning epochs in some cases, the initially low WA and GA impede effective fine-tuning. As a result, the train and test accuracy of standard DFA do not improve sub- stantially from the pre-trained weight parameters, especially for BERT-Small. This suggests that standard DFA struggles to adapt to the target dataset for fine-tuning, likely due the mismatch between its random feedback matrices and the pre-trained weights. In other words, it overly re- lies on pre-trained weights in the hope that they will fit and perform well on new target fine-tuning data. These results indicate that the improvements in fine-tuning performance are attributed to the proposed weight matching. This is well exemplified in Fig. 26) where the train and test accuracies of DFAine show minimal or no improvement from the pre-trained weights, whereas those of DFA exhibit notable increases of approximately up to 0.27 from the pre-trained weights. In ad- dition, Transformer models achieve lower WA and GA than fully connected networks when trained with DFA from scratch. This is potentially due to the complexity of the attention operation (Sec. E). When the proposed feedback-wett weight matching is applied to Transformer models, it induces WA and GA, yielding better performance compared to fully connected networks.
5.3 ABLATION STUDY: FEEDBACK MATCHING, WEIGHT MATCHING, AND WEIGHT DECAY
Tab. 4 presents the impact of feedback matching, weight matching, and weight decay on fine-tuning with DFA. To assess their effectiveness, we remove each of them in isolation. Removing feedback matching results in marginal performance decline, such as reduction from 55.54% to 55.03% when the 6-layer network is fine-tuned from TinylmageNet to CIFAR-10. This marginal drop occurs because bypassing feedback matching applies random feedback matrices to the re-initialized weights that are amenable to arbitrary random feedback matrices, resulting in a reasonable level of WA and GA. In contrast, omitting weight matching leads to relatively bigger performance drop, e.g., classification accuracy decreases from 83. 13.1 to 79.77% when fine-tuning the 4-layer network from TinylmageNet to SVHN. Similarly, the correlation score drops from 0.76 to -0.06 when fine-tuning BERT-Small to STSB as shown in Tab. 6 (Scc. D). It is presumed that excluding weight matching causes the pre-trained weights obtained by back back-propagation mismatched feedback matrices, thus resulting in weak WA and GA.
not by DFA, to be fine-tuned with
When weight decay is not applied, fine-tuning of weedback-weight matching performance also ex- hibits some declines, e.g., classification accuracy decreases from 55.38% to 48.82% when fine- tuning the 4-layer network from CIFAR-100 to CIFAR-10. It should be noted that weight decay appears to have minimal impact fine-tuning of standard DFA when feedback- weight matching is
8

---

Published as a conference paper at ICLR 2026
Table 4: Ablation experiment. The fine-tuning performance when removing each component of weeback-weeigt matching: weight matching weFAweight), feedback matching (DFAfeed*) and weight decay (DFAdccay*). 'DFAours denotes applying all of them.
Source Data
Model
Target Data
CIFAR- 100
TinylmageNet
DFAxeighe
DFArer
DFAgecas
DFAqus
DFAgsi
DFAg
DFA
DFA
CIFAR-10
53.92
55.23
48.82
55,38
53.73
55.05
48.66
55.51
4 layers
SVHN
80.65
81.34
77.99
82.87
79.77
83.13
77.63
83.16
STI-10
44.25
45.20
40.00
45.30
44.05
45.42
40.47
45.61
CIFAR-10
53.47
55.03
46.21
55.39
53.50
55.03
45.77
55.54
layers
SVHN
79.70
82.53
76.71
82.67
79.77
82.76
76.76
82.72
STL-10
43.86
45.42
39.17
45.28
43.78
45,43
40.23
45.21
not applied: the classification accuracy even increases, such as from 54.38% 56.75% when fine- tuning the 4-layer network from TinyImageNet to CIFAR-10. This demonstrates the synergistic effect of feedback-weight matching and weight decay, i.e., reducing network error as in Sec.
5.4 FREDBACK- BEEDBACK-WEIGHT MATCHING AND WEIGHT DECAY
To evaluate the impact of wredback-weight matching on weight decay, we measure the fine-tuning performance with weight decay, with and without applying feedback-weight matching, which is shown in Tab. 5] The results indicate that weight decay enhances fine-tuning accuracy (reducing network output error) when used with beedback-weigh matching, with an average improvement of 8.35%. This demonstrates that feedback-weight matching facilitates weight decay in reducing network output error, thus improving fine-tuning accuracy, as provided in [g. (14) In contrast, weight decay is less likely to improve fine-tuning performance without feedback weight matching. In fact, when applied to the standard DFA (not applying feedback- weight matching), weight decay results in fine-tuning accuracy with minimal variation, providing similar accuracy.
Table 5: Feedback- Weight Matching and weight decay. 'DFAsne' applies weight decay without weedback-weight matching, compared with 'DFA applying both weight decay and feedback- weight matching. The following tables show the results for image classification and NLP tasks.
(a) Fine-tuning image classification tasks (fully connected networks)
4 layers
layers
Target Data
CIFAR- -00
TinyImageNet
C10AR-100
TinyImageNet
DFAm
DFA
DFAm
DFA
DFA
DFA.u.
DFA
DFA..
CIFAR-10
54.39
55,38
54,38
55.51
54.08
55,39
53.50
55.54
SVHN
80.77
82.87
80.74
83.16
78.73
82.67
79.57
82.72
STL-10
45.00
45.30
50.40
45.61
43.56
45.28
45.28
45.21
(b) Fine- tineittning NLP tasks (BERT)
Model
Training
CoLA (mat-cor)
SST-2
MRPC
QQP
MNLI
QNLI
RTE
(acc)
(acc)
(acc)
(ace)
(acc)
(pearson)
(acc)
(ace)
DFAdse
0.00
92.4
67.4
80.6
60.0
80,2
-0.17
51.2
51.0
TTy
DFAur
0.29
95.9
69.7
82.3
60,2
84.3
0.36
55.5
52.6
DEA
0.05
95.6
70.9
86.0
67.0
85.3
0.10
59.0
49.3
BERT-Small
DFAuus
0.53
97.3
92.5
86.9
65.8
87.2
0.76
59.0
51.0
plots the weight alignment (WA), gradient alignment (GA), training accuracy, and test ac- curacy across varying strengths of weight decay during the fine-tuning 4-layer network from CIFAR- 100 to CIFAR-10. Feedback-weight matching ensures strong WA and GA as discussed in Sec. 3.2 and 3.3 from the beginning, which helps mitigate alignment degradation, while exhibiting varying behaviors depending on different levels of weight decay. In the absence of weight decay (black curve), GA declines and exhibits significant oscillations, ultimately causing decrease in test accuracy. Conversely, when strong weight decay is applied (blue curve), both WA and GA de- crease sharply, followed by substantial reductions in both training and test accuracy. This supports [Cojj. 4.3 that applying an appropriate level of weight decay mitigate its adverse effect (the in- crease in error), thus leading to an overall error reduction. These observations suggest that proper weight decay strength is crucial for effective fine-tuning (green curve) of feedback-wed -weigt matching.
Fig.

---

Published as a conference paper at ICLR 2026
Weight Alignment (WA)
Gradient Aligument (GA)
Train Accuracy
Test Accuracy
100
1.0
1.0
55
0.8
0.8
45
40
0K
4K
2K
08
4X
2K Epoch
Epoch
Epoch
Epoch
Figure 3: WA/GA and train/test accuracy on various weight decays (0, 5c-4, le-3, le-2). A 4-layer fully connected network is fine-tuned from CIFAR-106 to CIFAR-10 by feedback-weight matching.
ACKNOWLEDGMENT
This work was supported by the Institute of Information & communications Technology Planning Evaluation (ITP)) grant funded by the Korea government (MSIT) (No.RS-2024400500466 and RS- DS-2025-25442469)
THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used large language models solely for polishing grammar and improving the readability of the manuscript. The contents and research contributions were written entirely by the authors.
REFERENCES
AF Agarap.
Deep learning using rectified linear units (relu). arXiv preprint arXiv: 1rXhv:1803.88375
2018.
Dominique Chu and Florian Bacho. Random feedback alignment algorithms to train neural net- works: why do they align? Machine Learning: Science and Technology. 5(2):025023, 2024.
Kenneth Ward Church, Zeyu Chen, and Yanjun Ma. Emerging trends: A gentle introduction to
fine-tuning. Natural Language Engineering, 2766)77637 2021.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215 123. JMLR Workshop and Conference Proceedings, 2011.
Brian Crafton, Abhinav Parihar, Evan Gebhardt, and Arijit Raychowdhury. Direct feedback align- ment with sparse connections for local learning. Frontiers neuroscience, 13:525, 2019.
Francis Crick. The recent excitement about neural networks. Nature, (37 1989.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-24822 255. Ieee, 2009.
Jacob Devlin, Ming- bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.04805 2018.
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
Shivam Garg and Santosh Vempala. How and when random feedback works: case study low- rank matrix factorization. In International Conference Artificial Intelligence and Statistics, pp. 4070-4108. PMLR, 2022.
Stephen Grossberg. Competitive learning: From interactive activation to adaptive resonance. Cog- nilive science, 11(1):23-66 1987.
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfim: factorization- machine based neural network for ctr prediction. arXiv preprint arXiv:1703: 04247, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1024-1034, 2015. 2015.
10

---

Published as a conference paper at ICLR 2026
Dan Hendrycks and Kevin Gimpel. arXiv:1606.088 00410, 2016.
Gaussian enror
linear units
(gelus).
arXiv preprint
Jeremy Howard. Imagenette, 2019, http:://ggttuub.mm/aaataaaaaaaaa Accessed: 2025-01-26.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of naacL-IHLT, volume pp. 2, 2019.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Virginia Klema and Alan Laub. The singular value decomposition: Its computation and some appli- cations. IEEE Transactions automatic control, 25(2):164-16, 1980.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Anders Krogh and John Hertz. A simple weight decay improve generalization. Advances in neural information processing systems, 4, 1991.
Julien Launay, Iacopo Poli, and Florent Krzakala. Principled training of neural networks with direct feedback alignment. arXiv preprint 1906. 2019.
Julien Launay, lacopo Poli, François Boniface, and Florent Krzakala. Direct feedback alignment scales to modern deep learning tasks and architectures. Advances in neural information processing systems, 33:9346-9360 2020.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature communications, 7(1): 13276, 2016.
Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin Akerman, and Geoffrey Hinton. Back- propagation and the brain. Nature Reviews Neuroscience, 21(6))335-346, 2020.
I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXi::1711.05104 2017.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.088433 2016.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. Advances in neural information processing systems, 29, 2016.
Maria Refinetti, Stéphane d' Ascoli, Ruben Ohana, and Sebastian Goldt. Align, then memorise: the dynamics of learning with feedback alignment. In International Conference on Machine Learning. pp. 8925-8935. PMLR. 2021.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747. 2016
David E Rumelhart, Geoffrey Hinton, and Ronald J Williams. Learning representations by back- propagating errors. nature, 523(6088):533--36 1986.

---

Published as a conference paper at ICLR 2026
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 2014.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein, Scene representation networks: Con- tinuous 3d-structure-aware neural scene representations. Advances in Neural Information Pro- cessing Systems, 32, 2019.
Ganlin Song, Ruitu Xu, and John Lafferty. Convergence and alignment of gradi- ent descent with random backpropagation weights. In M. Ranzato, Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.). Advances in Neural In- formation Processing Systems, volume 34, pp. 19888-19898. Curran Associates, Inc.. #.les/paper/2021/
2021. URL httpp::/pppeeel..g....re.......s
File/ F1124777776aaiaa1000SSS0A7777712
Iulia Turc, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better:
On the importance of pre-training compact models. arXiv preprini arXiv: wrrXiv:/08.000 1908.08962, 2019.
Ashish Vaswani. Attention is you need. arXiv preprint arXiv:/706.03766 2017.
Alex Wang. Glue: multi-task benchmark and analysis platform for natural language understand-
ing. arXiv preprint arX1v:1804.07461, 2018.
Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers, 2022. URL https:// //
arxiv. org. abs 2207.10666
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. The IEEE International Conference on Computer Vision (ICCV), December 20155
12

---

Published as a conference paper at ICLR 2026
A PROOF
A.1 PROOF OF PROPOSITION 3.3
Proof. We prove Prop. 3.3 for W WickE in Eq. (15) and the same reasoning applies Wi, in 16 Since A122 in Eq. (3) becomes such that A I as the training proceeds (Refinetti et al 202
the weight newly updated with DFA, which is denoted as W W comes to satisty Eq. (4) i.c.,
Wi W = F with some constant et. Given that we take the pre-trained weight W i<t,,, as the initial training point in our fine-tuning the overall weight W WickE obtained by DFA is expressed
as the sum of W i<<< and W
i<i< which is given by:
Wick W Wick + W Wi + &FF.LIXEEL
(15)
W w W W F
(16)
where Gieig is constant. In Eq. (15) since W Wikk is unlikely to be proportional F
ie., Wi<ter X FiFT_, the overall weight W Wi Wideer which includes W Wicter is also unlikely
to be proportional to F2F i.e., W F though Wt
ieleL FFFFT x F,F
Hence, Eq. (15) can hardly induce strong WA in Eq (4)
A.2 PROOF OF PROPOSITION 3.6
Proof. Similar to Eq. (15) and 16). the overall weight Wi obtained by DFA is the sum of W and Wi. Specifically, now that W Wiekez K<tK F,F,, and W L-11 these become proportional to
FIF,, and Fi.-1, respectivlyy as:
Wieter = WieiEt + W Wiek<- = + (1+ (1 & F,T,,
(17)
W1 = WT + W = F + F = + F F[
(18)
with constants ci<tg which aligns with the strong WA condition in Eq. (4)
A.3 PROOF OF PROPOSITION 3.8
Proof. The weight at the second layer of the network, W can be expressed with the pre-trained weight, W with the learning rate the number of neurons p. Fi E RP, and W E R2 as follows (Song et al, 202T).
W = W
W W W V
W e
(19)
2=
For the standard DFA that does not apply feedback- weight matching Eq. (6) and we have GDFA F1 and GBp = W. By using Eq. (19) the gradient alignment (GA) defined in Eq. (9)
between them, which is denoted as COSDFA 2(Fi, 1, Wt). at least as follows.
COSDFA ((F1, W;)
F
W
FF
F
W
F1
F
(W9 E W e¹)
F1
W
|
W.
2 VP
EE
WiX e'll
(20)
F
W
2
W
ee)
F
vp
+ +1 Ei WiXTe"
Conversely, when applying feedback feedback-weight matching in Eq. (6) and we have F₁ W for 1=22, Using Eq. (19)again, GA between them, cos M
Wh) (2).
least as follows.
F [Fll (W! 3 E''I w e¹) n. vp
F
3 Σ'-
wix T et)
COS LF, W/
[AT
W
W W el
F +1
W
T e'l
(21)
If we assume that both F, and W follow the standard Gaussian distribution, have
| w S
F 212 (Song et al. 2021). Thus, cospw L(F,, Wh) exhibits a higher lower bound compared to DF I.e., C FF, 1: W!) 2 COSDFA ((F1, W?), implying higher GA.
13

---

Published as a conference paper at ICLR 2026
A.4 PROOF OF LEMMA 4.1
Proof. We show that Fi<k<L 2 0 in Eq. (11) and the same reasoning extends to ry in 12 Given that W7 F,F1_1 x W - we can interpret Wi as scaled version of W which implies that || Will is small. Conversely, since Wi is not proportional to W i..,
WT x W F follows that
[ W is generally larger than W - -W]l- There-
fore, W W is likely smaller than W Will.
A.5 PROOF OF PROPOSITION 4.2
Proof. It is shown (Song et al. 2021 that the inequality in Eq. (13)
i.e., ||e**1| - (1 m
IA*) + >llyy holds for two-layer fully connected non non-lineee linear network when applying FA (Feedback Alignment) (Lillierap et al., 2016) with weight decay (Krogh & Hertz] 1991). Specifi-
cally, the right-hand side of the inequality, i.e., (1 (1-25 27 -n--nn)) -n + consists of the following
term as linear component in fine-tuning:
W
W s.t. W F
(22)
where W is the pre-trained weights. By assuming that W is replaced with the re-initialized
weights, W in Eq. (7), ||e+1] in
Eq. (13) reduced by 0222 since [W;- WW-WWll2 2- [
W
as
in Lem, 4.1
B DECOMPOSITION OF WEIGHT INTO FEEDBACK MATRICES
One way of finding feedback matrices F, and in Eq. (6) from W i<ik- other than SVD (Singular Value Decomposition) (Klema & Laub 1980), is to optimize the following objective LrM-
1 CFM 7 Σ (w -. h + ,
(W FZ-1h h 2) 1 Σ 1 (I-F F1)
(23)
Here, LFM is minimized ensure that the layer output, when replaced by the feedback matrix F,F,,, hi-1, matches the output obtained using the pre- pre-trained weight Wh while F, is to be orthogonal itself in accordance with the regular DFA condition (Lillierap al. 2016).
C LIMITATIONS AND FUTURE WORKS
Extending to different architectures. Although this study presents the significant potential of fine- tuning with DFA, its current application is restricted to fully connected networks. This limitation arises because, present, DFA is predominantly effective for fully connected architectures, and further research is needed to extend its applicability to other network types. In our future work, we plan to explore the application of DFA fine-tuning various network architectures, such CNNs. Meanwhile, we anticipate the development of more generalized methods that will enable DFA to be applied across broader range of network types, thereby enhancing the applicability of our work.
Improving learning performance. The learning performance of the proposed feedback-weight matching shown to surpass both 1) training networks with DFA from scratch and 2) fine-tuning networks with DFA using random feedback matrices. While fine-tuning with DFA applying the proposed method achieves superior and more stable performance compared to them, it still falls short of the performance achieved with fine-tuning using back proppaaopanntoon (Rumelhart et al.. 1986). We plan to explore how to achieve fine-tuning performance comparable to that of back maok-propggttion by investigating DFA from its fundamental mechanism, along with the proposed method.
Proving hypotheses. This work provides some hypotheses regarding fine-tuning and weight decay in the context of DFA. Conj. 4.3 posits that applying the proposed method to weight decay en- hances fine-tuning performance DFA for fully connected networks of arbitrary layers. However, formal proofs necessary substantiate these hypotheses and validate the efficacy of the pro- posed approach. In future research, we intend to generalize the propositions presented in this study encompass various types of fully connected network architectures.
14

---

Published as a conference paper at ICLR 2026
D ABLATION EXPERIMENT ON BERT
Tab. 6 presents the fine-tuning performance of BERT models when weight matching, feedback matching. and weight decay are individually removed. It is important to note that DFA is not applied to all fully connected layers in BERT, which limits the ability to properly assess the effectiveness of beedback-weight matching. Thus, this experimental setup may provide accurate evaluation.
Table 6: Ablation experiment. The fine-tuning performance when removing weight matching weight) feedback matching (DFAf and weight decay (DFAdocay*). *DFA denotes ap-
plying all of them.
CoLA
SST-2
MRPC (acc)
QQP
MNLI
QNLI
STSB (pearson)
RTE
WNLI
Model
Training
(mat-cor)
(acc)
(acc)
(acc)
(acc)
(acc)
(acc)
DFAuciga
94.7
81.4
-0.15
50.9
DFAted
0.00
95.8
68.9
82.4
60.8
86.9
0.35
50.0
BERT-Tiny
DFAasa
0.31
95.9
71.4
81.9
61.0
83.3
0.36
53.3
51.9
DFAuus
0.29
95.9
69.7
82.3
60.2
84.3
0.36
50.8
52.6
DFAgrigh
0.08
96.0
75.1
85.0
79.7
-0.06
61.8
50.1
DFA
0.54
97.0
91.5
87.4
65.2
85.3
0.75
62.0
50.2
BERT-Small
DFA 0.53
97.2
91.2
87.1
64.7
85.4
0.78
68.7
50,9
DFAmm 0.53
97.3
92.5
86.9
65.8
87.2
0.76
59.0
51.0
E LAYER-WIL WAYER-WISE GRADIENT ALIGNMENT ANALYSIS
Weight alignment and gradient alignment have generally been analyzed in the context of sequen- tial fully connected layers. However, Transformer models introduce attention mechanisms, which disrupt the sequential structure of fully connected layers. To investigate the impact at- tention our method, analyze layer-wise gradient alignment across different components of the Transformer. Figure illustrates the average gradient alignment each sequential fully con- nected layer in BERT model trained the GLUE dataset. The key, query, and value layers in the attention module function as usual. As result, the fully connected layer following the atten- tion operation exhibits notably lower gradient alignment compared others. This indicates that the attention operation interferes with gradient alignment and highlights the need for additional archi- tectural considerations tailored the attention module. Although our method alone improves both gradient alignment and performance in the sequential fully connected layers of Transformer models, further enhancing alignment within the attention module leads even greater performance gains.
Feedback Feeihack-Weight Matching (DF Aours)
DF DFArie
DF DFAscratch
Attention layer
Intermediate layer
Output layer
Grad Alignment
0.25
0.15
0.15
0.15
0.05
0.05
0.05
-0.05
.0.05
-0.05
2
3
5
5
2
Epoch
Epoch
Epoch
Figure 4: Gradient alignment comparison attention, intermediate, and output layer. The green graph denotes DFA fine-tuning with feedhack-weight matching (ours), yellow denotes DFA fine-tuning without feedback-weight matching, blue is DFA trained from scratch.
F COMPARISON WITH BACK-
To demonstrate the effectiveness of beudback-weight matching in enhancing DFA-based fine-tuning, presented empirical results in the previous sections. These results indicate that have success-
15

---

Published as a conference paper at ICLR 2026
fully achieved our objective of enabling and improving DFA-based fine-tuning. But, it still falls short of the performance achieved by backpropagation. In this section, we provide a detailed comparison between DFA and propppaopgattion in terms of fine-tuning performance. This analysis helps po- sition DFA fine-tuning within the broader landscape of learning algorithms and clarifies its current limitations and future potential.
For fair comparison, we use the same pre-trained weights and experimental setup as in the DFA experiments. The experimental details in [Sec. I We denote fine-tuning with backpropagation as Biee. and training from scratch with backpropagation as BP scratchh The performance of feedback- weight matching does not yet reach that of back propagation fine-tumi Line-tuning. However, is often com- parable to, and occasionally surpasses, the performance of back proppgaatonnonn trained from scratch. For instance, in the case of training on STI.-10 with a fully connected architecture, feedhack-wett matching can outperform back-proy propppropagttion from scratch.
Table 7: Image classification tasks with back-propagation feedback-weit weight matching (DFA on the 4 and 6-layer fully connected networks, compared with standard DFA fine-tuning (DFA and back proppaapanntiion fine-tuning (BPfine). The pre-trdi weights are obtained through back-propy pack-propagation (BP). For reference, we also present the from- scratch-training results of pack-propagation (BP (ePsehtch) and DFA (DFAsemtch). The bold indicates the best performance DFA fine-tuning
The fine-tuning performance of
Source Data
Model
Target Data
Scratch
CIFAR-I00
Tiny ImageNNt
DFAuemchh
BP
DFAfee
DFAaus
BP
DFA
DFAcors
CIFAR-I0
55.48
52.78
57.16
$3.79
55,38
57.66
56.75
55.51
4 layors
SVHN
85,10
82.93
84.32
79,55
82,87
84.69
80.31
83.16
STL-10
43.15
42.20
47.73
44.83
45,30
50.29
50.62
45.61
CIFAR-I0
54,93
51.94
58.85
53.04
$5.39
55.97
51.08
55.54
6 layers
SVHN
85.10
81.89
84.34
74.70
82.67
84.72
76.03
81.39
STL-10
43.10
40.48
47.78
43.42
45.28
47,63
43.33
45.21
Table 8: NLP tasks with back-propagation. The fine-tuning performance of feedback-weight matching (DFA on Transformer architectures (i.e., BERT-Tiny and BERT-Small), compared with standard DFA fine-tuning (DFA and pekk--rrrptooo-baaaee based fine-tuning (BP The pre- trained weights are obtained via back-propagation (BP). For reference, we also present the from- scratch-training results of back poc-ppropagation (BP scratch) and DFA (DFAseratch) The bold indicates the best performance DFA fine-tuning.
CoLA
SST-2
MRPC
QQP
MNLI
QNLI
STSB (pearson)
RTE
WNLI
Model
Training
(mat-cor)
(acc)
(acc)
(acc)
(acc)
(acc)
(acc)
(acc)
serach
0.07
96.3
67.4
82.8
63.4
89,2
-0.19
64.1
50.0
BP tine
0.00
93.5
70.7
86.9
73.8
88.2
-0.25
60.3
52.6
BERT-Tiny
DFAuznch
0.00
95.2
67.4
81.2
59.2
84.2
-0.11
50.2
50.0
DFAmm
0.00
92.4
67.4
80.6
60.0
80.2
-0.17
51.2
51.0
DFAaus
0.29
95.9
69.7
82.3
60.2
84.3
0.36
55,5
52.6
BPannch
0.55
96.3
95.4
91.3
75.3
93.4
0.67
89.8
51.9
BPne
0.87
98.9
96.7
98.0
93.0
99.1
0.90
94.0
53.3
BERT-Small
DFAuma
0.19
96.5
75.2
86.7
67.4
80.9
0.05
60.0
50.3
DFAin: 0.06
95.6
70.9
86.0
67.0
85.3
0.10
59,0
49.3
DFAAes 0.53
97.3
92.5
86.9
65.8
87.2
0.76
51.0
G RECOVERY ABILITY OF THE FEEDBACK MATRICES
We evaluate the resilience of feedback matrices in the Feedback- Weight Matching process. The feedback matrices are designed to mimic the existing pre-trained weights and preserve their informa- tion. Accordingly, we examine the extent of performance degradation when re-initialized weights, generated using the trained feedback matrices, are evaluated on the pre-traine pre-trained dataset CIFAR- 100. Tab. reports the performance of pre-trained weights and re-initialized weights obtained from the trained feedback matrices on CIFAR-100. For most classification tasks, the layer dimensions are larger than the number of classes. Consequently, feedhack matrices with dimensions equal to the
16

---

Published as a conference paper at ICLR 2026
Table 9: Comparison between Pre-trained and FWM weights on CIFAR-100
Pre-trained Weight FWM Weight
CIFAR-100
32.38
26.86
Table 10: Results according to epoch within the Feedback- process
Dataset
DF
DFAmid
DFAaur
CIFAR-I0
51.38
53.27
55.38
number of classes have lower rank than the pre-trained weights. This inevitably leads to some loss of information, resulting in slightly reduced performance of the re-initialized weights compared to the pre-trained weights.
To investigate how the training of feedback matrices affects fine-tuning performance, we measured fine-tuning results at different epochs in the Fredback-Matching process. In Tab. 10, DFA corre- sponds to epoch, DFAmia to 2 epochs, and DFAuurs to 3 epochs of feedback matrix training, which matches the original setting- When the feedhack matrices are less thoroughly trained, fine-tuning performance decreases. This indicates that insufficiently trained feedback matrices fail to capture the information of the pre-trained weights, leading to reduced performance.
H TRAINING COST
The training cost of our method does not differ significantly from that of standard DFA, as train- ing proceeds using standard DFA after re-initializing the feedback matrices and weights. Standard DFA theoretically enables layerwise parallel training and can achieve speed-up proportional to the number of layers compared to back- propaaappgg,,ion, but improving its training efficiency is not the fo- of this work. Consequently, during the fine-tuning phase, training speed and memory overhead remain equivalent to those of standard DFA. Furthermore, the overhead introduced by the feedback matching stage is minimal, as is conducted for only three epochs.
Although DFA theoretically supports layer-parallel updates, implementing true parallelism within single model remains challenging. Existing parallelization methods mainly focus on distributing data or model components across multiple GPUs, whereas they do not readily support inter-layer parallel execution on single GPU. our implementation, we approximate the behavior of DFA by splitting the backward graph at each layer and substituting the local gradient with the DFA error signal. This design preserves the learning dynamics of DFA, yet the update process still proceeds sequentially as in standard back prok-propagttion.
To illustrate the potential efficiency gains of parallel DFA, we report per-layer backward computa- tion times measured on six-layer MLP with 1000 hidden units trained on CIFAR-100 in Tab. 11 In propaarrpa,,,ion, lower layers must wait for the computations of upper layers, which results in cumulative backward times. In contrast, DFA can, in principle. update all layers simultancously so that the total update time would be determined only by the slowest layer. Although full paral- lelization is not implemented, the per-layer measurements provide an upper bound the possible speed-up.
Table 11: DFA BP layer-wise and total backward time comparison (ms). Layer-wise values indicate the backward computation time for each layer, whereas total values represent accumulated backward time for BP and the maximum layer time for DFA under ideal parallel execution. Results are measured on a six-layer MLP with 1,000 hidden units trained on CIFAR-100. The maximum DFA layer time is highlighted.
Type
layers
layers 2
layers 3
layers 4
layers 5
layers 6
DFA (layer)
0.0132
0.1003
0.0942
0.0910
0.0895
0.0811
BP (layer)
0.0098
0.0899
0.0927
0.0942
0.0957
0.0996
DFA (total)
0.1003
0.1003
0.1003
0.1003
0.1003
0.1003
BP (total)
0.4820
0.4722
0.3823
0.2895
0.1953
0.0996
17

---

Published as a conference paper at ICLR 2026
Regarding memory usage, DFA requires additional feedback matrices of size Ni x for each layer, where Nj is the number of neurons and e is the dimensionality of the error vector. As a result, DFA exhibits higher memory consumption compared reached a peak memory usage of approximately 130.99 MiB, while back-propagation used around 100.75 MiB. This difference reflects the cost of storing the feedback matrices. From com- putational perspective, DFA also differs from back pack-propagation in terms of the operations required for propagating error signals. Back-propagation computes Wi=1da++1 which requires (NN)+++)) operations, whereas DFA computes Fie, requiring O(N Since Nr is typically much smaller than M in classification tasks, DFA can require fewer operations and reduced data movement. Although exploiting this theoretical efficiency is not the main objective of our study, we include this analysis for completeness.
back- proppagtpa.ttio.. In our experiments, DFA.
I EXPERIMENTAL SETUPS
In this section, we offer an explanation of the experimental setup utilized throughout our research. Sec. 1.1 outlines the training details of the feedback matrix used for feedback matching in all models.
Sec. 1.2 covers the configuration settings required for the fully connected network experiments. Sec. 1.3 describes the setup necessary for experiments involving BERT, which employs a transformer architecture. [Scc. 1.4 provides the setup employed for experiments with the ViT model. For fine- tuning of BERT and ViT, feedback weight matching is applied to the attention, intermediate, and block outputs of the encoder layers in a similar way to previous works (Launay et al.] (0200 that attempt to apply DFA to Transformer's attention modules (Vaswanil 2017). To ensure the robustness of our findings, we report the average results over three different random seeds. All experiments were conducted on an NVIDIA GeForce RTX 3090 GPU with 24GB of memory.
1.1 FEEDBACK MATRIX
We train feedback matrices to reconstruct pre-trained weights that were trained using back- propagation (Rumelhart al. 1986). The loss function, in [... (23) used to guide the feedback matching process. The two learned feedbacks are then combined and re-initialized into single weight matrix for each layer. We use the Adam optimizer (Kingma 2014) without weight decay or any scheduler. In fully connected networks, learning rate of le-5 is applied, while in transform- ers (BERT) (Kenton & Toutanova 2019] Ture et al. 2019). learning rate of 1e-3 is used. For all experiments the model and dataset, training conducted for 3 epochs with batch size of 64.
1.2 FULLY CONNECTED NETWORKS
We pre-train two fully connected networks with four and six layers on the CIFAR- 100 (Krizhevsky al. 2009) and TineemageNet (Le & Yang 2015) datasets utilizing weights obtained through back propppropagtioo (BP). These pre-trained weights are subsequently fine-tuned on the CIFAR- 10 Krizhevsky et al. 2009). SVHN (Netzer et al. 2011], and STL-10 (Coates et al., 201 datasets. During the pre-processing phase, we apply image resizing and normalization, without any augmen- tations. For Direct Feedback Alignment (DFA) (Nokland 2016), the weights are initialized with a uniform distribution within the range of (-0.01, 0.01). Conversely, for back pack-propagation (Rumelhart) et al. 1986), we employ the He initialization (He et al. 2015). The optimization process is car- ried out using Stochastic Gradient Descent, and ReLU (Agarap 2018) is employed as the activation function. The hyperparameters for both the 4-layer and 6-layer architectures remain consistent. A comprehensive description of each hyperparameter under various training conditions is presented in
Tab.
I.3 BERT
We train BERT-Tiny and Small models (Kenton & Toutanova 2019 Ture et al., 2019 on the GLUE (Wang 2018) dataset using the Adam (Loshchilov) 2017) optimizer with fixed learn- ing rate and no scheduler. We apply weight decay and dropout techniques. GeLU (Hendrycks & Gimpel, 2016) is used for the activation function, which is commonly employed in BERT. Layers such as the encoder block outputs, intermediate outputs, and attention outputs are optimized using Direct Feedback Alignment (DFA) (Nøkland) 2016), while the projection layers for key, query, and
18

---

Published as a conference paper at ICLR 2026
Table 12: Hyperparameters for fully connected networks training.
Target Data
Hyperparmeters
BP inc
DFAumath
DFAin
DFAred
DFA
Learning Rate
le-3
le-3
le-3
le-3
le-3
Ie-3
le-3
Batch size
64
64
64
64
64
64
64
Hidden Dim
1000
1000
1000
1000
1000
1000
1000
Input size
3072
3072
3072
3072
3072
3072
3072
Epochs
5000
5000
5000
5000
5000
5000
5000
CIFAR-10
Weight Decay
Se-4
5c-4
5c-4
5c-4
Dropout
Epochs
5000
5000
5000
5000
5000
5000
5000
SVHN
Weight Decay
5e-4
Se-4
5e-4
5e-4
Se-4
Dropout
0.1
0.1
0
0
0
Epuchs
5000
5000
5000
5000
30000
30000
30000
STL-10
Weight Decay
5e-4
Se-4
0
0
le-3
le-3
le-3
Dropout
0.1
0.1
0.1
0.1
0.1
value are trained using back-propagation (BP) (Rumelhart al.) 1986). The weights initialized using a uniform distribution, and the feedback matrix is specifically designed to satisfy the left or- thogonality condition. A comprehensive description the hyperparameter values is presented in Tab. 13
Table 13: Hyperparameters for BERT training.
Model
Hyperpametees
Target Data
BPaazzhh
in:
DFAamac
DFAize
DFA
DFAecistt
DFAaan
Batch size
64
64
64
64
64
64
64
Dropout
0.1
0.1
0.1
0.1
0.1
0.1
0.1
Weight Decay
0.01
0.01
0.01
0.01
0.01
0.01
Epochs
6
length
512
512
512
2
Num layers
2
Hidden dim
128
128
128
128
128
128
Intermediate dim
512
512
512
512
512
512
512
CoLAA
le:5
le:5
Te:5
le:5
Te:5
1e:5
le:5
SST-2
1o-5
le-5
lc-5
le-5
le-5
le-5
le-5
BERT-Timy
MRPC
le-5
le-5
le-5
le-5
le-5
le-5
le-5
QQP
1e-5
le:5
1e-5
le:5
le-5
le.5
le:5
Leuming Rate
MNLI
le-5
16-5
le-5
1c-5
le-5
QNLI
5e.5
Se-5
STSB
lc:5
le:5
le-5
le:5
le-5
le:5
RTE
16-5
lc-5
lc-5
le-5
lc-5
lc-5
lc-5
WNLI
Se-5
Se:5
5e.5
Se:5
5e.5
5e-5
Se:5
Num of heads
8
8
8
8
8
Num of layers Hidden of dim Intermediate dim
4
4
4
4
4
512
512
512
512
512
512
512
2048
2048
2048
2048
2048
2048
2048
CoLAA
le-5
le-5
le-5
le.5
le-5
le-5
le.5
SST-2
1e-5
le:5
le-5
le:5
le-5
le-5
le:5
BERT-Small
MRPC
le-5
lc-5
le-5
le-5
QQP
le.5
Learning
MNLI
1c-5
le-5
le-5
le:5
le-5
lc.5
le:5
QNLI
56-5
5e-5
5e-5
Se-5
5e-5
5c-5
5e-5
STSB
le:5
1e:5
le:5
le:5
le:5
le:5
le:5
RTE
lc-5
le-5
lc-5
le:5
le-5
lc-5
le:5
WNLI
le-5
le-5
le-5
le-5
le-5
le-5
le-5
1.4 VIT
We fine-tune ViT-Tiny and Small models (Wu et al. 2022). both pre-trained ImageNet-1K (Deng) et al. 2009), on the CIFAR-10 (Krizhevsky et al. 2009). STL-10 (Coates et al. 2011). and Ima- geNette (Howard 2019) datasets. For preprocessing,
resize the 32x32 images from CIFAR-10 and STL-IO 22xx224 and apply normalization. We use the AdamW (Loshchilov) (017)) optimizer and GeLUU (Hendrycks & Gimpel 2016) as the activation function. Following the approach used in
19

---

Published as a conference paper at ICLR 2026
BERT, apply Direct Feedback Alignment (DFA) train the ViT models, specifically targeting the encoder block outputs, intermediate outputs, and attention outputs. While the Tiny and Small models have the same number of layers, they differ in the number of channels and attention heads. comprehensive list of hyperparameters for these models is provided
Tab. 14
Table 14: Hyperparameters for ViT training.
Model
Hyperparmeters
Target Data
DF
DFAm
DFA
Batch size
64
64
64
Dropout
0.1
0.1
0.1
Weight Decay Epochs
0.01
0.01
0.01
5
Image Size
224
Patch Size
16
16
16
Num of layers
12
12
12
CIFAR-ID 10
2e-5
2e-5
2e-5
Leamingg Rate
STL-10
20-5
20-5
20-5
ImageNette
2c-5
26-5
2c-5
Num of heads
3
ViT-Tiny
Hidden dim
192
192
192
Intermediate dim
768
768
768
Num heads
ViT-Small
Hidden
384
Intermediate dim
1536
1536
1536
20


Reviews:
[{"review_id": "sk8upT9inS", "reviewer": "Reviewer_4qY7", "summary": "This paper investigates Direct Feedback Alignment (DFA), an alternative to back-propagation, for the task of network fine-tuning. While DFA has shown promise for training from scratch, it is known to perform unreliably in fine-tuning scenarios. The authors provide a theoretical and empirical analysis arguing that this failure stems from a fundamental mismatch between the pre-trained weights (learned via BP) and the random feedback matrices used by standard DFA, which prevents the network from achieving 'Weight Alignment' (WA). To address this, the authors propose 'Feedback-Weight Matching,' a method that first reconstructs feedback matrices from the pre-trained weights and then re-initializes the weights to align with these new matrices before applying DFA. The proposed method is evaluated on fully connected networks and Transformer models (BERT-Tiny, ViT-Tiny/Small), showing significant improvements over standard DFA fine-tuning.", "questions": "Please see my comments above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem studied in this paper is interesting and potentially important for developing more efficient, BP-free learning paradigms.\n\n2. The proposed method is well-supported by a theoretical foundation.\n\n3. The experimental results are consistent and clear, which looks reasonable to me. \n\n4. The presentation is generally good. The paper is easy to read.", "weaknesses": "1. The authors claim that DFA is 'back-propagation free,' which implies a computational efficiency advantage. However, the paper lacks any empirical comparison of running time or computational overhead (e.g., FLOPs, wall-clock time) against standard back-propagation fine-tuning. This leaves the practical benefits of the method (beyond being a BP-free algorithm) unsubstantiated.\n\n2. The paper's scope may be somewhat narrow. The proposed method is only compared against existing DFA and standard back-propagation. Assessing its impact in the broader field of BP-free algorithms with comparisons to other methods can be a plus.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "beQZYNyNmx", "reviewer": "Reviewer_LvJQ", "summary": "In this work, the authors analyzed the limitations of the current Direct Feedback Alignment (DFA) methods for fine-tuning pretrained fully connected neural networks via weight alignment (WA) and gradient alignment (GA). They further demonstrated that the proposed feedback-weight matching method, when combined with weight decay, can not only effectively mitigate over-fitting but also further reduce the network output error.", "questions": "Q1:Since the proposed feedback-weight matching method is intended to replace backpropagation,have the authors ever considered doing experiments to directly compare between the computing time of the same pretrained models but fine-tuned based on the proposed method and back-propagation?\nQ2:Why the standard DFA method seems a little better than the proposed method when fine-tuning the 4-layer network from TinyImageNet in image classification tasks?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: It has mitigated the limitation that DFA could only train networks from scratch, by originally proposing the feedback-weight matching method to enhance the standard DFA,,thereby enabling fine-tuning using this method.\nSignificance:Currently, fine-tuning still plays a role as a critical training stage for maintaining strong performance in various large models. Introducing DFA ,a method enabling efficient parallel parameter updates—into the fine-tuning process to replace back-propagation holds significance.", "weaknesses": "Quality:while Proposition 4.2 and Conjecture 4.3 attempt to link FWM with weight decay for error reduction, the proof relies on strong assumptions (e.g., two-layer networks, specific activation properties). The conjecture for L-layer networks remains ​​unproven​​, and the theoretical analysis does not fully account for the interplay between feedback matching and weight decay in deep networks. This lack of rigor weakens the theoretical contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2uIWqGLZuB", "reviewer": "Reviewer_HVYL", "summary": "The paper proposes feedback-weight matching (FWM) to enable reliable fine-tuning with Direct Feedback Alignment (DFA). The method reconstructs feedback matrices from pre-trained weights and then re-initializes weights to match them, yielding stronger weight/gradient alignment and improved performance; the authors also analyze how weight decay synergizes with FWM. Empirically, FWM outperforms standard DFA across FC nets, BERT-Tiny/Small, and ViT-Tiny/Small (e.g., +7.97% accuracy on SVHN and +0.66 Pearson on STS-B).", "questions": "refer to weaknesses", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Simple, principled mechanism with theory: FWM is conceptually clean (feedback reconstruction + weight matching) and is supported by propositions showing it induces strong WA→GA; the paper further proves a GA advantage in a two-layer setting.\n\n2. Consistent empirical gains across modalities/models: Improvements appear on FC image classifiers, GLUE with BERT-Tiny/Small, and ViT variants, often turning DFA fine-tuning from unreliable to usable (e.g., STS-B 0.76 vs 0.10).\n\n3. Thorough ablations and interaction with weight decay: Ablations isolate feedback matching, weight matching, and weight decay, showing their contributions and a clear synergy between FWM and weight decay.", "weaknesses": "1. Scope and scale: Experiments focus on FC nets and compact Transformers/ViTs; CNNs are acknowledged as challenging for DFA, and there are no results on larger modern LMs, limiting claims of broad applicability.\n\n2. Efficiency evidence is missing: While DFA’s appeal is “backprop-free” and parallelizable, the paper reports no wall-clock time, memory, or throughput comparisons vs BP-based fine-tuning, so the practical benefit remains speculative.\n\n3. Theory primarily for simplified settings: Key guarantees (e.g., GA improvement) are proved for two-layer/linear cases, and extension to deep non-linear networks is left as a conjecture, weakening theoretical support for the full experimental regime.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VVUFRco9pX", "reviewer": "Reviewer_Bsfv", "summary": "This paper studies why Direct Feedback Alignment (DFA) struggles to fine-tune networks that were pre-trained with back-propagation (BP), and proposes a simple two-phase remedy called feedback-weight matching (FWM). The authors analyze DFA through Weight Alignment (WA) and Gradient Alignment (GA), arguing that randomly chosen feedback matrices prevent strong WA/GA when starting from BP weights. FWM first reconstructs feedback matrices from the BP weights (\"feedback matching\"), then re-initializes the weights to match those feedback matrices (\"weight matching\"), so that fine-tuning with standard DFA naturally induces strong WA/GA. They further prove that, together with weight decay, FWM reduces output error for two-layer settings and conjecture an extension to deeper nets. Experiments show consistent gains across FCNs, BERT (Tiny/Small), and ViT (Tiny/Small) models.", "questions": "Could the authors consider adding head-to-head comparisons against BP fine-tuning, LoRA/adapters, and partial-BP under matched budgets (same epochs/tokens, batch size, optimizer, augmentations). Report accuracy, calibration (ECE), robustness (seed/prompt sensitivity if applicable), and costs: wall-clock to target accuracy, tokens/sec or images/sec, peak memory/VRAM, and energy/FLOPs. Include at least one mid-scale vision task (e.g., ImageNet-1k fine-tuning) and NLP task (GLUE).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Clear problem framing & analysis.** The paper isolates a concrete failure mode of DFA fine-tuning from BP starts via WA/GA and proves why random feedback breaks alignment, then shows how FWM induces strong WA/GA. The link from theory (Defs./Props.) to the empirical WA/GA plots is tight and instructive for the learning-dynamics community.\n\n- **Simple, general mechanism.** FWM is architecturally light: decompose BP weights to build feedback, then re-initialize weights to match feedback; standard DFA then applies unchanged. This “match-then-tune” recipe is easy to implement and does not require modifying optimizers or loss functions, making it appealing for practitioners exploring BP-free training.\n\n- **Broad empirical scope.** Results span FCNs, BERT Tiny/Small, ViT Tiny/Small over CIFAR-10/SVHN/STL-10/ImageNette and GLUE tasks, consistently beating standard DFA fine-tuning and demonstrating stability improvements (WA/GA trends).", "weaknesses": "(1) **Limited transformer alignment coverage.** For transformers, alignment is only applied to dense layers; attention projections (Q/K/V) remain unaligned, which may cap attainable GA/WA. The paper mentions this but lacks systematic analysis of how much each component limits end-to-end gains, especially on larger ViTs or LLMs. Scaling experiments or per-block ablations would clarify.\n\n(2) **Baselines and practical metrics.** The main comparator is standard DFA fine-tuning (and DFA-scratch). Stronger practice-oriented baselines (e.g., BP fine-tuning, BP with adapters/LoRA, partial-BP variants) and wall-clock, memory, and parallelism measurements are missing; these are crucial to justify DFA-based fine-tuning in real systems beyond accuracy alone", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)