Scaling Direct Feedback Learning with Jacobian Alignment Guarantees
Authors: Paul Caillon
Erwan Fagnou
Blaise Delattre
Alexandre Allauzen
Venue: ICLR 2026 Poster (2026)
Keywords: backpropagation-free learning
optimization
Primary area: optimization

Abstract: Deep neural networks rely on backpropagation (BP) for optimization, but its strictly sequential backward pass hinders parallelism and scalability. Direct Feedback Alignment (DFA) has been proposed as a promising approach for parallel learning of deep neural networks, relying on fixed random projections to enable layer-wise parallel updates, but fails on deep convolutional networks, and performs poorly on modern transformer architectures. We introduce GrAPE (Gradient-Aligned Projected Error), a hybrid feedback-alignment method that (i) estimates rank-1 Jacobians via forward-mode JVPs and (ii) aligns each layer’s feedback matrix by minimizing a local cosine-alignment loss. To curb drift in very deep models, GrAPE performs infrequent BP anchor steps on a single mini-batch, preserving mostly parallel updates. We show that the forward-gradient estimator has strictly positive expected cosine with the true Jacobian. We relate this estimator-level guarantee to a standard stochastic-approximation result under a positive expected-cosine condition on the update direction, providing theoretical support for GrAPE’s alignment objective. Empirically, GrAPE consistently outperforms prior alternatives to BP, enabling the training of modern architectures, closing a large fraction of the gap to BP while retaining layer-parallel updates for the vast majority of steps.

Published as a conference paper at ICLR 2026
SCALING DIRECT FEEDBACK LEARNING WITH JACO- BIAN ALIGNMENT GUARANTEES
Paul Caillon', Erwan Fagnou', Blaise Delattre' & Alexandre Allauzen 1.2
1 Miles Team, LAMSADE, Université Paris Dauphine-PSI, Paris, France
2 ESPCI PSL., Paris, France
(name) (surname] @dauphine, psl.eu
ABSTRACT
Deep neural networks rely on backpropagation (BP) for optimization, but its strictly sequential backward pass hinders parallelism and scalability. Direct Feedback Alignment (DFA) has been proposed as a promising approach for parallel learning of deep neural networks, relying on fixed random projections to enable layer-wist parallel updates, but fails on deep convolutional networks, and performs poorly modern transformer architectures. We introduce GrAPE (Gradient-Aligned Projected Error), a hybrid feedback-aligmment method that (i) estimates rank-1 I Jacobians via forward mode JVPs and (ii) aligns each layer's 's feedback matrix by minimizing a local cosinc-alignmed performs infrequent BP anchor steps on single mini-batch, preserving mostly parallel updates. We show that the forward- -aaadgga estimator has strictly positive expected cosine with the true Jacobian. We relate this estimator-level guarantee a standard peehhaticcapproximati result under positive expected-cosine condition on the update direction, providing theoretical support for GrAPE's alignment objective. Empirically, GrAPE consistently outperforms prior alternatives to BP, enabling the training of modern architectures, closing a large fraction of the gap to BP while retaining layer-paralle updates for the vast majority of steps.
loss. To curb drift in very deep models, GrAPE
INTRODUCTION
Backpropagation (BP) (Rumelhart et al. 1986) remains the de facto standard for training deep networks. However, its memory footprint and energy cost have become critical bottlenecks as architectures deepen and scale up. In particular, two properties of the BP impede the development of parallel training methods: the weight symmetry between the forward and backward pass, and the sequential propagation of the error. These two properties also clash with biological plausibility. In this paper, we deliberately set aside the biological concerns to focus on non-sequential alternatives to BP. A rich literature has mainly explored two independent axes of relaxation:
Randomized feedback (Feedback Alignment, FA (Lillicrap et al.) 2016). Direct FA, DFA (Nokland 2016), etc.) replaces transposed weights by fixed adaptive random matrices, but suffers from misalignment on deep or convolutional layers (Bartunov et al.) 2018; 2018: Moskovitz et al., 2018 Launay 2019). Adaptive variants using weight mirroring (Akrout et al.) (2019) can approach BP performance, remaining sequential, however, offering limited practical advantages.
Forward-gradient and forward-only methods (Silver al.) 2021 Baydin et al.) [022 Hinton 2022
Dellaferrera & Kreiman 2022) replace the backward pass by Jacobian vector products or a second "perturbed" forward pass, at the cost of high variance and limited scaling to modern architectures.
Our work extends the first axis, starting from the following observation: fixed random feedback matrices often lose positive cosine similarity with true gradients in deep and structured layers (Nok) land 2016 Refinetti et al. 2021). As consequence, this kind of feedback fails to decrease the loss function. We therefore introduce a lightweight and data-driven correction using forward gradient estimates. With this alignment, combine the efficiency of randomized feedback with estimator- level Jacobian alignment guarantees from forward- -raddiet estimates, augmented by an occasional BP calibration step to reduce variance in very deep networks. Our core contributions

---

Published as a conference paper at ICLR 2026
1. Gradient-guie guidedd-ggddddddu feedback. feedback We introduce GrAPE (Gradient Aligned Projected Error), which computes a local cosine-alignment loss with forward -raddiett estimates. This realigns each layer's 's feedback matrix toward Jacobian-aligna directions prior to the parallel DFA update.
2. Leveraging forward-mode gradients, we derive a positive expected alignment bound for our rank-1 Jacobian estimator. We also recall standard conditional convergence- ierreencci--eeppctaation result under a positive expected-cosine cosine assumption on the update direction, which provides theoretical motivation for GrAPE's alignment objective.
3. Occasional BP calibration. To further mitigate drift in very deep highly structured networks, we apply a true BP step to a single mini-batch every T epochs, using its exact gradient to realign the weights. This yields hybrid two-timescale scheme in which most updates are layer-parallel GrAPE steps, interleaved with sparse BP synchronizations.
4. Scalability. We show for the first time that DFA-style method can train VGG-16, ResNet- 20/56 and Transformer models, narrowing the performance gap with full BP.
The paper is organized follows: in Section 2 we briefly recall the necessary background and notation (a more detailed survey can be found in the Appendix). Section 3 describes the GrAPE algorithm and the occasional BP calibration strategy. Section reports empirical results.
2 BACKGROUND AND RELATED WORKS
Let [x;;; 0) be a feed-forward neural network with L layers, where x = ho is the input and 0 (Wi)== is the set of parameters. Each layer computes a; = -ih followed by a non-linearity h4 =a)(), encompassing both linear and convolutional operations. The output is ğ h2. Given a loss function C(), y). the goal of backpropagation (BP) is to compute gradients VL = 0C/0a; recursively, starting from the output layer. The corresponding weight update is:
if = L 7 80t 1 ifl < I., with 8oz
VCLh
sW;
(1)
o d((())
This algorithm is by construction sequential: the update at layer i depends the backpropagation of errors through all subsequent layers. This reliance on weight symmetry and stepwise computation hinders parallelism. As architectures attain increasing size and depth, alternative methods that allow non-symmetric error transmission and enable parallelized training have emerged (see Figure
2.1 LEARNING WITH RANDOM FEEDBACK
Feedback Alignment (FA) proposes biologically inspired alternative to backpropagation by replacing transposed weights with fixed random feedback matrices B₁ (Lillicrap et al. 2016). The error still propagated sequentially, but independently the forward weights (W)):
daz= (Brd0141) O of(()), with dag VLz) o' (az)
This removes the weight symmetry constraint, aligning better with biological learning (Lillicrap) et al. 2020) but fails to scale to convolutional networks (Bartunov et al. 2018 Moskovitz et al. 2018). Adaptive variants using weight mirroring (Akrout et al.) 2019) can however approach BP performance, but remain sequential and thus offer limited practical advantages.
Direct Feedback Alignment (DFA) Nokland 2016) removes the need for sequential error propaga- tion by projecting the output error directly to each hidden layer:
8a; (B.VL)) @ o((a)) VI L]
This enables parallel updates but remains limited complex architectures like CNNs and Transform- ers. Attempts to mitigate this include adaptive feedback (c.g., weight mirroring (Akrout et al. 2019) or architectural variants like DRTP (Frenkel et al. 2021). falling short behind BP on large-scale tasks.
Launay et al. (2020) applied DFA to Transformers using either block-wise ('macro') or layer- wise ('micro') feedback, yet BP remains necessary within attention layers. Our method builds on this approach by providing more informative feedback signals, complementing the internal backpropagation still required within attention blocks.
2

---

Published as a conference paper at ICLR 2026
6.Fonaziori
Ouipett dwl
O O
O O
O
O
0000 O
W
W;
0
0000
0000
COOO
W;
W;
0
0000
0000
COOO
0000
W1
Wi
W
Wi
kppeTopp
0
0000
0
0
0000
Prees Feot
Figure 1: Overview of error propagation schemes, adapted from Dellaferrera & Kreiman 2022 (a)
BP. (b) FA, (c) DFA, (d) forward-only methods (PEPITA, Forward-Forward (e) GrAPE. Green arrows indicate forward paths; orange ones error signals and blue ones forward gradient estimates J.. Learned weights are denoted Wi, and layer-specific feedback weights as B₁.
The effectiveness of FA and DFA relies on the alignment between feedback vectors and true gradients as shown by Nøkland (016); Refinetti et al. (202T). A sufficient condition for descent is thus:
vl E [1, L), (os(wa) = V2, B; VEII-BBB B 0. This reflects the classical Zoutendijk condition (Noceaal
& Wright 1999), highlighting the importance of alignment for descent and convergence in classical deterministic line-search methods (see Appendix A.3
2.2 FORWARD-ONLD CALCULATIONS
Forward-only methods aim to completely bypass the backward pass. For instance in PEPITA
(Dellaferrera & Kreiman 2022) or Forward-Forward (Hinton 2022). double forward pass provides
a surrogate error signal via a perturbed forward step: 8Wi (4--h5") h("") (M "
h x Fe
Forward Gradient (FG) methods (Silver et al. 2021 Baydin et al. 2022) use forward-mode automatic differentiation AFw to obtain unbiased gradient estimates via directional (Jaco- bian-vector) derivatives along a random direction u, removing the need for an explicit backward While unbiased in theory, sampling in parameter space is inefficient for large models. Ren et al. (2022) address this by perturbing neuron activations instead of weights, significantly reducing variance and cost activation space is usually much smaller than weight space. Additional improvements use local auxiliary losses (Fournier et al. 2023), requiring however BP to train the auxiliary models.
pass: VC lims--0 2(0+5u)-CC
FwAD methods parallel the standard forward pass and can be implemented via dual numbers. Despite some runtime overhead (43% w.r.t. a simple forward with a naive implementation (Baydin al. 2022 ), optimized frameworks (e.g., Porch FWAD, JAX) promise broader applicability.
Other recent local learning rules also exploit forward computations without relying on FA/DFA-style feedback pathways. Nøkland & Eidnes (2019) attach shallow local classifiers and similarity-base objectives to each hidden layer, computing layerwise gradients from these auxiliary losses strictly feedforward manner (with an optional FA- variant), while Apolinario et al. (2025) propose a rule that drives each layer to align its activations with fixed periodic basis vectors using local cross-entropy loss. In both cases, learning signals are generated from local forward computations and do not rely on separate feedback matrices B₁ or on backpropagated errors from the global task loss.
While these methods can reach reasonable performance on relatively shallow compact architectures, their reported accuracies typically remain below backpropagation on deeper models, and they have not yet been demonstrated at scale modern Transformer-style networks. GrAPE is complementary: it also leverages forward-ma modee information, but uses JVP-based rank-1 1 Jacobian estimates to explicitly learn feedback matrices that approximate true gradient directions for DFA-style updates.
3

---

Published as a conference paper at ICLR 2026
3 GRADIENT ALIGNED PROJECTED ERROR (GRAPE)
Here, we introduce GrAPE (Gradient-Aligned Projected Error). This method leverages the lightweight serward-gradiin estimates to align the feedback matrices with the true Jacobians. This alignment is motivated by the Zoutendijk theorem, enabling scalable training of complex architectures on various tasks. We summarize the overall procedure in Algorithm T
3.1 LIMITATIONS OF FIXED RANDOM FEEDBACK MATRICES
In line-search methods, the update direction must align as closely as possible with the true negative gradient. DFA uses fixed "feedback" direction in place of VL, but cannot measure or correct the angle between its update and the actual descent direction. In convolutional layers, for example, the linear transformation can be represented by block-Toeplitz matrix (d d'AAsooli et al. 2019). Reproducing such a structure with a single, fixed, randomly sampled feedback matrix is impossible, as discussed by Refinetti et al. (202T). This explains why Launay et al. (2019) found that vanilla DFA often fails on convolutional networks: the convolutional weights cannot correctly capture the projected error if the feedback direction is misaligned with the true gradient. However, if we align the feedback projections with the associated gradients, this limitation can be overcome.
Beyond fixed random feedback, early weight-mirroring schemes (Akrout et al. 2019) already showed that it is possible in principle to adapt feedback weights to approximate W, in FA, albeit in fundamentally sequential setting tied to the forward weights.
Subsequent DFA-style approaches (Webster et al. 2021 Bacho & Chu 2024) extend this idea, either by using a Kolen-Pollack-type learning rule to adapt feedback weights or by tracking BP updates through auxiliary forward passes and momentum. Roy et al. (2025) "unlock" SVD-space by optimizing a composite set of 5 local losses, including a cosine-like term, to align feedback with forward singular vectors. While these methods can substantially improve feedback alignment and sometimes recover BP-level accuracy, they either inherit the sequential nature FA, rely on complex SVD-based multi-term loss machinery, or, in the of FDFA, raise reproducibility questions as discussed in Appendix A.4
None of them exploit mode JVPs obtain a simple, analytically tractable Jacobian alignment guarantee for rank-1 estimators. GrAPE takes a different route: it uses JVP-hased rank-1 Jacobian estimates and single cosine loss to learn feedback matrices that are both amenable layer-parallel DFA-style execution and analytically linked to the true Jacobians through a positive expected Frobenius cosine bound at the estimator level. Together with infrequent BP steps on a single mini- batch, GrAPE demonstrates in our experiments scaling to deeper CNNs and Transformer architectures where vanilla DFA and related methods have historically struggled.
3.2 ALIGNMENT WITH FORWARD GRADIENTS: STATISTICAL GUARANTEES
Alignment lower bound Consider the Jacobian matrix Ti = ag E RRaa XM1, the perturbation ahe
p (0, In,) and the Jacobian-vector product Ji P E Rd An unbiased rank-1 approximation of Ti is then Jt = (Jip) p E RRdo Xxi, We measure alignment via the Frobenius cosine
(F.
cosF (J. [[ 1
(A, : TrAA :
Because ~ (O, I, we can write p = rs with s uniform on the unit sphere and indepen- dent of r ||p||. direct computation shows cosp FT. =
l. so E (Jt = ||
Jil F Eg || Jis| Projecting onto the top singular direction of Ji and using a standard bound on the first coordinate of a uniform sphere point (Appendix B yields
E E[coss 3i)
2
|Jll
J T4 JA
which is strictly positive for J. * 0. For the batched estimator (average of B independent rank- estimates), Gaussian concentration implies that the empirical Frobenius cosine concentrates around equation 4 at rate O(1/ /VB (Appendix
4

---

Published as a conference paper at ICLR 2026
Beyond this layerwise alignment bound, Appendix [B.4 recalls standard result (Theorem B.I) showing that, under usual step-size conditions, a positive expected cosine between the update direction and the true gradient is sufficient for convergence to stationarity in expectation. This provides a theoretical support for learning feedhack matrices that improve layerwise alignment. GrAPE is designed so that its JVP-based estimator and alignment loss encourage improved layerwise alignment, consistent with such a positive expected-cos condition.
Zoutendijk theorem In classical smooth deterministic optimization, the Zoutendijk theorem states that if each search direction forms an angle uniformly bounded away from T/2 with the negative gradient and the step sizes satisfy Goldstein or strong Wolfe conditions (typically enforced by line- search; see Nocedal & Wright 1999). then the gradient norm converges to zero and the iterates approach a stationary point. our context, we view Zoutendijk primarily as a conceptual lens moti- vating cosine-based alignment: our JVP-based estimator yields a strictly positive expected Frobenius cosine between Ti and Ji (Eq. 4 and standard stochastic- watpaaticc-ppoximmatiiiioo results (Theorem B. in Appendix [B.4) show that a positive expected cosine between the update direction and the true gradient sufficient for convergence to stationarity in expectation under usual step-size conditions. This motivates the alignment objective used to train the feedback matrices.
Concretely, we update each feedback matrix with respect to the corresponding estimated Ft, before applying the DFA weight update (Equation [}. Here, the per-layer cosine with the estimated Jacobian serves as tractable local alignment objective that heuristically supports global descent when improved across layers:
AI E [1, L), cos(wr) == coss(Bii,))) = (Bt, Ji
> 0,
(5)
| 1|
In Appendix B.5
we further show via simple Frobenius cosine composition lemma (Lemma B.2 that
quantitative lower bounds on (Br and c Tr) induce a corresponding lower bound on cos (O,, Jt). This controls how estimator noise and imperfect feedback learning combine, although positivity of the two intermediate cosines alone does not automatically imply positive composed bound. practice, we use the empirical average of per-column cosines č as computationally convenient proxy for the Frobenius cosine cosp(Br Br, Ti): as detailed in Appendix C c is a weighted average of these columnwise cosines, and after column normalization of Bj and our JVP construction, the weights are close to uniform, so è is a convenient scalar summary of layerwise alignment.
In practice, for each layer we can choose to do a perturbation either in the weight space (as explained above) in the activation space (Ren et al. 2022). Since the variance and cosine of the estimator depend directly on the dimension of the perturbations, we pick the space with the lowest dimension. Although usually the weight space is of much higher dimension than the activity space, this is not the case for the first layers of a deep convolutional network, for instance.
3.3 LEARNING RULE AND ALGORIITH
We first define local alignment loss: Catign
(wr), where cos(wi) is defined in
Equation 5 We update B₁ by one gradient step on Lalign:
B
Bj 78, V B, Lalign (Bi). 1B, > O.
(6)
and then normalize columns to enforce purely directional alignment, B/[;,A] +- Bif: A Xi]l +)) with e 0 for numerical stability (cf. Nøkland 2016).
This local step uses only forward-mode JVPs and no BP is required. batched JVP per layer adds roughly a forward-pass pass-liikskeee cost and does not scale with the number of parameters. Furthermore, our rank- 1 Jacobian estimate has a strictly positive expected Frobenius cosine with the true Jacobian and concentrates the batch size grows (Appendix [B. This guarantee concerns estimator-Jacobias alignment and provides theoretical support for the full GrAPE update. We found one alignment step per batch sufficient in practice; additional steps offered no gain.
worth noting that a single
Finally, with this refined B₁ we perform the parallel update:
daz
(Br VLr.) da))),
5W: -n day ITT1-
5

---

Published as a conference paper at ICLR 2026
3.4 SPARSE BP CALIBRATION ON A SINGLE MINI BATCH
In order to counteract the increased variance of our forward-gradient estimates in high dimensions, additionally inject all W, with minimal interruption of the parallel update flow. Let T denote the number of epochs between two such BP calibration steps. Currently, we select the calibration mini batch uniformly at random; a promising extension would be to apply active-learning strategies to pick the most informative examples for each BP calibration step, for example with uncertainty sampling or core-set selection (Settles 2009 [ener& & Savarese 2017).
BP step single mini-batch, using its exact gradients re-anchor
The amortized per-epoch overhead of a single-mini-batch calibration is s 1/(TN,, of an epoch in units of mini-batches (O(N, + 1/T) vs. (NN) steps). For typical N₁ >>>>> 1, this is small in D provides a compact FLOPs and critical-path accounting
practice. Appendix
BP, DFA, and
GrAPE covering forward+JVP overhead, per-layer projection/alignment, and the 1 calibration term together with preliminary timing results on a small transformer. A full study of optimized parallel kernels is left future work.
Algorithm 1 GrAPE
Require:
Layers 1, I,...,,, weights {Wi). feedbacks [Bi), BP-interval T, epochs E, batch size B, learning rates 1]B Forward pass & JVPs: choose perturbation pl in the smaller of activation/weight space; run single forward trace carrying duals
2: for epoch 1 E do
3: for all minibatch (X, Y) of size B do
4:
1) Forward pass & JVPs: choose perturbation pr in smaller of activation/weight space ho <- X
5:
6:
for / to do
7:
a Wi Wih-1 h o(())) oi c JVP(S, h4, pi)
via forward-mode AD (rank-1 estimate)
8:
Jj 9
9:
end for 2) Feedback refinement:
11:
for / = to L do
12:
compute per-column cosines: ē = 1 En' cos(Bil: k], Jrle Ae)
Latign <- 1 G
14:
Bi 4 B 7B VB, Lalign
15:
for k = 1 to n; do
16:
B: k] + BiG. Bi
17:
end for end for 3) DFA-style weight update: VC, + OC/Oar. for all / [1, do
18:
19:
20:
21:
Parallel projection (via equation 2 8a; (B/VL)) (((a)
Weight update: 8W; t - 7) dag bi_1
Wi Wi+ 88W
24:
end for end for
26:
4) Occasional BP calibration:
27:
epoch mod = 0 then Sample one small minibatch Perform one full BP pass on Update {Wi by standard gradient descent
28:
29:
30:
31:
end
32:
end for
4 EMPIRICAL EVALUATION
We evaluate GrAPE in a variety of settings, with shallow and deep image classification models, as well transformers for language modeling. We empirically show that GrAPE consistently outperforms

---

Published as a conference paper at ICLR 2026
Table 1: Performances of shallow convolutional network (CNN) and a layer Multi Layer Per- ceptron (MLP) trained on the MNIST and CIFARIO datasets with different learning algorithms (in percentages).
Method Parallelizable
MNIST
CIFARIO
CIFARIO0
MLP
CNN
MLP
CNN
MLP
CNN
BP
No
98.73
0.04
99.03 + 0.02
54.09 + 0.14
74.66
0.08
28.18 + 0.45
44.22 + 0.19
FA
No
98.36 +: 0.04
98.7 + 0.07
52.18 0.15
71.05
0.18
24.54 + 0.22
35 0.27
DRIP
Yes
95.7 0.12
17.55
64.73 2: 0.62
13.63 + 0.43
30.54
DEA
Yes
98.21 0.07
98.6 +.6++00 0.04
51.32 +: 0.32
69.34 +: 0.4
22.44 +: 0.23
34.53 + 0.42
PEPITA
Yes
98.01 + 0.08
NA
52.01 + 0.13
NA
21.87 + 0.25
NA
GrAPE (ours)
Yes
98.53 +: 0.02
98.8 0.01
53.4
73.1 0.23
26 +: 0.33
38.0 0.31
DFA and other methods. even without BP calibration. Introducing the occasional calibration step improves both DFA and GrAPE, and brings GrAPE much closer to BP.
erucial requirement for any BP alternative rigorous implementation and evaluation under the same settings as BP. Since these methods are still nascent, fair and reproducible comparisons are essential. Although FDFA (Bacho & Chul 2024) also combines random feedback with forward-mode AD, does not, to our knowledge, provide an optimization-theoretie convergence argument. Our attempts to reproduce their reported numbers across CIFAR- 100 settings did not match the paper's ' figures using the authors' code BioTorch reimplementation; details are provided in the Appendix A.4
4.1 EXPERIMENTAL SETTING
We implement our method in Biotorch (Sanfiz, & Akrout 202T) to ensure full transparency and to leverage its existing beedback-alignnent rates, schedulers, etc.) are detailed in the Appendix E and code will be released upon publication. In current Biotorch implementation, computations are largely serialized on single GPU and we do not yet exploit true layer-pan serer-paalllel scheduling. We chose Biotorch to ensure a fair comparison with existing baselines and a standardized way of benchmarking In this regime, GrAPE incurs a modest 6 6-20% wall-clock overhead per training step compared to DFA or BP, depending on architecture and dataset, due to the extra forward-mode JVPs and local alignment updates.
and forward-mode hooks. All hyper-parameters (learning
method within the same framework.
To complement backward hooks in convolutions, we use PyTorch's 's low-level conv2d conv2d_input, conv2d we iih,, conv2d_ba bias from torch. nn grad and the JVP routines from FwAD, en- suring correct gradient estimates in convolutional layers without any backward pass. All experiments were conducted on a NVIDIA A100 GPU. We test our method on the following setups:
Shallow architectures. We first validate on 3-layer MLP (hidden size 1024) and LeNet-5-style CNN across MNIST, CIFAR- 10, and CIFAR-100. Baselines follow the strongest BP-free comparators in Srinivasan et al. ):23): FA, DFA, DRTP (Frenkel et al., 202T) and PEPITA (Dellaferrera & Kreiman 2022) with the recommended variance-reduction tweaks from [Siinvasan et al. (2023). Consistent with prior reports, DRTP/PEPITA do not scale to deep CNNs, hence we exclude them from
tables. The results are reported Table
Deeper convnets with BP calibration. Next, we tackle AlexNet and VGG-16 on CIFAR-100 architectures which vanilla DFA catastrophically fails (Launay et al. 2019). Here we inject one backpropagation update on a randomly selected mini-batch every 7 epochs (for both GrAPE and DFA as "calibrated" control). Surprisingly, this sparse BP step alone recovers large fraction of DFA's gap with BP and allows GrAPE to attain really close performance when compared to BP. The results are reported in Table 2 and Figure 2
Cost of BP calibration. In all calibrated settings, BP step consists of a single full backward pass on one mini- mini-batch every T epochs. On CIFAR-100 with ResNet-20 and batch size 256, this corresponds to one calibration batch out of R 195 per epoch; for T = 1 this is about 0.5% of the backward passes used by standard BP, and for larger T' the overhead is reduced proportionally. Thus, even at = 1 the calibration cost is negligible compared to the bulk of GrAPE or DFA updates.
Modern architectures with BP calibration. Finally, we scale to a Transformer Base on WikiText- 103, following exactly the protocols of Launay et al. (2020). We adopt macro (one feedback per encoder block) and micro (one per sub-layer) feedback approaches, replacing each fixed feedback by learned one via the same local cosine-alignment, with change to forward attention internals;
7

---

Published as a conference paper at ICLR 2026
Table 2: Performances of AlexNet and VGG-16 models trained CIFAR-100 with different learning
algorithms.
Method
AlexNet
VGG-16
BP
64.61 + 0.29
70.33 + 0.61
DFA
42.59 + 0.34
1.00 + 0.00
DEA calibration (T
49.37 + 0.16
29.40 + 0.82
GrAPE
45.45 + 0.20
32.40 + 0.32
GrAPE + calibration (T 1)
62.63 + 0.52
56.93 0.11
BP inside attention layers remains as in Launay et al. (2020), depending on the specific setting. We also scale to deep networks such ResNet-20056 on CIFAR-IO0 100 per canonical practice in He et al. (2016) and also on Tiny ImageNet, using as compute-cfficient proxy that preserves ImageNet-like statistics, mirroring network, we apply a BP calibration step on a single mini batch every epochs to both GrAPE and DFA. We report averages over 10 independent runs of each method's best checkpoint, with standard deviations to reflect stability. The results are reported in Tables 3 |andd
trends (Shleifer & Prokop 2019). Once again these deep
Potential gains under layer parallelism. To probe the potential gains under actual layer parallelism, we also implemented a small prototype on a Transformer with hidden size 128, depths 2/4/8, batch size 256 and sequence length 64 on single NVIDIA A100. Using Python-level CUDA streams and simple double-forward trick to compute JVPs (duplicating the batch and perturbing the duplicate to compute JVPs), the mean time per batch was roughly three times lower for GrAPE versus BP (Table Appendix DD. These numbers are conservative (no kernel fusion or custom kernels), but they illustrate that once layer-parallelism exploited, GrAPE can reduce wall-clock time relative to sequential BP, especially at larger depths.
4.2 RESULTS ANALYSIS
Shallow architectures (Table I) In our preliminary small-scale experiments on 3-layer MLP (hidden size 1024) and LeNet-5-style CNN across MNIST, CIFAR-10 and CIFAR-100 (Table T). vanilla GrAPE surpasses every other method - FA, DFA, DRTP and PEPITA without any BP calibration. This shows that low-dimensional estimates are sufficiently accurate to drive learning effectively without full backpropagation updates. AlexNet and VGG-16 (Table Z Figure BP achieves the highest accuracies. with 64.6% + 0.3 on AlexNet and 70.3% + 0.6 on VGG-16. Uncalibrated DFA performs poorly (only 42.6% + 0.3 on AlexNet and 1.0% on VGG-16). Introducing one BP calibration per epoch AlexNet and nearly 30 points on VGG-16. GrAPE without calibration starts higher (45.5% + 0.2 on AlexNet, 32.4% + 0.3 on VGG-16), but with = 1 almost matches BP, reaching 62.6% + 0.5 and 56.9% + 0.1, closely trailing the BP curve in Figure 2
shallow settings, GrAPE's forward-gradie
1) boosts DFA by
over 6 points
100
100
calibration
80
80
DEA
GrAPE
Accuracy
Accuracy
60
40
40
20
20
5
10
50
Never
5
10
50
Never
T
Figure 2: Accuracy vs. calibration interval T for AlexNet (left) and VGG-16 (right) on CIFAR- T number of epochs between two BP calibration steps (i.e., one calibration every T epochs). We compare BP, DFA, DFA + calibration (T), GrAPE, and GrAPE + calibration (T).
8

---

Published as a conference paper at ICLR 2026
Table 3: Performance (%) ResNet-20 and ResNet-56, for CIFAR-10 and Tiny ImageNet. Here, T' specifies the number of epochs between BP updates on randomly selected mini batch. Method
ResNet-20
ResNet-56
CIFAR-100
Tiny ImageNet
CIFAR-100
Tiny ImageNet
BP DFA
68.72 + 0.14
51.66 + 0.74
71.42 + 0.60
56.86 + 0.83
20.94 + 0.19
14.18 0.11
24.29 + 0.41
15.31 0.05
GrAPE (ours)
24.28 + 0.36
18.63 + 0.37
29.33 + 0.63
20.15 + 0.12
59.80 + 0.55
46.13 + 0.29
62.43 = 0.15
48.39 + 0.48
55.28 + 0.22
43.56 0.57
61.40 + 0.81
47.00 + 0.34
DFA + calibration
10
53.79 + 0.29
44.21 + 0.22
60.29 + 0.40
46.92 0.26
50
30.06 + 0.93
20.78 1778:0.0 0.71
53.91 0.54
22.83 + 0.78
64.82 + 0.55
48.96 + 0.21
66.92 + 0.26
51.68 + 0.48
63.09 + 0.53
0.11
65.92 0.62
49.02 - 0.73
GrAPE calibration
10
61.15 0.21
44.44 0.28
65.75 = 0.59
47.72 - 0.21
50
36.79 + 0.62
22.15 + 0.58
56.47 + 0.54
24.40 0.47
Table Best validation perplexity after epochs of a Transformer trained on WikiText-103 (lower is better). T specifies the number of epochs between BP updates randomly selected mini batch.
BP
DFA
GrAPE
DFA
calibration
GrAPE calibration
T
10
5
10
Macro
52.0
42.3
42.7
48.2
50.1
33.1
37.8
40.4
29.8
Micro
93.3
81.1
78.8
87.9
90.5
67.3
73.7
78.2
ResNet-20 and ResNet-56 (Table On CIFAR-100 with ResNet-20, uncalibrated DFA achieves only 20.9% + 0.2 and GrAPE 24.3% + 0.4, compared to BP's 68. 7% = 0.1. A single BP calibration every epoch (T = 1) elevates DFA to 59.8% +.6 and GrAPE 64.8% closing most of the gap with BP. As I' increases to 5, 10, and 50 epochs, both methods gradually lose accuracy, highlighting the need for frequent calibration. Similar trends hold for ResNet-56 and on Tiny ImageNet: GrAPE with T consistently outperforms calibrated DFA and approaches BP performance.
Transformer-Base on WikiText-103 (Table 4 For the language modeling task, uncalibrated DFA yields perplexities of 52.0 (Macro) and 93.3 (Micro), while GrAPE starts 42.3 and 81.1. With T= 1, DFA improves to 42.7/78.8, but GrAPE reaches 33.1/67.3, cutting its gap with BP (29.8) by nearly half. Calibration intervals of 5 and 10 epochs result in progressively worse perplexities, confirming that frequent BP injections are key to maintaining model quality.
Critical role of periodic BP calibration All experiments indicate that regularly performing one BP can greatly improve the training of both DFA and GrAPE. It seems particularly critical for larger and deeper models (e.g. VGG-16, Transformers and ResNets). One potential reason is that the variance of the forward gradient grows linearly with the hidden dimension: larger models lead to more noisy gradient estimates, which makes training more unstable. However, ablation studies with no calibration steps show that the GrAPE learning rule with adaptive feedback matrices is fundamentally better than DFA. some cases (eg. Transformer, VGG-16), GrAPE with no calibration even outperforms DFA with calibration. With matching calibration frequencies, GtAPE also consistently outperforms DFA.
Across all architectures and tasks, periodic BP calibration on a single random batch effectively bridges most of the performance gap between approximate methods (DFA, GrAPE) and full backpropagation. In particular, GrAPE with T matches BP performance very closely while relying on update rules with shorter arithmetic critical path than sequential BP; in current serialized implementation this manifests as a modest 6-20% per-step overhead, but simple layer-parallel prototypes already show potential wall-clock speedups at larger depths (Table 5)

---

Published as a conference paper at ICLR 2026
5 DISCUSSION AND LIMITATIONS
Our empirical results demonstrate that GrAPE achieves state-of-the-art performance among feedback- alignment variants in shallow models (Table D]. and recovers the majority of BP's accuracy on deep convnets with at most a single BP calibration per epoch (Tables 2 3 Furthermore, this BP calibration step yields substantial perplexity gains on Transformers (Table Nonetheless, several important caveats remain. First of all, because our BioTorcch-aase
prototype serializes updates, we do not report realized layer-parallel wall-clock gains: these are deferred to future purpose-built kernels. Necessity and cost of BP calibration While vanilla GEAPE suffices for low-dimensional / shallow settings, deep or wide networks depend critically on periodic BP updates: calibration frequency
T = 1 consistently outperforms T 1, but even single-batch BP step might incur nontrivial overhead compared to fully local methods. Designing adaptive or event-driven calibration schedules triggered by alignment metries rather than fixed epochs could reduce this cost, well as an advanced mechanism to select the samples contained in the considered batch.
This calibration can be interpreted in a federated- learning analogy: each layer's GrAPE update plays the role of independent "local training" on edge devices, using only cheap, parallelizable forward- gradient corrections; the occasional full BP calibration then acts like a central-server aggregation step, collecting the current model state, computing an exact gradient "global update," and redistributing the realigned weights back to all devices. In this way, we retain fully parallel local updates most of the time, yet periodically synchronize via trusted central pass to ensure that all feedback matrices stay coherently aligned across the entire network.
Variance in forward-mode estimates Our Jacobian estimate is unbiased but exhibits variance proportional to the model dimension (Section ( In wide layers, the alignment signal may collapse and noise dominates, necessitating frequent calibration steps and reducing the impact of vanilla GrAPE. Future work could explore multiple perturbation directions at each forward or lightweight local losses (as in (Fournier et al. (023)) to reduce variance, at the expense of additional computation. Convergence guarantees Our theoretical analysis provides estimator-level Jacohian alignment guarantees and recalls a standard conditional convergence result under a positive expected-cosine assumption on the update direction. In deep nonconvex landscapes, this leaves open the risk of saddle-point or poor-quality minima. Incorporating second-order information (e.g. diagonal Hessian approximations via forward-mode AD) may strengthen convergence toward high-quality minima. Architectural generality We validated GrAPE on multiple types of networks without designing adapted shapes for the feedback matrices. Imposing structures on the feedback matrices to respect the inherent composition of the Jacobians (for example block-diagonal for convolutions) could be a promising avenue to reduce reliance on BP calibration and further close the gap with full BP. In summary, GrAPE moves us closer to truly parallel, local learning methods, but fully matching BP on large-scale, modern architectures will require advances in variance reduction, adaptive calibration, and architecture-awan aware feedback design.
6 CONCLUSION
We have presented GrAPE, a novel aligmmet- algorithm that replaces the conventional backward pass with parallelizable feedback projections, aligned with the gradient direction. By computing cheap rank-1 Jacobian approximations during the forward pass and injecting occasional backpropagation updates, GrAPE combines the parallelism of forward-only methods with an accuracy close to that of standard backpropagation. Our empirical evaluation confirms that in shallow settings, GrAPE already outperforms all existing Seedback-alignmenn variants without any BP calibration. More importantly, on deep convolutional and residual networks and on a Transformer Base model, a backpropagation step per epoch on single batch is sufficient to recover nearly the same accuracy or perplexity as standard BP, closing most of the gap that purely local methods leave behind.
Although these results bring us significantly closer to truly parallel learning, several avenues remain to fully match BP on large, modern architectures. First, adaptive calibration strategies could reduce the overhead of intermittent BP steps. Second, pariance-reduction training in very wide layers. Finally, extending GrAPE's feedback projections to exploit the specific structure of the considered layers could also help to narrow the remaining performance gap without relying on frequent BP resets. By uniting forward-mode gradient estimates with targeted backpropa- gation corrections, GrAPE lays the groundwork for scalable, parallel and efficient training of deep neural networks.
techniques may further stabilize
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was granted access to the HPC resources of IDRIS under the allocation A0191016927 made by GENCI. This work has received support from the French government, managed by the National Research Agency, under the France 2030 program with the reference "PR[AI]RIE-PSAT" (ANR-23-IACI-0008) and "PEPR-SHARP" (ANR-23-PEIA-0000)
REFERENCES
Mohamed Akrout, Collin Wilson, Peter Humphreys, Timothy Lillicrap, and Douglas Tweed. Deep learning without weight transport. Advances in neural information processing systems, 32, 2019.
Marco PE Apolinario, Arani Roy, and Kaushik Roy. Lls: local learning rule for deep neural networks inspired by neural activity synchronization. 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (ACV). pp- 7807-7816 IERR, 2025.
Florian Bacho and Dominique Chu. Low-variance forward gradients using direct feedback alignment and momentum, Neural Networks, 169:572-583, 2024.
Sergey Bartunov, Adam Santoro, Blake Richards, Luke Marris, Geoffrey Hinton, and Timothy Lilli- crap. Assessing the scalability of biologically- deep learning algorithms and architectures. Advances in neural information processing systems, 31, 2018.
Atllm Güney Baydin. Barak A Pearlmutter, Don Syme, Frank Wood, and Philip Torr. Gradients without hackpropagation. arXiv preprint arXiv:2202.08558 2022.
Stéphane d' Ascoli, Levent Sagun, Giulio Biroli, and Joan Bruna. Finding the needle in the haystack with convolutions: on the benefits of architectural bias. Advances in Neural Information Processing Systems, 32, 2019.
Giorgia Dellaferrera and Gabriel Kreiman. Error-driven input modulation: solving the credit assign- ment problem without backward pass. In International Conference on Machine Learning, pp. 4937-4955. PMLR, 2022.
Louis Fournier, Stéphane Rivaud, Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Can forward gradient match backpropagation? In International Conference on Machine Learning, pp. 10249-10264. PMLR, 2023.
Charlotte Frenkel. Martin Lefebvre, and David Bol. Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks. Frontiers neuroscience, 15:629892, 2021.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition. pp. 770-778, 2016.
Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations. arXiv preprint arX14:2212.13345. 2022.
John F Kolen and Jordan B Pollack. Backpropagation without weight transport. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN' 94), volume pp. 1375-1380. IEEE, 1994.
Julien Launay, Iacopo Poli, and Florent Krzakala. Principled training of neural networks with direct feedback alignment. arXiv preprint arXiv: arXiv:1906.00 100..9905.4555 2019.
Julien Launay, Iacopo Poli, François Boniface, and Florent Krzakala. Direct feedback alignment scales to modern deep learning tasks and architectures. Advances in neural information processing systems, 33:9346-9360, 2020.
Yann Le Cun, Conrad Galland, and Geoffrey Hinton. Gemini: gradient estimation through matrix inversion after noise injection. Advances in neural information processing systems, 1, 1988.

---

Published as a conference paper at ICLR 2026
Timothy P LLillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synaptic feedback weights support enror backpropagation for deep learning. Nature communications, 7(1): 13276, 2016.
Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin Akerman, and Geoffrey Hinton. Backprop- agation and the brain. Nature Reviews Neuroscience, 21(6):335-346, 2020.
Charles C Margossian. A review of automatic differentiation and its efficient implementation. Wiley interdisciplinary reviews: data mining and knowledge discovery, e(4):e1305, 2019.
Theodore Moskovitz, Nicholas Roy, and Jonathan W Pillow. A comparison of deep learning and linear-nonlinear cascade approaches neural encoding. BioRxiv, pp. 463422, 2018.
J Nocedal and SJ Wright. Numerical optimization springer-verlag. New York, 1999.
Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. Advances in neural information processing systems, 29, 2016.
Arild Nøkland and Lars Hiller Eidnes. Training neural networks with local error signals, In International conference on machine learning, pp. 4839-4850. PMLR, 2019.
Maria Refinetti, Stéphane Ascoli, Ruben Ohana, and Sebastian Goldt. Align, then memorise: the dynamics of learning with feedback alignment. In International Conference Machine Learning, pp. 8925-8935. PMLR, 2021.
Mengye Ren, Simon Korbblith, Renjie Liao, and Geoffrey Hinton. Scaling forward gradient with local losses. arXiv preprint urXiv:22/0.0333 2022.
Arani Roy, Marco Paul E. Apolinario, Shristi Das Biswas, and Kaushik Roy. Unlocking SVD-space for feedback aligned local training, 2025. URL bttpss//ppee.eeeeww..et/////// 8AgcicOcsh
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back- propagating errors. Nature, 323:533-536, 1986. URL/https: swwanticscho..oo.........
org/CorpusID: 205001834
Albert Jiménez Sanfiz and Mohamed Akrout. Benchmarking the accuracy and robustness of feedback alignment algorithms, 2021.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint 1et::00880088 2017.
Burr Settles. Active learning literature survey. 2009.
Sam Shleifer and Eric Prokop. Using small proxy datasets accelerate hyperparameter search. arXiv preprint arXiv:1906.0480 04887, 2019.
David Silver, Anirudh Goyal, Ivo Danihelka, Matteo Hessel, and Hado van Hasselt. Learning by directional gradient descent. International Conference on Learning Representations, 2021.
Ravi Francesco Srinivasan, Francesca Mignacco, Martino Sorbaro, Maria Refinetti, Avi Cooper, Gabriel Kreiman, and Giorgia Dellaferrera. Forward learning with top-down feedback: Empirical and analytical characterization. arXiv preprint urXiv:2302.054400 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Matthew Bailey Webster, Jonghyun Choi, and changwook Ahn. Learning the connections in direct feedback alignment, 2021. URL
Bernard Widrow and Michael A Lehr. 30 years of adaptive neural networks: perceptron, madaline, and backpropagation. Proceedings of the IEFE, 78(9): 1415-14444,, 1990.
12

---

Published as a conference paper at ICLR 2026
A DETAILED RELATED WORK
A. LEARNING WITH RANDOM FEEDBACK
Feedback Alignment (FA) introduces paradigm-shifting and biologically plausible alternative
to gradient backpropagation (Lillicrap et al. (016)). At the core of this method, random feedback matrices simplify the weight update process and break the symmetry. FA sequentially multiplies the output error by the random feedback matrices in order to obtain the error at each layer, which is in turn used to update the weights. The random matrices replace the transpose of the weights in the original backpropagation equation. However, the sequential aspect of the update remains, as the error still propagated from layer to layer through the random feedback projections. With Bj the fixed random feedback matrix of the I-th layer, the update can be computed as follows:
d0g = (Bre) o d'(())) with daz = (Bjöa++1) O o((a), We [1,--11
This method draws its inspiration from biological neural networks, which do not exhibit symmet- ric weight transport during learning (Lillicrap et al. 2020), making this learning paradigm more biologically plausible.
Direct Feedback Alignment (DFA) Nøkland (016) goes step further: during training, the error signal is directly projected from the output layer to all hidden layers without modification or intermediate calculations. With this additional simplification, the updates can be easily parallelized:
VI E [l, L), = (Bte) O d((a)).
(7)
FA and DFA have been shown to perform reasonably well on certain tasks and architectures, especially when considering its profound shift from the backpropagation method. While they do not consistently outperform or even compete with backpropagation, their simplicity along with biological plausibility stimulate research scale up their use, well exploration understand their key limitations. Bartunov et al. (2018) for example, show empirically that FA variants perform significantly worse on CIFAR-10 and ImageNet than BP, for convolutional networks in particular.
This is further analyzed by Launay et al. (219)) in which they exhibit a bottleneck effect that prevents learning in narrow layers, especially in the case of convolutional networks. As workaround, some variants of FA showed promising performances on deep CNNs (Moskovitz et al.) 2018). A seminal work by Akrout et (2019) for instance used weight mirroring to adapt the feedback matrices during training, matching BP performances. However these approaches stay sequential, and similar approaches to DFA with target projection, such as DRTP (Frenkel al. 2021), do not compete with BP on more complex convolutional networks.
has also been empirically verified (Launay et al. 2020) that learning under synaptic asymmetry with DFA is possible even with Transformers (Vaswani et al.) 2017). In this particular work, the training the Transformer with DFA is done according to two settings: the 'macro' setting which the feedback is applied after every encoder block and the 'micro' setting, in which it is done after every layer in those blocks. As explained by Launay et al. (2020) in Appendix D, backpropagation through the attention mechanism itself still happens even in the 'micro' setting, meaning that the training process still relies on BP within transformer layers without reaching the same perplexity levels as full BP training.
In their papers. Nøkland (206)) and Refinetti et al. (202)) analyze the underlying dynamics in the FA-like algorithms better explain their ability and inability to learn. A key lesson is that the angle between the update and the true gradient must be lower than + Equivalently, if we denote wt this angle, and B the 1-th layer's feedback matrix, the following inequality must hold:
VL2 Bie
VI E [], cos(wr) = |V Biel
>0
We recognize particular case of the Zoutendijk theorem (Nocedal & Wright 1999), which ensures global convergence when the search direction makes an angle with the steepest descent direction bounded away from x /2, This theorem requires that the step length satisfies either the Goldstein or
13

---

Published as a conference paper at ICLR 2026
strong Wolfe conditions, and this is typically the case with standard learning rates. However, let us stress that the considered convergence is towards local minima and stationary points.
As previously mentioned, the recent work of (Akrout al. 2019) revisits the idea to learn the feedback by emulating a Gollnn-Pollack algorithm (Kolen & Pollack| 1994) or with an estimate of the transpose matrix. This idea facilitates the learning process of FA while reducing the angles w/. This first attempt clearly shows that adaptive feedback matrices enable the learning of networks on which FA previously failed. It also emphasizes the importance of Zoutendijk's theorem, even though the sequential learning process inherited from the FA still inhibits the potential improvements.
A.2 FORWARD ONLY CALCULATIONS
promising avenue toward backward-free training is the double-forward approach, in which two forward passes are used: first forward pass updates an auxiliary or feedback mechanism, while the second forward pass computes the weight updates. The recent paper (Srinivasan et al. 2023) follows this trend and exhibits similarities between two forward-only frameworks,
Forward and PEPITA (Hinton 2022 Dellaferrera & Kreiman 2022). They also show that such algorithms can be approximated by form of feedback alignment with adaptive feedback (AF) weights, modulated by the upstream network weights. The PEPITA learning rule essentially is: oW; (h) - - O (harr") ") with h6 = x - Fe, where F can be viewed as a feedback mapping on the input. Srinivasan et al. (2023) showed that PEPITA implements feedback matrix that depends on the forward weights when F is computed with weight mirroring. Although promising, this method fails to scale to networks deeper than 5 layers.
learning algorithm with an adaptive
Forward Gradient (FG), introduced by Silver et al. (2021) and Baydin et al. (022) employs Forward-Mode Automatic Differentiation (FwAD) proposed in Margossian (2019) to estimate gradients solely through forward passes. Focusing on these forward calculations, recent works explore
unbiased estimations of the gradients, thanks directional derivatives (Fournier al. 2023 Baydin
iet al. (022).. These gradients are then used to update the weights like in standard BP, without needing an explicit backward pass. The essential idea in FG descent is that given a direction vector E 1 computing the Jacobian-vecte product (JVP) of the gradient of the loss along u gives the gradient of the loss function according the direction given by u. This is defined as:
L(O + ô u)) L())
VLu u
lim 8-2
at the parameter point 0. This is used to estimate partial derivatives of the loss with respect to subsets of parameters activations, along random directions.
Although Baydin et al. (2022) showed that sampling the perturbations u the weight space can provide unbiased gradient estimates, |Ren et al. (2022) revealed poor scalability when the number of parameters is large. To address this, they proposed to draw perturbations in the activation space, inspired by Le Cun al. (1988) and Widrow & Lehr (1990). Since the total number of neurons is usually much smaller than the total count of parameters, sampling u, E R. for each layer can substantially reduce both the cost and variance of gradient estimation.
However, recent advances (Fournier et al., 2023) show that in modern settings, even variance-reducing techniques with local losses do not allow to reach performance on-par with standard backpropagation.
A.3 REMINDERS ON ZOUTENDIJ'S 'S THEOREM
Let f R" + R be a twice continuously differentiable function, bounded below on R". Consider the iteration
Øk+1 = 22 + a&pks
where each pk is descent direction (i.e. (s( pk < 0) and the step length og satisfies the Wolfe conditions:
f(z + akpk) < f(x)) + a Vf(x)) Pk. 0<a < <1
(8)
fx(x + axps) T Pk 2 c2 Vf(x)) Pk-
(9)
14

---

Published as a conference paper at ICLR 2026
Theorem A.1 (Zoutendijk). Under these assumptions, the series
-V)(xx)) [[(() ||px||
2o |[
where cos 02
converges. In particular.
inf [V 0.
Sketch of Proof. Starting from the decrease guaranteed by the Armijo condition equation g and invoking the curvature condition equation | that there exists a constant M >> O such that
one shows (see Nocedal Wright (1999), Chapter 3)
f(xx) 2 M cos [[
Summing over I then yields the claimed convergence of the series.
Zoutendijk's 's theorem thus states that if each search direction Pa remains positively aligned with the negative gradient, i.e. ( Ps < 0, and the step lengths or > satisfy the Wolfe conditions equation equation 9 then k== co [[V](=))]l* < co, which in turn implies
lim inf [V
A.4 FDFA REPRODUCTION DETAILS
We deliberately excluded Bacho & Chu (2024) because our attempts to reproduce their results revealed multiple inconsistencies:
On AlexNet (CIFAR-100), their code yiclds 55.03% + 0.16 accuracy over 10 runs, instead the 57.27% = 0.11 they report.
Their DFA baseline (from the Webster al. 2021 code) achieves 48.03% + 0.61, yet they report only 35.75% + 0.58. This discrepancy appears to arise from applying batch-non norm exclusively FDFA.
Re-implementing their method in BiocTorhh with their hyperparameters gives no improve- ment over standard DFA (33.59% for FDFA vs. 42.59% for DFA).
Finally, integrating GrAPE into their code gives 60.35% + 0.26 on AlexNet (no augmen- tation), essentially matching BP, which conflicts with the known gap between BP and DFA-style methods.
Collectively, these points suggest unintended reliance on backpropagation in their pipeline and cherry-picked reporting, undermining reproducibility and justifying our decision to omit FDFA from our baselines.
Novelty of GrAPE, While previous works combine DFA with forward-mode AD, GrAPE is the first to:
derive feedback updates from a cosine-similarity alignment loss grounded in the intuition of Zoutendijk's theorem and supported by estimator-level Jacobian alignment guarantees, demonstrate scalability to deep architectures (VGG, ResNet, Transformers) with rigorous empirical validation,
provide estimator- levell Jacobian alignment guarantees together with extensive benchmarks using verified implementations (e.g. BioTorch).
While open to citing FDFA in future versions for completeness, believe that its omission in the present version is fully justified and does not diminish the novelty or relevance of our contribution.
B FORWARD-GRADIENT ESTIMATOR: FROBENIUS COSINE AND CONCENTRATION
In this appendix we justify the lower bound on the expected Frobenius cosine between the true Jacobian Ji and its rank-1 estimator J1 (Jp)) where NN In)), together with an O(1/VB) concentration rate for the batched estimator.
15

---

Published as a conference paper at ICLR 2026
B.1 REPRESENTATION VIA THE SPHERE
Let ~ ((0, I. We write
p= r:= IPpl s:=p/lpp
It is classical that is uniform on the unit sphere Sth=1 and independent of For the rank-1 estimator
T (Jip)p", ", straightforward computation shows that the Frobenius cosine satisfies
cOsP (J. = (J.. (J.
1Jpll
|
| 11
pl
|
so that
1
E COS (Jr, = | Es|[Jss||
S ~ Unif(S"
(10)
B.2 LOWER BOUND VIA THE TOP SINGULAR VECTOR
Let the singular value decomposition of Ji be Ji UEVT, and denote by 0max || the largest singular value, with associated right singular vector and left singular vector u1 For any unit vector and any z, || u æl; choosing u u1 yields
J
Ju Jis] = omax v s| = I| v
Taking expectations gives
E Tisl
|| E s/.
By rotational invariance of the uniform distribution on the sphere, the scalar v s has same distribution the first coordinate 81 of Unif(5m=-1) Hence
Egg Jis
2
| F/s1
standard computation (properties of the spherical distribution) shows that
T
E s1
Using classical bounds on ratios of Gamma functions (Gautschi's inequality) one obtains, for all ny 2,
2
E E|s
(11)
Combining this with the previous inequality yields
2
(12)
Substituting equation 12 into equation
10
yields the lower bound used the main text:
TTll
cosy (J,, (Jr, Fi]
Z
(13)
[
[
which is strictly positive whenever Ji # O.
B.3 BATCHED ESTIMATOR AND CONCENTRATION
Consider the batched estimator obtained by averaging B independent rank-1 estimates based on P1..... PB N(, ):
B
F(BB = B
(JTp:)P
C = cos (J. (p:))
16

---

Published as a conference paper at ICLR 2026
We are interested in the empirical mean
1 B CB
Σ C.
The map p C(p) (()) is smooth, bounded function of p and is Lipschitz with respect to p with a constant depending only on J. By standard concentration results for Lipschitz 0 (depending the dimension and functionals of Gaussian vectors, there exists constant
condition numbers of Ji) such that for all > 0,
P([BB BC(p) 2 1) S 2exp( -c
Equivalently,
Std(CB) = o 1 )
VB
so the empirical Frobenius cosine concentrates around its expectation at rate O(I/VB).
B.4 CONVERGENCE UNDER POSITIVE EXPECTED COSINE
We recall a standard stccastic-approximatinn result tailored to our setting.
Theorem B.1 (Convergence under positive expected cosine). Let IRd R. be differentiable, bounded below, with Lg-Lipschit gradient. Consider
Oct1 = 0g M,d,
= VL((:),
where (Fi) is the natural filtration and the step sizes satisfy
α The
20
Σ 1/2 00,
Σ r 0o.
Assume there exist constants K > 0, < 00, < oc such that, for all
E [[ de)
Fi] xklgg 112
(14)
E [||d 1
Fi] s C'Ilg 1| + 022.
(15)
Then
n El <
and Iim Elg. 0.
In particular, the iterates converge to stationarity in expectation.
Proof.
Since I, has Lig-hipschitz gradient, the standard descent lemma gives, for all 0, 0' E Rd.
L(0') < L(0) + (VL(0), 0) -) + Lg 110' 011
Apply this with 0 = Og and 0' 0,+1 0, mdde
L(0(+1) - L(0,) + (V -mide) + Lg ni |ddel1
2
= L()) - n (g., de) + Lg ni ||d/112
2
Taking conditional expectation w.r.t. the filtration F.,
E [[(00+1) | < I((..) me Kgc, de) Fi] T.
m E[||de|l F...
2
Using the two assumptions
B((g,, d+) [F]] x
E[|dd:11 Fi]
Cll +
17

---

Published as a conference paper at ICLR 2026
obtain
E [[(((0+11) |Fr ((0+)) n Lg
2 n (Cllbel12 + 02)
-((,) - (1= L melgeel2 + 0 1/2
Because E: vi < oc, we have Th + 0 as 1 co.
Hence there exists index such that for all
t t to.
L LgC
K 2
11t
2
2
For L 2 lo. this yields
0g
B[L(0++1) |F]] S L(O.)
-2 mh wellle11
1/2
(16)
2
Now take the full expectation of equation 16 and use the tower property E EE
and
Fil = [[:
Tg
EI(0++1) I EI((+)
2 The Ellga 11
nt
z to-
2
Rearranging,
Lg
2 Th EE gg s BL(O.) (E(((++)
1/:
2
2
Sum this inequality from f = to to T:
T
T
(EL(O)) - EL(((+1)) + Lyo²
T
K
M
El9
12 <
Σ
n
2
2
t=
t=to
t=
=E(()t,) EL((r--) + Tgg
Σ
T
t=fo
By assumption, L is bounded below, say L(0)= 2 I., for all 0. Therefore EI(0r+1) 2 L,, and we obtain
T
0g0²
T
K
g < EI((O.)) L. +
2
2
t=
t=
Letting T + oo and using >, vi < oo yields
n g 00.
Adding the finite partial sum over t < to shows that
g < oc.
It remains to show that this implies lim inf E Eggel 0. Suppose, for contradiction, that there
exists ε > and é such that Ellgell 2 a for all 2 t1. Then gl 2 s" for all 2 f1, and therefore
ne g
n
By assumption, E 7 xc, so the right-hand side diverges, contradicting the finiteness of Σ. 1l Hence we must have lim inf._ Ellge|| = O.
This establishes the claimed convergence stationarity in expectation.
18

---

Published as a conference paper at ICLR 2026
B.5 A COMPOSITION BOUND FOR FROBENIUS COSINES
We quantify how noise in the JVP estimator and imperfect learning of B, interact.
Lemma B.2 (Frobenius cosine composition). Let Be, Jr. Je E mn be nonzero and define
(A, B)r
cos (A,
(), Tr(AA
||||B|B||
Then
(Bs, 2 cosp(Be: J1) cos p (e) V c Jt) V cos3 Je).
Consequently, if cos (Be, J1) 2 on and (Jr, Jo) So, then
cosy J&) 2 Ya 3p) a V 11-a V 1
Proof. Let H. be real Hilbert space with inner product (+ and induced norm ||-|| In our application. H. is the space of matrices with the Frobenius inner product (A, B)F F = Tr(A B), but the argument holds for any Hilbert space.
Let u, w, x E be unit vectors:
||
|w|||
||a|| 1.
Define
a (u, w), Jao), [8), |c| < 1.
3 = (w, u)}
c= (u, v).
By
Consider the 3 Gram matrix of (u, w. v):
uu u)
(u, w)
(u, v)
a
G =
(w, u)
(w, w)
(w, v)
1
(v, u)
(v,w) w))
(v,0)
3
1
Since is Gram matrix, it is positive semidefinite (PSD), hence det 0.
We compute the determinant explicitly. A direct calculation yields
a 1
det(G)
a
3
c
B
a² + 2a,c - -2 +1
(e2² 2a + (a² 32 -1)).
The constraint (el(G) 2 is therefore equivalent
- 2a3c (a² 32 1) 0. We view the left-hand side as a quadratic polynomial in c,
g(c) c² 2a,c + (a² - 1). 1).
Since the coefficient of is > 0, the inequality q(c) - means that c lies between the two (real) roots of q. Compute the discriminant:
A (-2a:3)2 - 4( + 3² 1) 4a?3² 4(a² + 82 -1)
=4(a23² a² B +1)
441- ((1-a) 2² 8² +
4(1 - az B²).
Since |a:1, [8] < 1, we have 1 0 and - f² > 0, A expected. The roots of g are
c = of + / (1 a²) 32).
19

---

Published as a conference paper at ICLR 2026
The inequality g(c) 0 therefore implies
S c S Ct.
i.e.
aB - v(1- 32) S c z aB + a11_a²(11- 32).
In particular, we obtain the desired lower bound
2 0.3 v a² V B
We now instantiate this with Frobenius-normmalzeed matrices. Let Bž, Je, Je mmxx be nonzero, and define
Be
Je
J
U
w
o:
Bell
||
|elll
viewed elements of the Frobenius inner-product space. Then
a (u, w = cosp(BB, Jt), B (w, u)F = cosF(Je,
c
Jt),
and the bound above reads
cos Jt) 2 cosr( Be, Je) cos (Je, - V cos (Be, Je) V (Je. Je).
This is exactly the claimed inequality. The "Yo(ag, 30)" form in the lemma statement follows by substituting lower bounds oo, Po for the two intermediate cosines.
C FEEDBACK MATRICES UPDATE DETAILS
In the main paper, the theoretical alignment measure between feedback matrix Bj and the JVP-based Jacobian estimator Ji is the Frobenius cosine
(Bt,
cosp (Bi.
:=
| 1|
Writing B; [b/ and J. li! ............... ..", ..""
we can express this as
Σn by j
cos Bi, Ti
Σ uk cos (f)
- |b* 12 V EE 115
where
by |b
| 11:1
cos (bs/ ())
and
1.
| B |/1F
Thus the Frobenius cosine is weighted average of the columnwise cosines, with weights proportional to the product of column norms. In our implementation we normalize the columns of B, after each aligmment step, and Ji has columns of the form j Gaussian perturbation and 9i a common JVP), so the |lif || differ mainly through |px Since these scalars are i.i.d. and concentrate around their mean, the weights w do not vary dramatically across k. and the unweighted average of per-column cosines provides a simple and effective proxy for cosp J)).
St (with pk: scalar component of the
In practice we therefore minimize the local cosine alignment loss Calign (Bi) = Cty where
cos cos Jf')
TM
the empirical mean of per-column cosines. This loss coincides with the Frobenius cosine up to the weighting discussed above, and is cheaper compute while still encouraging layerwise alignment.
20

---

Published as a conference paper at ICLR 2026
over columns, the gradient can be written column by column. Specifically, for each & minimize the single-term contribution [1 - coof scaled by 1/m. Let 11B > be the learning rate for layer I. Then the per-column update can be written as
Because Calign decomposes
bf bf
BB bbb
cos (b/ (bf
k= == l......,
y xx] Tyll so the gradient of [1 cos(x. y)] its negative.
We recall that Vx cos(x, y) Each column by is thus updated
T Txx|aa
bf
by
BB
V
cos(bf.Jt))
(b b b
by
7]Bt
(17)
|
|i
15
Applying this update for each 1,...,,.. ny increases the mean columnwise cosine and thus aligns Bt with T.. Because Lalign decomposes as sum per-column terms, updating all columns in parallel is equivalent taking gradient step Calign whole. We then renormalize each column by after this step to keep norms bounded and maintain the interpretation of the cosine purely directional alignment measure.
D FLOPS, CRITICAI- PATH AND TIME ANALYSIS
D.1 FLOPS AND CRITICAL-PATI ANALYSIS
Throughout this subsection we adopt the standard GEMM view, modeling each layer's compute (batched) matrix-matrix multiply after the usual reshapes/lowering, which yields simple, comparable
FLOP counts.
Setup and notation.
Let the network have layers l = -....., Denote by
F Σ (
the FLOPSs of one mini-hatch forward pass under the GEMM view (for conv layers this corresponds to the lowered GEMM). Let dout the output dimensionality,, and define the per-layer costs:
Opro) = Ane dow)) (feedback projection),
Otllian
Ang dom) (local alignment update on B₁).
We model a single Jacobian- vector product (JVP) as an overhead a times the layer's forward cost; writing layer-averaged gives total forward+JVP factor + a).
Backpropagation (BP).
L
batchh BPP
(((eeinne) + Ana-ina)) ~ (1+ t=1
Σ
(18)
where the backward-to-forvee ratio 3 is typically in the range of 2 depending on layer type and implementation. The backward sweep sequential across layers on the critical path.
Direct Feedback Alignment (DFA).
batth DFA
F
Σ
Croj E=1
F
of dow Σne)
(19)
The forward is sequential across layers; the per-layer projections layer-local and may be launched in parallel, their contribution to the critical path is max, Caroj
21

---

Published as a conference paper at ICLR 2026
GrAPE (alignment-only).
pbatch GrAPE
F1 + Σ Opto
Ce t=1
(1+ + Odou Ens)
(20)
Here JVP per layer runs during the forward trace (scaling the forward path by ()). Projection and alignment are layer-local and parallelizable, contributing max; (Cr + C. the critical path.
GrAPE with calibration every epochs. A single BP mini-batch is added per T epochs. an epoch contains No mini-batches, the amortized per-epoch overhead is
Eepoch GrAAP+Cal = Ng bbatch GrAPE
+
Fbatch
(i.e., TNs
of an epoch units of mini-batches).
T
(21)
Rule-of-thumb comparison (per mini-batch). If Cowd Σ, O(n--1nn))) then
Cfwd
and Opro Calign
Fbatch BP
2 2F,
Fbatch GrAPE (1+ F + lower-order terms in dumt Σn ng.
Practical JVP overhead. Here denotes the JVP overhead relative forward pass: the JVP alone costs aF, so the combined forward+JVP cost is (1 a. modern GPUs with fused dual-number implementations, we typically observe 0.8 hence
forward+JVP s (1+a)F E [18 2.
Transformers (macro/miero). For maero (per-block) and micro (per sub-layer) variants, Cproj and Calign apply at the chosen granularity. Between calibrations, multi-head attention backward chain is bypassed; GrAPE's JVPs run in the forward trace, while block/sub-layer projection and alignment remain parallclizable.
Arithmetic bound The expressions above capture arithmetic work and dependency structure; realized wall-clock depends on kernels, fusion, memory bandwidth, and scheduling. Under an idealized critical-path model where BP's backward cost about twice the forward cost (i.e., f so BP takes (1 + F)F = 3F), (ii) GrAPE's per-layer projection/alimmment are fully overlapped across layers, and (iii) fused dual-number JVPs yield 0.8-13. the arithmetic critical-path ratio is
BP critical path
1 1
3
3
E
[1.3, 1.7].
GrAPE critical path
1+a
[1.8,
2.3
We therefore view 1.3-1.7x as an optimistic arithmetic upper bound (not measured wall-clock speedup). Establishing realized parallel speedups left to future work.
D.2 PRELIMINARY TIMING EXPERIMENT
To complement the arithmetic and critical-path analysis above, we implemented small proof-of- concept layer-parallel prototype on a Transformer with hidden size 128 (batch size 256, sequence length 64) single NVIDIA A100. We Python-level CUDA streams and simple double- forward trick to compute JVPs, while BP run in the usual fully sequential manner. This implemen- tation does not kernel fusion custom CUDA kernels, the numbers below should be viewed as conservative.
E EXPERIMENTAL DETAILS
As mentioned earlier, we base most of our experiments on the Biotorch open source library. While the experiments Transformer exactly the same hyper parameters in Launay al. (2020) the
22

---

Published as a conference paper at ICLR 2026
Table 5: Mean per-batch runtime (ms) for small Transformer (hidden size 128, batch size 256, sequence length 64) on a single A A100, comparing sequential BP and layer-parallel GrAPE prototype using Python-level CUDA streams.
Depth
BP (ms)
GrAPE (ms)
2
9.1
3.0
4
17.5
6.2
8
35.8
12.1
other experiments are set with specific set of tuned hyperparameters for each method. For GrAPE, we set the optimizer of the local updates to SGD with 0.9 momentum with learning rate 0.01. The BP calibration steps are conducted on batch of size 128, with SGD with 0.9 momentum and learning rate 0.01. train AlexNet and VGG-16 for 100 epochs with 128 batch size. The Resnets are trained for 200 epochs with 128 batch size. If SGD is used, default 0.9 momentum was applied.
The specific used hyper-parameters are:
AlexNet
VGG16
ResNet-20 20
ResNet:56
BP
DFA
GrAPE
BP
DFA
GrAPE
BP
DFA
GrAPE
BP
DRA
GrAPE
Opeimizer
Adam
Adam
Adam
SGD
SGD
Adam
SGD
Adam
Adam
SGD
Adam
Adam
LR
0.001
0,0001
0.0001
0.1
0,001
0,0001
0.1
0
0.01
0,0005
docay
Milestones
[40,
75]
[30,
90]
(60. 120, 160)
160]
same
Weight decay
0.0001
0.0001
0.0001
0.0005
0.0001
0.0001
0.000S
0.0001
0.0001
0,000S
0.0001
0.0005
Table Hyperparameter settings for deep networks on CIFAR-100 and TinylmageNet: AlexNet, VGG16. ResNet-20, and ResNet-56.
23


Reviews:
[{"review_id": "ZXITYPXiyM", "reviewer": "Reviewer_WNQG", "summary": "The paper proposes GrAPE (Gradient-Aligned Projected Error), a feedback-based learning rule designed to relax the strict sequential dependence of backpropagation (BP). Building on Direct Feedback Alignment (DFA), GrAPE updates feedback matrices by minimizing a cosine loss that aligns them with rank-1 Jacobian estimates computed via forward-mode Jacobian–vector products (JVPs). This alignment is theoretically shown to produce descent directions in expectation. To mitigate drift, the method introduces BP calibration, a full BP step performed every $T$ epochs. Experiments on CNNs  and a Transformer  show that GrAPE outperforms DFA and approaches BP performance, especially when calibrated frequently (e.g., $T=1$).", "questions": "• Could the authors provide a more explicit derivation of Eq. (4) and clarify the assumptions under which the expected positive cosine alignment holds?\n\n•  How much wall-clock speed or memory reduction does GrAPE actually provide compared to BP or DFA, particularly when T=1?\n\n•  Since the strongest results rely on frequent BP calibration, is the approach truly parallelizable in practice? How would these updates be scheduled on real hardware?\n\n•  How does GrAPE differ technically and empirically from prior feedback-alignment methods such as Deep Learning Without Weight Transport [1] and SVD-Space Feedback Alignment [2]?\n\n•  Could the authors discuss more explicitly how GrAPE compares to other local learning rules such as LLS [3] and Local Error Signals [4], which achieve near-BP performance without periodic BP corrections?\n\n•  What are the specific deployment scenarios where GrAPE would offer meaningful advantages over BP, given that BP calibration reintroduces global synchronization", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper addresses an important question: can we train deep networks without the strictly sequential backward pass of BP? Its motivation, enabling layer-parallel or biologically plausible learning, is timely. Using JVPs to align feedback matrices offers an interesting bridge between forward-mode differentiation and DFA-style local updates, supported by a simple convergence argument. The empirical evaluation is broad, spanning CNNs and Transformers, and shows that learned feedback alignment can indeed reduce the performance gap with BP. The inclusion of a calibration mechanism acknowledges practical instability and provides a workable hybrid between local and global learning.", "weaknesses": "Despite its conceptual appeal, the practical advantages over BP remain unclear. The paper claims that GrAPE reduces computational cost, but forward-mode JVPs roughly double the forward-pass work, and the “BP calibration” step reintroduces full sequential backpropagation, often every epoch. No runtime, memory, or parallel-efficiency results are reported, so the efficiency claims remain speculative. Moreover, it is not clear when parallel updates become practically important if the memory requirements of the method remain the same as those of BP.\n\nThe contribution of alignment itself is also not novel: prior work such as Deep Learning Without Weight Transport [1] and the more recent SVD-Space Feedback Alignment [2] have already shown that feedback matrix alignment can restore BP-level performance. In particular, [2] explores a local cosine loss function for aligning the singular vectors of the feedback matrices with those of the forward path, yielding better gradient descent directions; this connection should be discussed further. GrAPE differs mainly in how it estimates the Jacobian (via JVPs), but the conceptual insight, that alignment matters, is already well established. Similarly, other local learning rules such as LLS [3] and Local Error Signals [4] achieve BP-like performance without relying on periodic BP corrections.\n\nFinally, the theoretical section is terse: the derivation of Eq. (4) and the connection to Zoutendijk’s theorem are only sketched, leaving unclear the precise assumptions under which the convergence argument holds.\n\n[1] Akrout, M., Wilson, C., Humphreys, P., Lillicrap, T. and Tweed, D.B., 2019. Deep learning without weight transport. Advances in neural information processing systems, 32.\n\n[2] Roy, A. et al. (2025) ‘Unlocking SVD-Space for Feedback Aligned Local Training’. Available at: https://openreview.net/forum?id=8Agcic0csh.\n\n[3] Apolinario, M.P., Roy, A. and Roy, K., 2025, February. Lls: local learning rule for deep neural networks inspired by neural activity synchronization. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (pp. 7807-7816). IEEE.\n\n[4] Nøkland, A. and Eidnes, L.H., 2019, May. Training neural networks with local error signals. In International conference on machine learning (pp. 4839-4850). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TBxQWifItY", "reviewer": "Reviewer_DBs7", "summary": "This paper introduces GrAPE (Gradient-Aligned Projected Error), a learning algorithm designed to scale Direct Feedback Alignment (DFA) to modern deep architectures such as CNNs and Transformers. It uses the follwing udeas:\n- Forward-mode Jacobian estimation: rank-1 Jacobian approximations obtained via forward-mode automatic differentiation ) are used to align each layer’s feedback matrix through a local cosine-similarity loss.\n- Sparse backpropagation calibration: occasional full BP updates on a single mini-batch serve to re-anchor weights and mitigate drift in very deep networks.\n\nThe paper provides a theoretical analysis showing that the expected cosine alignment between the estimated and true Jacobians is positive, and invokes Zoutendijk-style convergence arguments to guarantee convergence to stationary points. This is a very interesting developemn in the context of DFA. Empirically, GrAPE achieves substantial improvements over DFA, DRTP, and other BP-free baselines on CIFAR and WikiText benchmarks, and approaches full backpropagation performance on VGG, ResNet, and Transformer models when using one BP calibration per epoch.\n\nOverall, the work provides both a conceptually clean and empirically convincing bridge between feedback alignment and optimization theory, showing that layer-parallel learning with forward-only signals can be made competitive with BP.", "questions": "i) There has been recent progress on optical implementations of DFA (e.g., Wang et al., 2024, arXiv:2409.12965) showing impressive throughput and energy efficiency at scale. It would be interesting to hear the authors’ thoughts on how GrAPE might relate to or benefit from such hardware developments. Could its adaptive feedback and occasional BP calibration be implemented efficiently in optical or neuromorphic systems, and would the JVP alignment step introduce additional cost or complexity?\n\nii) The paper provides a clear FLOP-based comparison between BP and GrAPE, but no timing or memory results. Could the authors share any empirical measurements, even on small networks, to verify that the theoretical analysis matches practical runtime or memory usage?\n\niii) How robust is GrAPE to the choice of calibration interval T? In particular, could an adaptive strategy that monitors cosine alignment offer a better trade-off between accuracy and calibration cost? Relatedly, have the authors tried using multiple perturbation directions per layer to reduce variance, and if so, how does that affect parallelism and runtime?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is a strong and timely contribution to the literature on biologically inspired and parallel learning algorithms. The combination of feedback alignment, forward-mode differentiation, and provable convergence is original and well executed. The empirical scope (from LeNet to Transformers) convincingly demonstrates the generality of the approach. The work provides theoretical clarity missing from previous DFA/forward-gradient studies and could serve as a reference point for future algorithmic or hardware developments.\n- Novel combination of feedback alignment and forward-mode gradient estimation, with clear theoretical grounding.\n- First rigorous convergence argument for DFA-like methods based on Zoutendijk’s theorem.\n- Empirical scalability: works on VGG, ResNet, and Transformer models, closing much of the gap to BP.\n- Reproducibility and rigor: transparent experimental setup, use of BioTorch, and fair baselines.\n- Balanced discussion of limitations and potential extensions.\n= Provides a conceptual bridge between biologically plausible local learning rules and modern optimization.", "weaknesses": "i) Lack of demonstrated computational gains.\nAlthough GrAPE is designed to enable layer-parallel updates, the current implementation remains fully serialized. No wall-clock or energy improvements are reported, and the FLOP analysis in Appendix C is only theoretical. It would be useful to include even small-scale timing or memory comparisons to assess the practical benefit.\n\nii) Dependence on frequent BP calibration.\nMost deep-network experiments require a full backpropagation step every epoch (T = 1) to reach competitive performance. This limits how “BP-free” the method truly is, and raises questions about whether GrAPE’s main advantage lies in its theoretical framework or in genuine computational savings.\n\niii) Variance of the forward-mode estimator.\nThe rank-1 Jacobian approximation introduces variance that grows with layer width. While the issue is discussed, no practical mitigation (such as averaging multiple perturbations or using low-variance directions) is explored experimentally.\n\niv) Limited discussion of hardware relevance.\nGiven the recent success of hardware-accelerated feedback learning (e.g., optical DFA systems, Wang et al., 2024), it would strengthen the paper to discuss how GrAPE could map onto such platforms, or at least estimate realistic parallel speed-ups under non-ideal hardware constraints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PGS6I49DXl", "reviewer": "Reviewer_onV1", "summary": "This paper introduces GraPE (Gradient-Aligned Projected Error), a hybrid feedback-alignment method that enables layer-parallel training of deep neural networks. The GraPE algorithm builds on the ideas of (direct) feedback alignment, where the error is back-propagated to the layers of a neural network via random feadback connections. The core innovation of GraPE is to use relatively cheap, forward-mode Jacobian-vector products to estimate rank-1 Jacobians, which are then used to align each layer's fixed random feedback matrix via a local cosine-alignment loss. In very deep models, GraPE also performs infrequent backpropagation (BP) steps on single mini-batches. The authors provide theoretical guarantees of positive expected alignment and convergence using Zoutendijk-style arguments. Empirically, GraPE significantly outperforms vanilla FA or DFA and similar BP alternatives and, with occasional BP calibration, closes most of the performance gap with standard BP on models like VGG-16, ResNet-20/56, and Transformers, while retaining layer-parallel updates for the vast majority of training steps.", "questions": "- Could the authors provide any preliminary estimates or a more detailed discussion of the expected wall-clock time compared to standard BP, even with a serialised implementation? I would gladly increase my score if this issue is addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- This work achieves a major step forward of alternative learning algorithms to BP: making feedback-alignment training viable for modern, deep architectures (CNNs, ResNets, Transformers), a problem where prior methods (DFA, FA) have consistently failed. \n- The key idea of using forward-gradient estimates to align fixed feedback matrices is novel and neat. It creatively combines ideas from two previously separate lines of work (random feedback and forward-mode differentiation) into a single algorithm.\n- The paper is well-written: the arguments are clear, and the authors performed extensive tests of their algorithm in a reproducible experimental setup using the BioTorch library.", "weaknesses": "- Perplexingly, the authors do not address the most obvious benefit of their algorithm: the wall clock speed-up due to parallelisation. There is a tentative FLOPs analysis  in appendix XC, but the actual runtime benefit remains unclear. The authors seem to explain this gap in the paper by the standard libraries they use, but leaving this test out substantially weakens the point of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y8kgqVpIGR", "reviewer": "Reviewer_AHBU", "summary": "The authors address the problem of computing an efficient descent direction for training deep neural networks. They propose GrAPE, an adaptation of direct feedback alignment techniques in which the feedback matrices are updated to improve the alignment between the computed descent direction and the true gradient. The practical performance of the method is compared against backpropagation, FA, DFA, DRTP, and PEPITA across several network architectures.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* **S1**: I find the paper globally clear and well written, except in one point (see **W2**).\n* **S2**: The paper leverages Zoutendijk’s theorem to propose a modified version of feedback alignment that yields better descent directions.\n* **S3**: The method is evaluated on a wide range of architecture and achieves good performance, closing the gap with backpropagation.", "weaknesses": "* **W1**: The code for the experiments is not provided, hindering reproducibility.\n* **W2**: By reading 3.4, I understand that the calibration step happens once every $T$ epoch, and thus at most once in one epoch. However, the way Algorithm 1 is presented suggests that when $\\mathrm{epoch}\\, \\mathrm{mod}\\, T = 0$, a calibration step is performed for each minibatch. Could the authors clarify this point?\n* **W3**: The JVP estimator has variance proportional to layers' dimension. This limits the interest of GrAPE compared to BP for wide neural networks because of the potential need of frequent BP calibration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YETmRt9RFL", "reviewer": "Reviewer_8skU", "summary": "The paper introduces GrAPE, a hybrid direct-feedback method that learns layerwise feedback matrices using forward-mode Jacobian–vector products (JVPs).\n\nFor each layer~$\\ell$, it forms a rank-1 Jacobian estimate $\\hat J_\\ell = (J_\\ell p)\\,p^\\top$ from a random perturbation $p$, and then aligns a trainable feedback matrix $B_\\ell$ to $\\hat J_\\ell$ via a cosine-similarity objective.\n\nThe network parameters are updated in a DFA-style manner using the output error routed through $B_\\ell$, with infrequent backpropagation calibration steps to stabilize training in deep/wide models.\n\nThe analysis establishes positive expected alignment (with variance decreasing by batching) and invokes Zoutendijk-type convergence to stationarity under Wolfe/strong-Wolfe step-size assumptions and descent directions.\n\nEmpirically, the method scales to modern CNNs and Transformers, narrows the gap to BP relative to DFA, and leaves the feedback-refinement step layer-local and parallelizable in principle.", "questions": "Formal convergence theorem:\nCurrently, Zoutendijk and the Goldstein/strong Wolfe conditions are used heuristically to motivate the alignment loss and feedback refinement, but there is no formal convergence guarantee for the actual stochastic update.\nSince the expected positive cosine is derived between $J_\\ell$ and $\\hat J_\\ell$, while the algorithm maximizes cosine between $B_\\ell$ and $\\hat J_\\ell$, could you please provide a \\emph{formal theorem} that yields convergence to stationarity for GrAPE.\nConcretely, specify conditions under which the full update direction is a descent direction with high probability (e.g., $\\nabla f(\\theta_k)^\\top \\Delta\\theta_k < 0$), and state the resulting convergence claim (stationarity) and how it depends on width $n_\\ell$, perturbation/batch count $B$, etc.\n\nAlignment guarantees between $B_\\ell$ and $J_\\ell$\nIs it possible to provide a lower bound on the alignment between the learned feedback and the true Jacobian,\\cos(B_\\ell,J_\\ell)?\nAs supporting evidence, could you also please add diagnostics:\n  (i) an empirical descent test: the frequency of $\\nabla f(\\theta_k)^\\top \\Delta\\theta_k < 0$ under GrAPE vs.\\ DFA; and\n  (ii) alignment to the true Jacobian: plots of $\\cos(B_\\ell,J_\\ell)$ over training.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Clear hybrid design: Learns layerwise feedbacks via forward-mode JVPs and performs DFA-style updates, combining DFA's simplicity with forward-gradient signal quality.\n\nTheoretical support for feedback learning: Establishes positive expected alignment between the rank-1 Jacobian estimate and the true Jacobian, with variance reduction via batching, justifying alignment of $B_\\ell$ to $\\hat J_\\ell$ with Zoutendijk-style arguments.\n\nPragmatic stability via sparse BP: Proposes a novel occasional backpropagation calibration that improves stability/quality in deep or wide networks while retaining layer-parallel updates for the vast majority of steps.\n\nParallelizable in principle: The feedback-refinement step is layer-local, enabling parallel execution across layers (implementation permitting).\n\nStrong empirical results: On CNNs and Transformers, GrAPE narrows the gap to BP and consistently outperforms standard DFA under comparable settings.", "weaknesses": "Convergence is heuristic rather than theoretical: The paper invokes Zoutendijk under Wolfe/Goldstein step-size assumptions and descent directions, but the actual update is stochastic and the per-layer cosine is only \\emph{positive in expectation}, with the bound scaling like $1/n_\\ell$, which can be very small in wide layers.\nEq.~(5) is labeled a ``\\emph{sufficient local condition},'' yet there is no theorem showing local convergence. The claim should be softened or supported by a formal (e.g., high-probability) descent lemma.\n\nCompounding noises: \n(i) The rank-1 JVP estimator $\\hat J_\\ell=(J_\\ell p)p^\\top$ aligns with $J_\\ell$ only in expectation and can yield negative per-batch cosine unless perturbation count/batch size is large;\n(ii) the feedback $B_\\ell$ is then aligned to $\\hat J_\\ell$, compounding estimation error.\nThe paper would be substantially improved by providing lower bounds on $\\cos\\big(B_\\ell, J_\\ell\\big)$.\n\nParallelism not demonstrated:\nAlthough feedback refinement is parallelizable in principle, the implementation serializes these updates and reports no wall-clock speedups. Concrete parallel benchmarks (speed/memory vs.\\ BP) would be appreciated.\n\nBP calibration is under-discussed:\nWhile the paper ablates the calibration period $T$, the role of BP calibration remains under-theorized and under-explained.\nSince calibration is pivotal to performance (often $T=1$), more discussion about the role of BP calibration and clarification o;n its theoretical role would be appreciated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)