Accelerated training through iterative gradient propagation along the residual path
Authors: Erwan Fagnou
Paul Caillon
Blaise Delattre
Alexandre Allauzen
Venue: ICLR 2025 Oral (2025)
Keywords: optimization
efficient training
Primary area: optimization
TL;DR: We propose Highway backpropagation, a parallelizable algorithm that accelerates training by leveraging residual connections, achieving significant speedups with minimal performance loss across various architectures.

Abstract: Despite being the cornerstone of deep learning, backpropagation is criticized for its inherent sequentiality, which can limit the scalability of very deep models.
Such models faced convergence issues due to vanishing gradient, later resolved using residual connections. Variants of these are now widely used in modern architectures.
However, the computational cost of backpropagation remains a major burden, accounting for most of the training time.
Taking advantage of residual-like architectural designs, we introduce Highway backpropagation, a parallelizable iterative algorithm that approximates backpropagation, by alternatively i) accumulating the gradient estimates along the residual path, and ii) backpropagating them through every layer in parallel. This algorithm is naturally derived from a decomposition of the gradient as the sum of gradients flowing through all paths, and is adaptable to a diverse set of common architectures, ranging from ResNets and Transformers to recurrent neural networks.
Through an extensive empirical study on a large selection of tasks and models, we evaluate Highway-BP and show that major speedups can be achieved with minimal performance degradation.

Published as a conference paper at ICLR 2025
ACCELERATED TRAINING THROUGH ITERATIVE GRADIENT PROPAGATION ALONG THE RESIDUAL PATH
Erwan Fagnou', Paul Caillon', Blaise Delattre & Alexandre Allauzen 1.3
1 Miles Team, LAMSADE, Université Paris Dauphine- PPL, Paris, France
2 Foxstream, Vaulx-en- on-Velin, France
3 ESPCI PSL., Paris, France
{ {surname @ddaeeeindddeeeeee periiinne.ss........
ABSTRACT
Despite being the cornerstone of deep learning, backpropagation is criticized for its inherent sequentiality, which can limit the scalability of very deep models. Such models faced convergence issues due to vanishing gradient, later resolved using residual connections. Variants of these are now widely used in modern ar- chitectures. However, the computational cost of backpropagation remains major burden, accounting for most of the training time. Taking advantage of residual-like architectural designs, we introduce Highway backpropagation, parallelizable it- crative algorithm that approximates backpropagation. by alternatively i) accumu- lating the gradient estimates along the residual path, and ii) backpropagating them through every layer in parallel. This algorithm naturally derived from a decom- position of the gradient the sum of gradients flowing through all paths, and is adaptable to a diverse set of common architectures, ranging from ResNets and Transformers recurrent neural networks. Through an extensive empirical study on a large selection of tasks and models, we evaluate Highway-Ba -ar and show that major speedups can be achieved with minimal performance degradation.
INTRODUCTION
Often copied but never matched, the backpropagation algorithm (Rumelhart et al. 1986) is still at the heart of deep-learning optimization, coupled with the gradient descent. However, as model sizes continue to grow, its memory overhead and computational time become more and more prohibitive.
This was especially the case for recurrent neural networks (RNNs) (Elman) 1990: Hochreiter & Schmidhuber 1997: Cho et al. 2014). Considered as state of the art for sequence processing (e.g. natural and spoken language), the time required to run the backpropagation through time for stacked RNNs (Sutskever et al., 2014) has motivated the design of transformers (Vaswani et al. (017)) that process the sequence in parallel. However, with the advent of deeper and larger models in NLP Ka-
plan et al. 2020 Hoffmann al. (022) and computer vision (Dosovitskiy) 2020 Dehghani al.,
2023). the problem persists: the sequential aspect of backpropagation implies computational cost that clearly limits further advancements model design and scalability.
Frugal alternatives to backpropagation, such as forward-only methods (Hinton 2022 Nøkland 2016) and exact parallel backpropagation (Lim et al.
al. 2024 Danieli al. 2023), have shown promis- ing results, but often involve impractical trade-offs between speed and task performance. More- over, these methods often do not to leverage the recent advances that made the success of modern deep-learning models, like Batch and normalizaionn (loffe & Szegedy 2015 Ba et al.] 2016). Another important example the widespread use of residual connections, which enables efficient gradient propagation across layers, prevents vanishing gradients. and significantly improves training convergence very deep models (Srivastava et al. 20155 He et al. 2016). Most contemporary deep models incorporate residual paths that connect the loss to intermediate layers.
In this work, we focus on deep sequential models, i.e. models that rely on large and sequential computational graph like RNNs, ResNets, and Transformers. We introduce Highway backpropaga- tion (Highway-Bii -B)), an iterative algorithm to transmit the error signal backward through the network. Derived from original approach, Highway-BH leverages residual paths to instantly backpropagate

---

Published as a conference paper at ICLR 2025
gradient estimates to earlier layers. By varying the number of iterations, our method allows to readily trade the level of approximation of the gradient for speed-up and precision, which lets the user choose a dedicated optimization strategy in the context of a limited computational budget.
Our main contributions are the following:
We introduce Highway-BP, parallelizable iterative algorithm that approximates backprop- agation for accelerating the training of deep sequential models.
By leveraging architecture-awant components such as residual connections, Highway-BP is highly efficient and can be directly adapted to many different architectures.
The algorithm is motivated by an intuitive decomposition of the gradient. In particular, the speed versus accuracy trade-off of the algorithm be controlled by stopping after k iterations, resulting in an interpretable approximation.
We evaluate Highway-BPP on a large range of models and tasks, including ResNets, Trans- formers, and RNNs, and empirically show that it converges to the exact gradient in only handful of iterations.
RELATED WORK
2.1 RESIDUAL CONNECTIONS
Gradient descent is the fundamental method for training deep learning models, but very deep net- works encounter significant challenges, including vanishing and exploding gradients (Bengio et al.) Pascanu et al. 2013 Zucchet & Orvieto 2024). To address these problems, network architec- tures have been modified with connections that bypass intermediate layers. Residual connections, first introduced in ResNets (He et al. 2015) and later adopted in transformers (Vaswani al. [017
1994
Radford & Narasimhan 2018), are one such solution. Similar concepts are found in Highway net-
works (Srivastava al. 2015 and the gated mechanisms of LSTMs (Hochreiter & Schmidhuber
1997) and GRUs (Chung et al. 2014).
In addition mitigating vanishing gradients, residual connections (Srivastava 2015 He al. 2016) help address the shattering gradient effect, where gradients in deep networks become noisy and uncorrelated, leading poor signal-to-noise ratios during backpropagation (Balduzzi et al. 2017]. By introducing shortcut paths that allow gradients to flow more effectively, residual con- nections preserve meaningful signals across layers and simplify learning by enabling networks to approximate identity mappings when needed. This facilitates the optimization of deep models and allows networks to scale in depth without suffering from performance degradation.
Veit et al. (2016) in particular observe that residual models actually behave like an ensemble of shallow models. They show that the gradient that goes through many residual connections has the most impact in the training. This is precisely the motivation behind our work, where we provide an algorithm to compute these gradients, faster than backpropagating through the entire model.
2.2 PARALLELIZING BACKPROPAGATION
Exact parallel backpropagation Backpropagation be computed exactly in parallel, with com- plexity in Olog, L), where L is the number of layers. This is done using prefix scan algorithms (Hillis & Steele 1986; Blelloch| 1990). However, while this seems attractive, there serious limi- tations in practice since the algorithm involves i) computing the Jacobian matrices of all layers, and ii) many matrix matrix multiplications, both of which are extremely time and weemory-consuingg multiplications lead to the algorithm's 's true time complexity being in
In particular, matrix- O log, T.) and memory in O(Bd2 L.). where is the batch size and the hidden dimension of the model. The cubic complexity with respect to the dimension completely prevents the use of this algorithm for large models. Still, DeepPCR and DEER (Danieli et al.) 2023 Lim et al. 2024) obtained significant speedups for small-sized models. Gonzalez et al. (2024) also proposed ELK as a more stable and scalable improvement DEER, in particular by approximating the Jacobians with diagonal matrices, which reduces the time and memory complexities to match that of backpropaga- tion. Our method uses the same prefix scan algorithm but leverages the structure of the Jacobians to keep a low complexity.
2

---

Published as a conference paper at ICLR 2025
fz
gt
ACg
8hg
hz-1
{
(m)
(h-(h))))
Gia({33)
(4) layer (forward)
BCy
()-1)
Gw((2))
xJ
(ar((1)
BC
BC
(k 2)
Gra(O)
6 K,KK
GG((33)
6 J,KgK,
8h;-1
8h,
Gs(((2)
5. K,J,K,
Ga((1))
8. KgKgJ1
Ohi-1
(b) Residhall layer (backward)
(c) decommpocition example
Figure 1: (a) Illustration layer fi decomposed as the composition of a block 9i and residual function rt. (b) Backpropagation through the residual layer. Gray terms correspond intermediate losses that are only present in RNNs. (c) Illustration of the gradient decomposition (Theorem D as 0 in blue), through only one block (k- = 1, in yellow, orange and red), and so on for k = 2, 3, until k = L.
the sum of the gradients flowing through only the residual connections
Backpropagation system of equations Backpropagating through sequential model can be
seen solving system of equations of the form 8C 8C ah++1 Vi. This leads multiple Th.
Th Sh,
works applying solvers to accelerate training. Günther al.] (2020) and Moon & Cyr (2022) use ODE interpretations of ResNets and GRUs respectively to use the parallel MGRIT solver (Falgout) ctt al. 2014). Similarly, [Wang & Ragni (202)) reformulate backpropagation in RNNs as finding fixed point, which they compute iteratively. The method works well because each hidden state has its own local loss, which has more importance than losses further down the sequence. This however does not scale well sequence classification and sequential models, which is why Trinh et al. (2018) introduce local auxiliary losses along the sequence, akin to pre-training in language modeling, keep good performance when truncating the gradients.
Accelerating the forward pass Some of the aforementioned methods are also applied to approx- imate the forward pass: Lim al. (2024); Danieli et al. (2023); Gonzalez al. (2024); Wann & Ragni (202I): Günther et al. (2020). While this is out of the scope of this paper, our method is orthogonal only approximates backpropagation, and any method could be used concurrently to accelerate the forward pass.
Our approach differs from the system of equations and ODE interpretations, as we introduce Highway-BP through an intuitive decomposition of the gradient. Each element of the decompo- sition progressively recovered at each iteration, giving clear interpretation of the estimate after any & iterations. Drawing inspiration from Danieli et al. (2023); Lim al. (2024), we generalize their use of scan algorithm and leverage architecture-awars components to make the computation much more efficient and scalable.
3 NOTATIONS AND ASSUMPTIONS
We consider sequential model composed of L layers f1, . fL, each parameterized by 6,. We denote h; (hb as the hidden state after layer i, with x ho being the input of the model. We also define a loss function L to be minimized.
While most of the time the loss only function of the last hidden state h1, we build our framework using a more general formulation, with a loss of the form: L(hi....,hy = Σl= C.(h).. Allowing the loss to depend each intermediate state allows the framework to handle more models and tasks (e.g. RNNs and transformers).
Main assumption. tions g; and n:!
We suppose that the layers f. can be expressed the composition of two func-
f((x) ri(xx, gx(x))
3

---

Published as a conference paper at ICLR 2025
leading to the following Jacobian:
0f: J; K;
Ōr
where:
Ji (r.(a,)) 0g.(z) is computationally expensive to compute and can only be multiplied
3z Oy
vector (e.g. Jacobian of convolution),
K;
ar((z,) 8x
is computationally cheap to compute and multiply (e.g. diagonal matrix).
This general formulation allows us to consider wide range of architectures and models. For in- stance, in a simple residual model, we would choose 9: to be the residual block (e.g. convolutions), and r,( 2)
would be the residual connection. Table I provides more examples including ResNets, Transformers (either pre- or post-layernorm), as well as recurrent neural networks like GRU and LSTM. Figure also illustrates the decomposition of into gi and r;.
Note that there is no requirement for the residual function to be linear. For instance, ResNets use a ReLU activation after the residual connection. This opens up our method a wider class of models that may have more elaborate residual-like connections. Throughout the paper, we will designate any such architectural design as a residual connection or residual path.
Examples of decomposition of the layers fi of common models as the composition of an expensive block g (x) and cheap residual connection ri (2 z), in Equation
Table 1:
Model
Sz (x)
(:(x)
r: (2z
Pre-activation ResNet
+ Block(x)
Block(x)
+ 2
ResNet
ReLU (lock(x))
Block(z)
ReLU +
Transformer (Pre-LN)
x + (aayer(LN(ar))))
Layer(LN(a))
x + 2
Transformer (Post-LN)
LN(x + Layer( (z))
[a(z), b(x))
21
x + Z2
GRU
a(x) O x b(z)
la(x), b(x)
21
22
LSTM
[a(x)) O b(x)) c(x))
[a(x). b(x)).(()
[2] + 22,
HIGHWAY BACKPROPAGATION
Our method is based gradient decomposition into terms corresponding to different paths (The- orem []. Based on recursive relation, we introduce an iterative algorithm, which progressively includes gradients from longer paths (Theorem Finally, we describe how an iteration can be parallelized. We provide proofs of the theorems in Appendix
4.1 GRADIENT DECOMPOSITION
At each layer fi one part of the gradient is backpropagated through the residual block (using J:) and the other through the residual connection (using K,). This leads to 2 possible paths. The following theorem states that the gradient 2h.
Ø6 is the sum of the gradients backpropagated through each path.
The different paths are depicted in Figure
Theorem (Decomposition of the gradient over all paths). Given layer indices i S j. and set of indices J C [i + 1, we define Giz (J) as the gradient backpropagated from C,(h)) to ha going through either the Jacobian Jk of the residual blocks (when k E J and otherwise through the residual connections with Kg (see Figure for visual example). It can be expressed as:
Giy(T) ac,
j-i-1
(
JH-
ifj kj-kEJ] otherwise
II kmo
(Ky
(3)
Oh,
Then, for any hidden state h;, its gradient oh,
ac is the sum over all paths starting at index i:
Σ
Giy(T).
(4)
0h+
iSiSL
JSH+1
4

---

Published as a conference paper at ICLR 2025
4.2 ITERATIVE ALGORITHM
Using the previous decomposition of the gradient, we can design an iterative process to compute it, described in the following theorem:
Theorem (Iterative computation of the gradient). Let us define w the sum of the gradients of all paths starting from i going through at most Jucobians Jj. which we obtain by truncating the sum in Equation
Σ Gty(T)
(5)
iSiSL
JS JCH+ 171S
Then, w can be computed iterafively using the recursive relation:
Σ
1. 0C1
KjKjKKJi- ... Ki+1
(6)
Bhy
j=1
w
k+1
uf +
Σ w/y 1 Ki41
s=i+1
In particular, k 2 L we get the exact gradient w
BC
Oh
Following this theorem, the algorithm requires at most k. L iterations. It is then straightforward to finalize backpropagation and get the gradient with respect to the parameters:
8c
ac 0f
0h, 08.
4.3 PARALLEL COMPUTATION
Breaking down Equation 7
we can see how an iteration can be computed in two steps:
A parallel backpropagation through the expensive Jacobians Jj:
x+1 -
J++1
Vi [0,
(8)
2. A sequential backpropagation through the residual path, which also parallelizable eff- ciently given our assumptions about K;: w.
#+1
w + k+1 where:
k+l K2K K1t1 rtt
k+1 u K++1
(9)
While it is clear that step is parallelizable this is less obvious in step 2. This is however possible using prefix scan algorithms (Blelloch 1990; Bochm al. 2019). A prefix scan aggregates series of values (e.g. vectors) using an associative operator (e.g. sum), which is general formulation that has many applications, including solving linear recurrences like the use Hillis and Steele's parallel algorithm (Hillis & Steele 1986) in our experiments, and we indicate pseudocode of this algorithm adapted our needs in Appendix B (Algorithm []] We denote this algorithm as CumSumProd as in Boehm et al. (2019), and use it to rewrite Equations and 7
have in Equation We
((32))
(BCs)) L i=1 (Ki)
w
CumSummrro
(10)
k+1 w
w! + CumSumProd (wotz Jim i=1 (K))
L-1
(11)
with:
CumSumProd(a, M); = Σ a M. ...M
(12)
523
On single process, the parallel version of CumSumProd has O(L L) time complexity, how- ever as the inner loop is parallelizable the effective computation time grows in Olog L). In addition, it can be implemented using in in-place operations for optimal memory efficiency.
An important note is that the parallel CumSumProd algorithm relies on all the K, being computed ahead, and involves matrix-matrix multiplications between the K;. This is not issue in our case
5

---

Published as a conference paper at ICLR 2025
Table 2: Summary the models and datasets used in our experiments. L is the number of skip- connections for sequential models, and the sequence length for RNNs. Only some RNNs have intermediate losses (i.e. one loss per cell)
Dataset
Model
Layers
Params
L
Intermediate losses
CIFARIO
Pre-act ResNet32
32
464k
15
X
CIFARIO
ResNet110
110
1.7M
54
X
ImageNet32
ResNet56
56
917k
27
X
Wikitext103
GPT-2
12
14.5M
24
X
MNLI
RoBERTa
12
124M
24
X
Wikitext103 char
LSTM
2.3M
256
Wikitext103 char
GRU
1.8M
256
Wikitext103
GRU
21.1M
256
CIFARIO pixel level
GRU
29.8k
1024
X
as we decompose the layer fi into gi and r; precisely such that K; behaves nicely (e.g. scalar, identity, diagonal, low-rank). Lim et al. (2024) also use CumSumProd but replace K; with the Jacobian of the whole layer, which is extremely inefficient in time and memory for large models. Also note that Highway-BP could still be applied situations where K; prevents the of the parallel CumSumProd algorithm, as it is always possible to solve sequentially the recursive relation Equation 9 which only involves pector-Jacobiag products.
4.4 APPROXIMATING BACKPROPAGATION
While the iterative process from Theorem 2 converges to the exact gradient after L iterations, we propose to stop after a small number & of iterations and use the current estimate w instead of the exact gradient -h to update the model's parameters.
The number h of Highway- BP iterations becomes hyperparameter, and allows users to freely control the tradeoff between the speed and accuracy of the algorithm. Moreover, at any iteration k the current estimate is interpretable by design: w is the sum of all gradients flowing through at most residual blocks.
It is reasonable to expect that gradients going through fewer blocks are statistically more useful for learning. The reason why residual connections are so effective at improving training is that gradients can flow directly from the loss to any intermediate layer. All layers can learn at the same time, which greatly improves convergence. This suggests that the most important part of the gradient comes from the residual connection (or at least at the beginning of the training). Veit et al. (2016) have shown that this the case for ResNet models. We also empirically confirm this throughout all of our experiments, described in section
This leads to Highway-BP only requiring & iterations, each having O(log I.) substeps for the Cum- SumProd operation, thus reducing the computation time T from TBP TTorward + Thackward to:
Highhay-BBP Trorward
Toackward + O logz L)
(13)
5 EXPERIMENTS
The Highway-Ba framework and notations have been designed to be highly flexible, and in particu- lar to handle both deep sequential models and recurrent neural networks. We evaluate Highway-Ba on such models for several tasks, which we summarize in Table
5.1 DEEP SEQUENTIAL MODELS
We evaluate Highway-BP image classification with three ResNet models, well language modeling tasks by pre-training and fine-tuning two transformer models. The models greatly vary in size and depth, ranging from 464k 124M parameters.

---

Published as a conference paper at ICLR 2025
100
100
100
95
95
30
Accuracy
Accuracy
Perplexity
40
85
45
30
80
20
20
Rackpmgagsttion
10
10-
60
Highway
3 9.30
(a) Pre-act ResNer32 CIFARIO
(b) ResNetll 10 CIFARIO
(c) ResNet56
(d) GPT-22
(c) RoBERTa MNLI
ImageNet32
Wikitext103
Final performance of deep sequential models versus the number i of Highway- BBraay-BBBP itera- tions used for training (red), compared to backpropagation (black).
Figure 2:
5.1.1 EXPERIMENTAL SETUP
The first ResNet model is ResNet32 with pre-activations as introduced by He et al. (2016). is easy to handle since we have r2( z) + and K; = J. We also use deeper ResNets with the original architecture (He et al. 2015), which apply ReL.U activation after the residual connection: T.( ReLU(x 2). We train these models CIFARIO (Krizhevsky) 2009) and ImageNet32 CChabaszzz et al. 2017) for image classification. We only modified the downsampling layers as described in Appendix Go simplify the use of Highway-BP.
We also pre-train a small transformer model Vasswani et al. 2017) for language modeling on the Wikitext103 dataset. The model is based the GPT-2 architecture (Radford et al. 2019) with 12 layers but a smaller hidden dimension. Finally, we fine-tune pre-trained RoBERTa model (Liu al. 2019) the MNLI dataset (Williams al., (208)), which involves predicting the entailment information of a pair of sentences and is part of the GLUE benchmark (Wang et al., (018). Note that for both transformers, we split the layers into two sublayers self-attention and feedforward which means we have L = 24 for 12 layers.
any of these models, we define gi and ri described in Ta- ble T However, for both transformer models, we used slightly different choices as described in Appendix C3 This is done after observing that transformer layers tend learn to cancel part of their residual connection.
When applying Highway-BP
5.1.2 RESULTS
We report the results in Figure where we compare models trained either with backpropagation or with Highway-Bi -B using different numbers of iterations. As expected, more iterations increase the performance of the models. However, the quality of training good even with very small values of h compared to L.. This is especially impressive for the ResNetl 10 model, which requires only iterations to match backpropagation, while = 54. This confirms our intuition that most of the gradient in deep residual models goes through the residual layers.
Surprisingly, performing & iterations already leads to very reasonable performances (e.g. 85% on CIFAR100 10). By definition of Highway- BBP using Equation 4 the estimate after l: 0 corresponds to only backpropagating the gradient from the classification head through the residual path. Each layer then receives the gradient ac at its output, and uses this to update its weights. This is very similar boosting (Freund & Schapire 1999). where many small models summed together, and each one of them learns to compensate for the errors of the previous models.
The training curves of the GPT-2 model shown in Figure for different values of k. Even when is too low and deteriorates the model's performance, it still makes the model learn smoothly at the same speed, only converging higher loss.

---

Published as a conference paper at ICLR 2025
Backpoouagtion Highway-BP 0)
5.25
5.00
Highway-BP (R= 1)
Highway BP (R
4.75
Highway 3)
Validation
4.50
Highway-BP (R
4.25
4.00
3.75
3.50
2500
5000
7500
10000
12500
15000
17500
20000
Training steps
Figure 3: Validation loss during training for the GPT-2 model, with different algorithms.
5.2 RECURRENT NEURAL NETWORKS
Recurrent neural networks (RNNs) also fit our framework: instead of considering a sequence of layers along the depth dimension, we consider a repetition of the same cell along the time dimension. More formally, for an input sequence (I:) we can see each cell as layer fi parameterized by 0. (0, .r (the parameter 9 common all cells, and the external input æ;)). The input state ho the initial state of the RNN.
Some RNNs such as LSTM (Hochreiter & Schmidhuber, 1997) and GRU (Chung et al., 2014) possess long-term memory that is updated by each cell using linear gating. This memory allows model to keep information for long distances and helps with gradient issues (Zucchet & Orvieto 2024]. Akin to residual connections in deep models. design with Highway-BBl We show in Table how LSTM and GRU cells can be represented using the di and T; functions.
take advantage of this architecture
5.2.1 EXPERIMENTAL SETUP
As baselines to compare the performance of Highway-BP on RNNs, we use i) backpropagation, and ii) fixed-point iteration (FPI), which is the method used by Wang & Ragni (021 to approximate the backward pass of RNNs, and simply consists of repeating k backpropagations through all layers Note that this
in parallel. FPI is special case of Highway-BP with 9: algorithm can only perform well if there are intermediate losses at each time step, otherwise, this is equivalent backpropagating only through the last ls cells, which be seen as a form of extreme machine learning (Huang et al. 2006).
and rı 2)
We train one layer LSTM and GRU on a language modeling task the character level Wiki- text103, as well as 3 GRU layers stacked trained on Wikitext at the word level (same task as the GPT-2 transformer in the previous section). Finally, we use a task from Long Range Arena (Tay et al. 2021): image classification on CIFAR10 using the flattened image, i.e. sequence of 1024 3-dimensional pixel vectors.
5.2.2 RESULTS
Similarly to sequential models, we show in Figure performances of models trained with dif- ferent algorithms, and for different numbers of iterations Highway-BP constantly outperforms the fixed-point iteration algorithm in terms of convergence speed, while the algorithms are practi- cally identical in terms computations performed. As mentioned the previous section, FPI is a special case of our method when we do not consider the residual connection at all (g,(x) = f(())
and r; (z, 2) = 2). Highway-Bi -P uses additional knowledge about the architecture to improve the convergence speed over naive, model-agnostic approaches.
8

---

Published as a conference paper at ICLR 2025
Perplesity
3.7
Parplcity
Perplcaity
60
Ascuric
30
30
Bickgeoppptiin
3.5
10-
Foundlun fxmatim
Highoog--i
10
(a) LSTM x1 Wikitext103 char-level
(b) GRU x1 Wikitext103 char-level
(c) GRU x3
(d) GRU x1 CIFAR10 pixel-level
Wikitext103
Figure 4: Final performance of RNNs versus the number & of Highway-BPP iterations used for
training (red), compared to backpropagation (black) and fixed-point iteration (blue).
Backpropugation
Highway BP (t 0)
Highway BP
Table 3: Speedup of training with Highway- BP vs. backpropagation for the RNN experi- ments (more details in Table 6)
5.00 -
Highway-BB BP =
Validation loss
Highway
4.75 -
4.50
Model
L
k
k
5
t: 10
LSTM x1
256
x3.0
x1.7
x1.2
GRU x1
256
x3.2
x1.8
x1.3
3.75
GRU
256
x2.9
x1.8
x1.3
0h
1h
Zhh
3h
GRU x1
1024
x3.5
x3.1
x2.9
Runtime (hours)
Figure 5: Loss vs. runtime of the 3-layer
GRU RNN, for different algorithms.
The pixel-level CIFARIO an especially hard task, as the model needs to learn features from all parts of very long sequence (1024 pixels). Moreover, the gradient is sparse the prediction made using the last hidden state h1024, which means the gradient needs backpropagated through long distances. While FPI fails to do so as expected, Highway-Bi reaches the same accuracy iterations.
as backpropagation with
We additionally report the speedup obtained with Highway-BP in Table 3 the optimal number of iterations, all model trainings get a speedup between x2 and x3. Moreover, the gains get more significant for longer sequences.
We observe that using
Finally, in Figure 5 we show the training curves the largest RNN model, the 3-layer GRU, using the real training time for the x-axis. It can be seen how & controls the tradeoff between training speed and model performance.
5.3 TRAINING DYNAMICS ANALYSIS
In this section, we investigate how the convergence of Highway-BP evolves during training. Intu- itively, at initialization, all the layers start learning using the residual connection. As the layers start using relevant features from earlier layers, they start working together and we expect the contribution of high values of l to increase over training.
In Figure 6, we analyze how Highway-BP behaves throughout training. The top reports the cosine similarity between the estimated gradient and the true gradient, which seems to require more iterations the end of training to reach 1. The bottom row also validates this claim, it shows how the contribution of each iteration slowly shifts toward higher values of k. The transformer seems to be the most consistent model, the cosine similarity stays mostly constant during training.
The special case = 0, which corresponds to only backpropagating through the residual connection, slowly decreases in accuracy time for all models. Still, its contribution to the total norm remains

---

Published as a conference paper at ICLR 2025
Training progress
Cosine similarity
1.0
100%
0.9
0.8
0.6
0.7
Relative step nor
50%
0.4
0.2
0.2
25%
0.2
0.0
0.0
0.0
0%
6
10 0
10
k
(a) GRU
(b) Pre-act ResNet32 CIFARIO
(c) GPT-2 Wikitext103
Wikitext103 char-level
Figure 6: Convergence analysis of Highway-BB for different number & of iterations and different times during training (from 0% in blue, to 100% in red). (Top) Cosine similarity between the true 0C
gradient 20 versus the approximation using Highway-BP iterations. (Bottom) Norm of the k-th Highway-BP iteration step (relative to the total norm).
high during all the training, which suggests that residual connections play a key part in training deep models, and not only the beginning of the training.
6 DISCUSSION AND CONCLUSION
We introduce Highway-BI as an architecture-aware algorithm for approximating backpropagation in deep sequential models. Its parallelizability and effectiveness unlock new possibilities, for in- stance, in training RNNs over long sequences or large models in layer-distributed setting. We show through decomposition of the gradient that each algorithm iteration adds another component to the estimate, until completely reconstructs the gradient. As such, each intermediate estimate interpretable as the sum of the gradients associated with paths going through at most A residual blocks.
Empirical findings show that our method can replicate backpropagation with much lower time com- plexity, as it often converges in a few iterations. We observe this for models, with promising results deep models and RNNs long sequences. As recently shown by [Bck al. (2024), when scaling LSTMs to billions of parameters, these models perform favorably in terms of perfor- mance compared mate-of-thhearr Transformers, showcasing superior expressivity. Our framework may thus be applied to allow fast training of very large RNNs, with billions of parameters.
Our general framework allows us use the same generic code train all models using Highway-Ba However, simplicity comes at the cost of less optimization, and we believe that archiiecture-specific implementations must be done to benefit the most from Highway-BP. in this paper is to demonstrate the high training quality of Highway-Bi almost matching backprop- agation with few iterations. We leave practical implementation for training large models in distributed setting for future work. We however show that RNNs can be sped up considerably single GPU, as all cells share the same weights. Still, we believe the prefix sean algorithm could be much more optimized, using custom CUDA kernel for instance.
addition, our main purpose
Finally, the tradeoff between training speed and quality can be adjusted at any time. While low numbers of iterations are enough the beginning of training, it is possible to increase the number of iterations during training and end with an exact backpropagation. This versatility not only allows to perform increasingly more accurate optimization steps speed up training while attaining the same performance, but also allows the user to choose a dedicated optimization strategy in the context of limited computational budget.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was performed using HPC resources from GENCI-IDRIS (grants AD011015154 and A0151014627), and received funding from the French National Research Agency (ANR SPEED- 20-CE23-0025) and the French Government via the program France 2030 ANR-23-PEIA-0000. SHARP.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL
https://arxis: org/ halg//as//tt aata///s////0/ 06450
David Balduzzi, Marcus Frean, Lennox Leary, Lewis, Kurt Wan-Duo Ma, and Brian Mc McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In Proceed- ings of the 34th International Conference on Machine Learning (ICML), pp. 342-350 350. PMLR, 2017.
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. arXiv preprint arXiv:2405.04511 2024.
Y. Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent difficult. IEEE transactions on neural networks / publication of the IEEE Neural Networks Council, 5:157-66, 02 1994. doi: 10.1109/72.299181.
Guy E. Blelloch. Prefix sums and their applications. Technical Report CMU-CS-90-190, School of Computer Science, Carnegie Mellon University, November 1990.
Matthias Boehm, Alexandre Evfimievski, and Berthold Reinwald. Efficient data-parallel cumu- lative aggregates for large-scale machine learning. In BTW, pp. 267-286, 2019.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using mnn mecoder-decodee for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pp. 1724-1734, Doha, Qatar, October 2014. As- sociation for Computational Linguistics. URL http: www. , www..w.......... www./o/gttto.oooooooooo ogy/
D14-1179
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative the cifar datasets. ArXiv, abs/1707.0881/ 2017. URL Inttps://ap)
ottp:pooaa..oo///ooorraaaa:a:::. 73045442
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. 12 2014.
Federico Danieli, Miguel Sarabia, Xavier Suau Cuadros, Pau Rodriguez. and Luca Zap- pella. Deepper: Parallelizing sequential operations in neural networks. In A. Oh, T. Nau- mann, Globerson, K. Saenko, M. Hardt, and Levine (eds.), Advances Neural Curran Associates, Inc., Information Processing Systems, volume 36, pp- stcp/pppper_ille//ppp////33 Conpe .pdf
2023. URL
Eil L/ htle:9/cccbalsseocca000136663333
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pp. 7480-7512. PMLR, 2023.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:20/0.1 /929, 2020.
Jeffrey L. Elman. Finding structure in time. Cognitive Science. 1412):179-211, 1990. doi: 10.1016/ 0364-0211399))900022E URL httpp::/gropp................... Paper/elmar990 ructure. html
edu/ aattt//maa/aaggg///./// langev/

---

Published as a conference paper at ICLR 2025
R. D. Falgout, S. Friedhoff, Tz. V. Kolev, S. P. MacLachlan, and J. B. Schroder. Parallel time integration with multigrid. SIAM Journal on Scientific Computing, 36(6):C635 C661, 2014. doi: 10.1137/130942330.
Yoav Freund and Robert E. Schapire. A short introduction to boosting. 1999.
URL https:
//api. seapp.sssmmiiischolaaaa. org. I 9621074
Xavier Gonzalez, Andrew Warrington, Jimmy T.H. Smith, and Scott Linderman. Towards scalable and stable parallelization of nonlinear rmns. ArXiv, abs/2407. 19110.9..11,, 2024. URL
Stefanie Günther, Lars Ruthotto, Jacob B. Schroder, Eric C. Cyr, and Nicolas R. Gauger. Layer- parallel training of deep residual neural networks. SIAM Journal on Mathematics of Data Sci- ence, 2(1):1 2020. doi: 10.1137/9M1147720, URL. https: //oci org/ 11377
19M1 (9M1247620)
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 770-778, 2015. URL https: scccsssccssccccccs........ ar www.or///ooooooo. Corpg/ ID: 206594692
Semant cccholschlllccoooocc
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 630-646 645, Cham, 2016. Springer International Publishing. ISBN 978-3-319. 46493-0.
William D. Hillis and Guy Steele. Data parallel algorithms. Commun. ACM, 29:1170-1183, 1986. URL Bttttt::s///pp semant wwt.intisschhho leschollar
wwt.:o/ro/ooooio/..... http://aiccaaaaaa.aooooorroooooe 23159666
Geoffrey E Hinton. The forward-forward algorithm: Some preliminary investigations. arXiv, 2022.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8):
1780, November 1997. ISSN 0899-7667.
doi: 10.1162/neco.1997 199.9....99.............. URL
www.:...11nnnoo..m 1997.999999 1997.99.8.1735
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoe, Au- relia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Lau- rent Sifre. An empirical analysis of compute-optimal large language model training. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 30016-30030. Curran Associates, Inc., 2022. URL https:
File/ httpps::/ppoccce.inngs/o//oo//es http:///ooeeerco.ooooe.eeee.esaa wtcpappss.ccpppeeee wttp://errreeeeeeeee/eeeea pd
Guang-Bin Huang, Qun-Yu Zhu, and Chee-Kheong Siew. Extreme learning machine: Theory and ISSN 0925-2312. doi: https://doi.or//
applications. Neurocomputing, 70(1):489 501, 2006.
11.006/j.neccom2200......1 URL https: :ttpspp::/www//
atticlless1/80223310000333
Neural Networks.
Sergey loffe and Christian Szegedy. Batch normalization:
Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456.
pmlr, 2015.
Jörn-Henric Jacobsen, Arnold W.M. Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. In International Conference on Learning Representations, 2018. URL https::/ openreview net ftrttt/foummmam ?i J.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXhv:2001.0828 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.69800, 2014. URL https: /api. semantisscholar.cooooooo wttp:/chhoo.aa.ooo//ooossssslola 6628106
12

---

Published as a conference paper at ICLR 2025
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. URL https: wew.eanniicsccooo...oooooooooooa
Yi Heng Lim, Qi Zhu, Joshua Selfridge, and Muhammad Firmansyah Kasim. Parallelizing non-linear sequential models over the sequence length. In The Twelfth International Confer- Learning Representations, 2024. URL https:: http:::///penreview. net forum? B34AIVLNOV
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zetrlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907. 1692, 2019. URL https::/app semanticscholass org/ ScrpusID:19895332
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2017. https::/app. http:s///pp...emmmt.cccccch..... h0t.
CorpusID: 53:9222770
Gordon Euhyun Moon and Eric C. Cyr. Parallel training of gru networks with a multi-grid solver for long sequences, 2022. URL https::/aaxiv...rrt wwt.vvoorr/a///oo/ abs 220//2203..0 04738
Arild Nokland. Direct feedback alignment provides learning in deep neural networks. In Advances in neural information processing systems, 2016.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on International Conference on Machine Learning Volume 28, ICML' pp. I11-1310-111-1311 JMLR.org, 2013.
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre- training. 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. rearesinttree by back- propagating errors. Nature, 323:533-5388 1986. URL org/Corpus 2000000 205001833
Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. Training very deep net- works. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https:
neuriinneueii fi F1102/10712278665622222221111111
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems
(NIPS) 27, pp. 3104-3112. 2014. URL
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena A benchmark for efficient transformers. In International Conference on Learning Representations, 2021. URL https: //openzeview.nee wttp://eee/oooomid///oeeeeom grCCk
Trieu H. Trinh, Andrew M. Dai, Thang Luong, and Quoc V. Le. Learning longer-term depen- dencies in mns with auxiliary losses. ArXiv, abs/1803.000444, 2018. URL https://api,
semanticscholas wwwp:oopo//oooo.oooooooom 477766633
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All you Need. In Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://pproeecccn....eeurrr wtw.errris..ccppppaeeeeeeeeeeeee https//eeeeeeeeae1111777/1 Saael//.aaaa22333eeeeeeeeeeaaad2
13

---

Published as a conference paper at ICLR 2025
Andreas Veit, Michael Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 550-558, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 978151083881
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupata, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP,, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. 10.11863/11/1185440 URL.
httpps//aclaamthoooyy.oooo
5446
Zhengxiong Wang and Anton Ragni. Approximate seedtpoints in recurrent neural networks, 2021.
httth:ts:://eet www.///axvvvvvom. org/ abs 2106. 02417
Adina Williams, Nikita Nangia, and Samuel Bowman. broad-coverage challenge corpus for sentence understanding through inference. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.). Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp- 1112-1122 11222, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-111
TRR.https:// httttps://allanthooogoeoo 0rgg111111.//11111111111
Nicolas Zucchet and Antonio Orvieto. Recurrent neural networks: vanishing and exploding gra- dients are not the end of the story. ArXiv, abs/2405.21064 2024. URL /httts:://// //apps
semanticscho ar. wttp:o/orrrssai:2011190090
A PROOFS
A.1 PROOF OF THROREM
The result is obtained by completely expanding the product of Jacobians obtained with the chain rule:
OC Oh.
0C, Oh,
Σ
Thy Ōh.
1 OCj
Σ Ohi
(J, + (,,)))j++++KK- (J + Ki+1)
0C,
j-i-1
Σ
Σ
II
if - E else Kj-
Oh,
j=i sci++
k=0
Σ
Gir(T)
KUSL JC[+1.1
A.2 PROOF OF THEOREM 2
From the definition of wh:
w 1+1
Σ Giy(T)
iSiSt
JCK+1
1. Sk+1
Σ
GRj 0
+
Σ
G (mnin())))mnnn Kmin( K4t2K111
A=
JCHt
14

---

Published as a conference paper at ICLR 2025
Σ
Σ
Gmi(J) I Jm ... Kit2Ki41
w
m=i+1
MSiSL J[m++
1315K
+
Σ
K+t2Kit1
j=++1
In particular, from Theorem
we have that w
for k 2 L i. OC Th+
B PSEUDOCODE OF THE PARALLEL PREFIX SCAN ALGORITHM FOR CUMSUMPROD
Algorithm (Parallel CumSumProd) Parallel cumulative sum-product algorithm (reversed)
Inputs:
sequence of L vectors (u;) with v; Rdd sequence of L matrices (K,) with K₁ Rd Output: (u.)/=1 such that u, = Σ)= PyKg... K441
(0) t v p(0) + K
M + [log, L] for m + 0 to .M do
for i + 1 to L in parallel do
if
2 then
" (m+1) < u; (m)
(m)
((m)
+ 0 W
p. p(m+1) P+2m
pmm)
else
u (m+1)
u; (m)
plm+1) x- (mm)
end if
end for
end for
return (M)
C EXPERIMENTAL SETUP DETAILS
C.I TRAINING SCHEME
Most models are trained using the Adam optimizer (Kingma & Ba 2014). case weight decay, use the AdamW variation (Loshchilov & Hutter] (017)) We also use cosine learning rate scheduler to decrease the learning rate to a tenth of its initial value. Additionally, the first 10% of the training is performed with linear warmup. For both ResNet experiments, however, we use training scheme close to the original paper (He et al.) 2015), which uses an SGD optimizer with momentum and divides the learning rate by twice during training. All experiments were conducted on single GPUs, either Nvidia A 100, A40, or RTX A6000.
C.2 DATASETS
CIFAR10 The CIFARIO (Krizhessky 2009) dataset contains 50k images with 10 classes. Each image is 32 by 32 with 3 channels for the RGB values. We normalize the images to have zero mean and unit variance. Additionally, for the ResNet models, we apply the same lata-augmentation tech- niques as in the original ResNet paper (He et al., 2015): horizontal flipping and random cropping.
15

---

Published as a conference paper at ICLR 2025
CIFARIO pixel-level In Long Range Arena (Tay et al. 2021). CIFARIO images flattened as sequences of 3-dimensional vectors. Sequence models such as RNNs can then be applied to image classification.
ImageNet32 The ImageNet32 (Chrabaszez et al. 2017) dataset contains 1.3M images with 1000 classes. Each image is 32 by 32 with channels for the RGB values. We process this the same way as CIFARIO.
Wikitext103 Wikitext103 is dataset containing texts extracted from Wikipedia. We used two variants depending on the tokenizer used convert the text into token indices: character-level (the 210 most common characters in the dataset) and word-level (a BPE tokenizer with 16k token, trained on the dataset as in GPT-2).
MNLI The Menti-Genre Natural Language Inference dataset (Williams et al. 2018) is a task from the GLUE benchmark (Wang et al. 2018). It contains 433k sentence pairs, labelled with entailment information. The model has to predict whether the two sentences are an entailment, a contradiction, or neutral. For evaluation,
the validation split with domains matching the training set.
C.3 MODELS
Pre-act ResNet We the same architecture the original ResNet for CIFARIO (He et al.) 2013), using pre-activations as in (He et al. 2016). We only modify the downsampling layers as described in Appendix
We use:
g (x) (Baumple,(Blokk., (Dowissppl)))))))
=x
(14)
ResNet Similarly, we use the same architecture as the original ResNet for CIFARIO (He et al. 2015), and we only modify the downsampling layers described in Appendix G
We use:
ge(x)) = Upsample, (Block, (Downsampleees
ry ra 2) = ReL.U( + +
(15)
GPT-2 We use a transformer model following the original GPT-2 architecture (Radford et al, 2019), with only smaller dimensions and vocabulary size. Note that GPT-2 uses pre-nomaalizaton, Le. the layer-norm is applied at the beginning of all layers, and the residual connection is purely linear. This implies that the residual connection is linear:
g (x) Block; (x))
r (x x +++
(16)
However, the above choice for 9; and r; is not suited and makes Highway-BB BP diverge. We believe this is because the layers learn to cancel part of their residual connection. Taking this into account, in our experiments we used:
9a (a) (lock((x:) + y
(xx 2) = (1 7))x +
(17)
where we picked 7 0.2 with minimal tuning. Finding better ways of choosing ?, understanding why it is necessary, and studying how this impacts Highway-BR. are future works that could help to considerably improve Highway-BP
such models.
Note that the two equations are mathematically equivalent when doing backpropagation, but the latter greatly improves the convergence of Highway-BP. In particular, ? is equivalent to the previous equation, and = 1 makes Highway-BP behave exactly like the fixed-point iteration baseline.
RoBERTa We finetune the pre-trained RoBERTA- base model introduced by Liu et al. (2019). Compared to GPT-2, RoBERTA uses post-layernorm, i.e. applies LayerNorm layer after each
16

---

Published as a conference paper at ICLR 2025
Table 4: Hyperparameters used in the deep models experiments. In language modeling tasks, the last input dimension is the vocabulary size.
Model
Pre-act ResNet32
ResNet110
ResNet56
GPT-2
RoBERTa
Dataset
CIFARIO
CIFARIO
ImageNet32
Wikitext103
MNLI
Optimizer
SGD
SGD
SGD
Adam W
Adam W
Epochs
200
200
100
5.3
2
Learning rate
0.1
0.1
0.1
lc-3
3e-5
Batch size
128
128
128
128
32
Warmup steps
0
2k
2.5k
Momentum
0.9
0.9
0.9
0.9
0.9
Adam B2
0.98
0.999
Weight decay
le-4
le-4
le-4
0.1
0.1
Gradient clip
1.0
1.0
Input shape
(32, 32, 3)
(32, 32, 3)
(32, 32, 3)
(256, 16k)
(128, 50k)
Hidden dim
+
+ 64
256
768
Layers
32 464k
110
56
12
12
Params
1.7M
917k
14.5M
124M
residual connection.
f1(2) LN,(x + (lock;
(18)
14 (m)
r + Block, B(r Block,
O + 3
(19)
Var(x r + Block, (x)) +e
(x))
x O a;
a
+ (z) - (i(x)) @ of(x)
+ 8
(20)
0:
(x(z)
bz(x)
① (x) + (x))
(21)
Which leads the natural choice for ga and ry:
ge (x) (z), (z))
(2
(22)
However, similarly to the GPT-2 experiment with Equation 17 we introduce small change to improve the convergence of Highway-BP.
(xx) [11 (I), (x)) yor O a;(x))
(23)
This comes naturally when, instead of splitting + (lock;(x: into x and (lock; we split it into (1 7) and yx + (lock, We found y there are still many things to understand with lot of room for improvement, which we leave as future work.
be good choice, however again we believe
RNNs The RNN models contain linear layer project the input the hidden dimension, fol- lowed by the RNN layers, and then a classifier. The classifier is linear for language modeling (Wikitext103). and is two-layer MLP for sequence classification (CIFAR10 pixel-level). addi- tion, when stacking multiple RNN layers, we introduce residual connections which greatly improve convergence speed.
C.4 HYPERPARAMETERS
We report the hyperparameters used for sequential models in Table and for RNNs in Table Parameters that are not showed are set using their default values.
17

---

Published as a conference paper at ICLR 2025
Table 5: Hyperparameters used in the RNN experiments. In language modeling tasks, the input dimension is the vocabulary size.
RNN type
LSTM
GRU
GRU
GRU CIFARIO
Dataset
Wikitext103 char
Wikitext103 char
Wikitext103
Training steps
10k
10k
10k
2.5k
Learning rate
le-3
le-3
le-3
1e-3
Batch size
128
128
128
128
Warmup steps
lk
lk
250
Sequence length
256
256
256
1024
Input dim
210
210
16k
Hidden dim
512
512
512
64
Layers
Params
2.3M
1.8M
21.1M
29.8k
D MEMORY ANALYSIS
In this section we perform simplified estimation of the memory usage of backpropagation and Highway-Bi Backpropagation requires the following memory for each layer:
MBP Mwesight + Meache + 2MM
(24)
where Mweights is the size of the layer weights, Meache is the space taken by the intermediate vari- ables kept in memory for backpropagation, Mi, the size the hidden state h2 (and its gradi- ent).
Highway-BP iteration involves a few variables per layer:
ac in backpropagation, as / (the current gradient estimate): it plays the same role as ah; is backpropagated through the residual block gs, does not take additional memory.
ac in backpropagation. u ++ similarly, plays the same role as Th.-1
w (the initial estimate); this variable needs to be stored.
K; (the Jacobian of the residual connection): already included in the intermediary variables stored by backpropagation.
CumSumProd operation: we see from Algorithm| I that requires two tensors per layer we inplace operations. The first tensor is (mm) the product of the K, (if K, I as in transformers for instance, there nothing to store) which usually takes the same space as h; (e.g. if K; is diagonal). The second tensor is u, for which we can reuse the memory allocated to of ++ using inplace operations.
(m)
This leads to only two additional tensors to store (w) and (m))
M MHigwwyyBP = Mweiighs + Meache + 4,M
(25)
Note that practice Menche is what takes most of the memory. transformers for instance, Mexche R 17.M..- Nevertheless, there can be a non-negligible memory overhead for models with small residual blocks.
E PRACTICAL CONSIDERATIONS IN A DISTRIBUTED SETTING
While Highway-BP is a new way of speeding up training, it has to be noted that is also does not conflict with the standard techniques used in distributed training. The most widespread technique data parallel, which splits the input bach across devices to process each part in parallel. This can still be used with Highway-BP. since algorithm is only used for computing (or approximating) the gradient. In a layer parallel (or pipeline parallel) setting, layers are on different devices, which exactly where Highway-Bi -BP would be useful.
18

---

Published as a conference paper at ICLR 2025
The way Highway-BP may conflict with other distributed modes is if the number of devices is constrained. One would then have to decide which devices should used either for increasing the batch size (data parallel) speeding up backpropagation (Highway-BP),
Regarding the potential communication overhead, the main difference with backpropagation the CumSumProd operation which performs prefix scan of tensors stored on all devices (one per layer). While the algorithm is parallel, is nontrivial to implement efficiently and there can be different strategies. With fully distributed strategy, there will be more communications since layer needs to send its tensor to two other layers at each iteration. Another approach is a centralized strategy, where all tensors are gathered together, CumSumProd is computed locally, and the results are sent back to the layers, which reduces the number of communications between devices. Finding out which implementation is better is an important future work. Note as well that there are multiple parallel prefix scan algorithms, each with different advantages.
F SPEED COMPARISON BETWEEN TRAINING ALGORITHMS FOR RNNS
Table 6: Computation times for different RNN tasks, algorithms, and l values.
Task
Algorithm
Training step time (ms) w.r.f. i
0
2
3
5
10
20
1 LSTM layer (L=256)
Backpropagation
261
Fixed-point iteration
81
92
99
111
128
173
262
Highway-Bi
87
101
114
125
154
218
347
1 GRU layer (L=256)
Backpropagation
209
Fixed-poie iteration
62
69
73
79
92
119
176
Highway-BB
66
77
88
97
118
166
266
3 GRU layers (L=256)
Backpropagation
687
Fixed-point iteration
226
243
260
281
316
400
573
Highway-Bi
240
268
300
332
389
538
839
1 GRU layer (L=1024)
Backpropagation
715
Fixed-point iteration
191
207
204
203
209
216
241
Highway-Bi
205
210
210
210
230
248
292
G HIGHWAY-BP AND DOWNSAMPLING LAYERS IN RESNETS
For Highway-BP be effective on ResNet models, we need adapt the downsampling layers from the original architecture. Indeed, they are not convenient to handle in our framework since the downsampling occurs in the residual connection, and the default downsampling layers have no simple way of computing and factorizing their Jacobians.
We take inspiration from i-RevNet (Jacobsen et al. 2018), in which the downsampling layers are invertible: the image is split into blocks of size s x which are then flattened to produce one vector per block. This is already much easier to handle Highway-BP, for instance in a pre-activation ResNet this means K; is simply a permutation. Furthermore, they are also easily factorizable, since downsampling with block size 81 and then 82 equivalent to doing it once with block size 8182- However, limitation is that when this layer divides the width and height by s, it also increases the vector dimension by s², as opposed to the original ResNet where the dimension increases by s. This leads to the number of parameters exploding.
The above downsampling layer can be modified to fix this issue. Before flattening the blocks, we their height dimension. We thus obtain rows of vectors, which once flattened produce vectors scaled by instead of s². We used this modification our implementation.
perform average
Note that the ReRevNet downsampling and function such that Downsample( (psample(x)) = æ. We take advantage such properties in our implementation by wrapping all residual blocks by downsample layer and upsample layer, using
right-invertible, i.e.
define an Upsample
19

---

Published as a conference paper at ICLR 2025
the same block sizes the original ResNets, instead of placing the downsample layers in the residual
connections. This way, the residual connections are even simpler, since we have K; = I. For pre- activation ResNets, this is mathematically equivalent to the original implementation. For ResNets with ReL.U between blocks, this is slightly different since (aapsammppeeeeeLLUUuaaapppllpll but has no noticeable impact.
# x,
20


Reviews:
[{"review_id": "gFiZspfHGC", "reviewer": "Reviewer_3hir", "summary": "This paper proposes a method to accelerate the backward pass of backpropagation through parallelization of the gradient calculation in deep neural networks with residual-like architecture. This is done by decomposing the gradient into a sum of terms, where the k-th term is the sum of all gradients passing through at most k blocks which are not a residual (i.e. skip-like) connection. By parallelizing this computation, an acceleration is achieved as long as the maximal k we compute is not too large (i.e., we need to drop values of k above some threshold). It is shown empirically that k=5 is approximately enough for good accuracy on several (small) benchmarks, while still achieving significant acceleration.", "questions": "3. I think there are some small mistakes in the math. For example, in eq. 8: \n\n$\\quad$ a) The transition to eq. 8 from eq. 6 is not clear. Specifically, why did $w_{i}^{0}$ disappear?\n\n$\\quad$ b) The right side of eq. 8 seems different from the middle part of the equation, since $v_{i}^{k+1}$ is multiplied by $K_{i+1}$ in the middle part of the equation, but not in the right side of the equation.\n\nCan the authors please check these?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem of accelerating Backprop is important.\n\n2. The idea is novel, original, and quite interesting.\n\n3. The empirical results are promising.\n\n4. The presentation is mostly clear.", "weaknesses": "1. The main issue is the scale of the experiments, which is rather small (e.g., no ImageNet). It is not clear to me that a small k would be enough to get both high accuracy and acceleration on more complicated problems than those shown in this paper. \n\n2. The acceleration is only relevant for the backward pass, which is roughly a third of the total time of Backprop (which includes the forward pass, backward pass, and parameter gradient) and so this limits the overall benefit of this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IFuIwssEcM", "reviewer": "Reviewer_vqXj", "summary": "The paper introduces Highway backpropagation (Highway-BP), an algorithm that speeds up training of deep neural networks by approximating traditional backpropagation using parallelizable iterative methods. It leverages residual connections to propagate gradients efficiently and is adaptable to various architectures like ResNets and Transformers. Empirical studies show that Highway-BP achieves significant speedups with minimal performance loss, making it a promising method for accelerating deep learning model training.", "questions": "How does Highway-BP compare with other advanced optimization techniques, especially in terms of convergence speed and final model accuracy?\nWhat are the performance implications of Highway-BP when scaling to models with billions of parameters, and how does it handle memory and computational constraints?\nCan the authors provide more theoretical analysis or proofs regarding the convergence properties of Highway-BP compared to traditional backpropagation? For example, give a lower bound for the improvement.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Parallelization: The algorithm is designed to be parallelizable, which can significantly reduce training time, especially for very deep models or large-scale problems.\nArchitecture Flexibility: Highway-BP is adaptable to a variety of common neural network architectures, making it a versatile tool for different applications.\nTheoretical Foundation: The paper is grounded in mathematical theorems that provide a theoretical foundation for the proposed method.", "weaknesses": "Generalization to All Models: Although Highway-BP is adaptable to many architectures, there may be specific models or scenarios where it does not perform as well as traditional backpropagation or other optimization methods. For example, experiments on Graph neural network, spiking neural network and mamba.\nHyperparameter Tuning: The algorithm introduces a new hyperparameter (the number of iterations, k), which requires tuning and may lead to different optimal values depending on the model and task. This can add complexity to the training process. A ablation on how to tune the parameter should be included.\nMissing baselines: The paper compares Highway-BP with backpropagation and fixed-point iteration but may not fully address how it stacks up against other state-of-the-art optimization techniques. Here are some papers authors should include in their baseline:\nHuang, Kai, et al. \"Towards Green AI in Fine-tuning Large Language Models via Adaptive Backpropagation.\" arXiv preprint arXiv:2309.13192 (2023).\nWang, Ziteng, Jianfei Chen, and Jun Zhu. \"Efficient Backpropagation with Variance-Controlled Adaptive Sampling.\" arXiv preprint arXiv:2402.17227 (2024).\nYang, Yuedong, et al. \"Efficient low-rank backpropagation for vision transformer adaptation.\" Advances in Neural Information Processing Systems 36 (2024).\nScalability to Distributed Settings: The paper mentions the potential for distributed training but does not provide empirical results or a detailed discussion on how Highway-BP would perform in a distributed setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "40FoVDDJKA", "reviewer": "Reviewer_rbdV", "summary": "This paper considers gradient computation in network architectures with multiple residual connections. The authors propose the Highway-BP method to improve the efficiency of gradient computation/estimation by decoupling and rearranging multiple computational paths in vanilla BP. The proposed algorithm leverages the fact that computational burdens differ between the two paths within each block and iteratively approximates the gradient from easier to more challenging paths, allowing trade-offs between accuracy and efficiency through adjustable iteration steps. Numerical results indicate that the approximate gradient, with a certain number of iterations, achieves comparable results to the true gradient, demonstrating its potential applicability.", "questions": "- The authors should provide a more detailed experimental setup in the paper. In Figure 2, the final performance of vanilla BP appears to differ significantly from its usual training outcomes. Might this be due to insufficient training? As observed in Figure 3, the effectiveness of Highway-BP's approximation seems limited to the relatively easy early stages of training, suggesting that the accuracy results in Figure 2 may need to be presented at multiple time points.\n\n- The paper includes empirical tests of the algorithm, but could the authors provide a theoretical analysis of how the approximation error varies with $k$? Alternatively, could a more rigorous boundary be identified for the types of network structures where this approximation applies? Additionally, conducting a comprehensive analysis of the time and space complexity of Highway-BP is also worthwhile and important.\n\n- Since the paper aims to alleviate the sequentiality in BP, which is a challenge that typically arises in large models, could the authors evaluate whether the complex computation of Highway-BP might hinder the application of other crucial techniques in this area, such as data parallelism or distributed training? Furthermore, how might these structural changes to gradient computation impact communication and read/write overhead?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors present an interesting idea for improving the sequential computation in BP, utilizing the shared structural characteristics of mainstream models, which may facilitate further refined explorations in the future. The experimental results in the paper are comprehensive, demonstrating potential advantages in terms of efficiency.", "weaknesses": "My main concern is that unrolling the recursive computation in vanilla BP requires storing a significant number of additional variables, thereby increasing the algorithm’s storage demands. The authors do not appear to provide test results for memory consumption, which is an important consideration when developing algorithms for large-scale models. From the results in Table 3, the proposed algorithm shows a higher time overhead than the baseline, and with only 20 iterations, the time consumption is nearly equivalent to that of vanilla BP, indirectly suggesting that the additional read/write overhead may not be insignificant.\n\nThe efficiency of the proposed algorithm lacks theoretical guarantees, and the experimental results have minor flaws. Please see the question part for details. The comparison includes only the fixed-point iteration algorithm. Algorithms mentioned in the literature and other approximation-based acceleration methods for BP are not included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EOGQGloa40", "reviewer": "Reviewer_TXde", "summary": "Summary: This paper proposes Highway Backpropagation, an algorithm for accelerating the training of neural nets with residual connections (such as ResNets, RNNs, and Transformers) by skipping some of the expensive backpropagation steps. In a nutshell, backpropagating a gradient through a layer consisting of a weight matrix plus a residual connection consists of a ‘cheap’ part (the residual) and an expensive part (the weight matrix), and the total gradient can be represented as the sum of all combinations of going through either. By omitting some of the terms of that summation - specifically those that have more than $k$ ‘expensive’ components, the gradient calculation can be sped up. The authors address the practical concerns of the implementation and provide experimental results validating their setup.", "questions": "* Why is the CIFAR-10 and CIFAR-100 model accuracy so low for the chosen architectures, or, conversely, why were architectures with such low accuracy chosen?\n\n* Especially for image classification, why was this method not tested in the most standard setups, for instance Imagenet-1K on a standard ResNet50? Admittedly, in a standard RN50 architecture, the residual layer spans several weight matrices, but this seems like a straightforward extension of the proposed method. For the architectures chosen, why is the number of parameters much lower than that of the standard models (for instance, a traditional ResNet30 would have about 25M parameters, versus this paper’s 4.1M)? What is meant in line 317 by “standard ResNet”?\n\n* In line 367, what is meant by “very reasonable performances”? \n\n* Is there an advantage to varying $k$ during the training process?\n\n* Why are the intermediate losses necessary for the theoretical framework to be effective?   Likewise, why is the proposed algorithm restricted to generative and classification tasks (liNE 153)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper tackles a crucially important problem, which is the enormous cumulative expense of training machine learning models, especially in the world of transformers. The idea seems to be quite original, and makes a great deal of intuitive sense. It is also straightforward to implement.\n\nOverall, the main body of the paper is generally well-written and lays out its argument in a logical and even exciting way. The mathematical argument is very clearly presented. Further, the practical considerations of creating an implementation with real-world speedups are addressed.\n\nThe training dynamics section is nicely presented and interesting.", "weaknesses": "### Main weaknesses:\n\nThe main weakness of the paper is the experimental section. While the method is clever, the real value of the idea is in whether it is actually useful, and unfortunately the experimental section does not sufficiently cover this point. Of course, we cannot expect a finely optimized implementation in a research work. However, this reviewer doesn’t see a reason that a basic, workable implementation couldn’t have been built for standard architectures (e.g., ResNets) or something much closer to them. Because of the authors’ use of these nonstandard implementations, at least some of the actual numerical results are very far below what could be expected.\n\nFor instance, for CIFAR100, a standard ResNet50 architecture would have an accuracy of over 85%; the 50-layer architecture used in the paper (albeit with about a quarter of the total parameters of a RN50)  has an accuracy of 40%. A ResNet20, which has far fewer parameters, should still be able to get something like 70-75%. The authors do not address or justify this gap.\n\nLikewise, real-world speedup numbers are not presented. Again, the point regarding the practical difficulty of creating a CUDA-optimal distributed setup is well taken, but the question remains unanswered. Perhaps it may have been possible to create a less-optimized standard implementation of backpropagation (e.g., on CPU) and compare this to the proposed one.\n\nFinally, there is no guidance for selecting the right $k$ for a new task. This omission makes it difficult to use Highway Backpropagation in practice, even if an efficient implementation is available. \n\n### Smaller issues:\n\nThe writing style of the introduction/related work sections is often vague or confusing.\n* Like 36: what is meant by “transformers only defer the problem”?\n* Line 41: What is meant by “often involve trade-offs between speed and task performance”? Aren’t these tradeoffs always involved?\n* Line 42: What is meant by “leverage the recent layers”?\n* Line 45: What is meant by “significantly improves optimization”? Can vague statements like this be made more precise?\n* Line 51: What is meant by “derived from an original derivation”?\n* Line 75: “where gradients can either diminish or grow uncontrollably” is completely redundant with line 73.\n* Line 92: “While this seems attractive, the reality differs” - can this be made more clear and precise?\n* Line 93: “computing the Jacobian matrix of all layers”: this is misleading, since the proposed Highway-BP algorithm also requires this computation.\n\nLine 192 typo provie -> provide\n\nLine 231 typo $K$ -> $k$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7PEE4IXd5W", "reviewer": "Reviewer_MEWH", "summary": "This paper introduces a new algorithm for approximating backpropagation in deep sequential neural network models: Highway Backpropagation (Highway-BP)\n\nMotivation: Backprop is computationally expensive, and has sequential dependence in depth making it hard to parallelize.\nKey idea: Highway-BP leverages residual connections in modern architectures to decompose the gradient into paths of different lengths. It then iteratively computes an approximation of the gradient. Pruning some of the gradient paths enables Highway-BP to compute the approximate backprop in parallel. Doing this iteratively allows the approximate gradient to approach the true gradient over iterations.\n\n\n\n\nI really like this paper. In my head I'm drawing a parallel with the \"Residual Networks Behave Like Ensembles of Relatively Shallow Networks\" paper. In Figure 1 from that paper, they \"unravel\" the residual connection of an N=3 block network.\nWhen I read your paper, I immediately drew a parallel with that paper where, a view of your method, is that you only take backprop thru the right diagonal portion of the \"unraveled\" representation of Figure 1(b) of the \"Residual Networks Behave Like Ensembles of Relatively Shallow Networks\" paper.\nIts brilliant!\n(I am not an author of that paper :) )", "questions": "The results obviously show that the using the method produces worse models.\nHave you looked at methods for recovering performance?\nHave you tried increasing k over training?\nIn figure 3, I can see a schedule in which, for the first 500 steps, you use k=1, then for the next 500 steps, you use k=2, then for the next 1000 steps, you use k=5, then eventually you use backprop a the end of training to fully recover performance.\n\n\nEdit:\nCan Figure 3 be redone but with training time on the x-axis (instead of time)? Appendix C shows that your method will look more favorable when this is done.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "(1) Flexibility: Can be applied to various architectures like ResNets, Transformers, and RNNs. \n\n(1.5) Experiments: Tested on image classification (CIFAR10/100) and language modeling (Wikitext103) tasks. Compared against standard backpropagation and fixed-point iteration (for RNNs). Achieved comparable performance to backpropagation with fewer iterations (k ≤ 5 in most cases). Showed consistent speedups (2x-4x) over standard backpropagation.\n\n(2) Paper presents a tradeoff frontier: Number of iterations (k) controls the trade-off between speed and accuracy\n\nEdit:\n(3) The parallelism unlocked by this method can greatly accelerate training speeds producing more efficient and faster training setups. The fact that it generalizes to a lot of settings makes this paper :chefs-kiss:\n(see Appendix C for potential speedup wins (but this could be better presented in the main body))", "weaknesses": "This proposes a way to accelerate training at the cost of model performance\nThe true cost of AI is inference NOT training. I'd pay more for training if it produced a better model (and therefore made TCO cheaper)...\n\nThe results obviously show that the using the method produces worse models\n\n\nI'll be the echo chamber (somebody has gotta do it): I want to see this applied to large scale autoregressive LLMs. Show me LLaMa training or I don't care! \n\\s", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)