Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learning
Authors: Bei Li
Tong Zheng
Rui Wang
Jiahao Liu
Qingyan Guo
Junliang Guo
Xu Tan
Tong Xiao
JingBo Zhu
Jingang Wang
Xunliang Cai
Venue: NeurIPS 2024 poster (2024)
Keywords: Transformer
ODE
predictor-corrector
architecture
Primary area: natural_language_processing
TL;DR: We introduce a predictor-corrector based Transformer (namely PCformer), where the predictor is an EMA augmented high-order method, and the corrector is a multstep method.

Abstract: Residual networks, as discrete approximations of Ordinary Differential Equations (ODEs), have inspired significant advancements in neural network design, including multistep methods, high-order methods, and multi-particle dynamical systems. The precision of the solution to ODEs significantly affects parameter optimization, thereby impacting model performance. In this work, we present a series of advanced explorations of Transformer architecture design to minimize the error compared to the true ``solution.'' First, we introduce a predictor-corrector learning framework to minimize truncation errors, which consists of a high-order predictor and a multistep corrector. Second, we propose an exponential moving average-based coefficient learning method to strengthen our higher-order predictor. Extensive experiments on large-scale machine translation, abstractive summarization, language modeling, and natural language understanding benchmarks demonstrate the superiority of our approach. On the WMT'14 English-German and English-French tasks, our model achieved BLEU scores of 30.95 and 44.27, respectively. Furthermore, on the OPUS multilingual machine translation task, our model surpasses a robust 3.8B DeepNet by an average of 2.9 SacreBLEU, using only 1/3 parameters. Notably, it also beats LLama models by 5.7 accuracy points on the LM Harness Evaluation.

Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learning
Bei Li
Tong Zheng' Rui Wang
Jiahao Liu Qingyan Guo Junliang Guo
Tong Xiao Jingho Zhu' Jingang Wang Xunliang Cai? Northeastern University "Meituan Inc. Miirosoof Research *Tsinghua University com tzheng24@umd, edu {ruiwa, janaajunnaannan xuta) crosooo com gqy220mails tsinghua, edu. en (xiaotong, zhuj ingbo) @mail neu, edu cn
Xu Tan
[libei17, liujiahaol2, wangjingango2,
Abstract
Residual networks, as discrete approximations of Ordinary Differential Equations (ODEs), have inspired significant advancements in neural network design, including multistep methods, high-order methods, and multi-particle dynamical systems. The precision of the solution to ODEs significantly affectt parameter optimization, thereby impacting model performance. In this work, we present series of advanced explorations of Transformer architecture design minimize the error compared to the true "solution." " First, we introduce predictor-corrector learning framework to minimize truncation errors, which consists of a high-order predictor and multistep corrector. Second, we propose an exponential moving average-ber based coefficient learning method strengthen our higher-order predictor. Extensive experiments on large-scale machine translation, abstractive summarization, language modeling, and natural language understanding benchmarks demonstrate the superiority of our approach. On the WMT' 14 Biglish-German and English-French tasks, our model achieved BLEU scores of 30.95 and 44.27, respectively. Furthermore, on the OPUS multilingual machine translation task, our model surpasses robust 3.8B DeepNet by an average of 2.9 SacreBLEU, using only 1/3 parameters. Notably, it also beats LLama models by 5.7 accuracy points on the LM Harness Evaluation.
Introduction
Residual networks [16], formally Vr+1 = F(ye: 0:), represent a cornerstone the development of deep neural networks [59, 10], primarily due to their capacity to facilitate the flow of information across multiple layers. Beyond their pivotal role in convolutional networks, residual connections have become essential element in the architecture of more complex models, including the Transformer [59] and its various derivatives. This concept can be likened to the discretization process in the Euler method [63, 37, 15, 51, 29], which serves first-order solver for ordinary differential equations (ODEs), where dy(() de (t)(t), (tt) both cases, the new state (be it the next layer's output in ResNets or the solution the next time step in the Fuler method) is computed by taking the current state and adding an adjustment term.
Given this analogy with ODEs, there has been a surge of interest in improving residual network architectures by using more powerful numerical methods for ODEs. For instance, the linear multistep method [62, 37, 71, 24] has been employed to bolster the optimization of deep models. Other efforts have included redesigning the Transformer architecture from multi-particle dynamical system
Part work was done during an internship Microsoft Research Asia CCoresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

perspective [38. 12] and improving parameter learning efficiency through high-order methods [29]. Additionally, ODEs have been extensively studied for their potential accelerate diffusson processes, with multistep and high-order solvers offering more accurate predicted noise among each denoising process, generating comparable images but consuming much fewer NFEs [31, 35, 36).
In this work, we mainly focus on advancing the architecture design, specifically by minimizing the truncation error across each timestep. Building upon the ODE Transformers [29], which replace the first-order Euler method with a high-order method for more precise numerical solutions, our focus extends to addressing two key limitations. First, high-order solutions, such as those from the Runge- Kutta method or multistep methods, are found not to lead to significant improvements when we scale up training data and/or model size. Second, the gated fusion coefficient learning method, which is widely used in previous work, is not well suited for higher-order solutions.
Our work draws inspiration from the predictor-correcto method [11], well-established approach in numerical analysis known for its accuracy in solving differential equations. This method involves a two-step process: prediction step that estimates the solution based on known conditions, followed by correction step that refines the prediction for a more accurate result. We introduce a novel family of PCformers that embrace this predictor-correcton paradigm. Our approach integrates the final solution using an exponential moving average (EMA) method, capitalizing on the insight that higher-order intermediate approximations tend be more accurate. This assertion is supported by the truncation error analysis presented in Section 3.1.2. Our method is not only readily extensible to arbitrary higher orders but also consistently outperforms the gated fusion method.
Our contributions are summarized below:
We extend explicit ODE solutions implicit ODE solutions via predictor-correctt learning paradigm. This kind of iterative refinement can attain more accurate solutions than previous studies both thcoretically and empirically. In particular, we choose the high-order method as the predictor and the multistep method as the corrector.
To further strengthen the learning ability and training stability for high-order methods, we propose an exponential moving average coefficient learning method to replace the constant coefficients. This leads a much stronger predictor.
Our extensive experimental evaluation several benchmarks. including WMT 14 English- German, WMT' 14 English-French. WMT' 16 Romanian Engllshh, and the OPUS multilingual machine translation benchmark, demonstrates the superior effectiveness of our PCformer models. Notably, our model surpasses the 3.8B DeepNet by an average BLEU score of 2.9 with only 1/3 of the parameters. Furthermore, our model can be extended to other domains. Results abstractive summarization, language modeling, and language understanding tasks demonstrate its generality.
Background
We build our method upon Transformer [59] it is one of the most popular models in NLP. The encoder is a stack of identical layers. Each layer consists of a self-attention block and a feedforward network (FFN) block. Both of them equipped with residual connection [16] and a layer normalization unit [25]. The output of a block can be defined as
y = + F(v,,
where F-)) is either the self-attention FFN block. This equation illustrates that the layer output Vt+1 is determined by the layer input y/: and learnable derivative estimated by the current function F. This approach aligns with the benefits of the Euler method, which will discuss in the following sections. In this work, we use F, denote the (-th layer representation and F, to denote the i-th order intermediate approximations.
The Euler Method The Euler method is the most basic solution to solve ODEs given the initial value, involving function v(t) of variable and its derivatives. The Euler method defines the first-order derivative of yt)
dy(t) = t))
(2)
dt
where (v((1), () defines a time-dependent vector field if we know its value at all points of y and all instants of time t. Eq. 2 illustrates that the change of variable is determined by its current value

---

MultsShop Corrector
Fit:
7.
P-
EMA Rahanced High Order Predicter
L-Oder, 1-kp
L-Onlos, Mutii Siep
High-Onker, 1-Step
High Under Prodicter, Mulli Step Cienacte
Ge Prodictur-Chily Parneligm
Pradicter-Caraate Paradigm
Figure 1: Illustration of several advanced numerical methods and our proposed predictor-corrector paradigm. The right part plots 4-order method as the predictor to obtain Pe+1; Fe+1 then estimated via a function F(-); A 4-step method as the corrector to obtain the Vt+1-
and time variable t. deep learning, we can use a trainable function F(+) to estimate f t). In nutshell, residual networks could be regarded as a st-oorder discretization of the Fuler method [63, 29]. The advantage obvious since residual networks deliver consistent performance gains in the artificial intelligence, but the precision of yt-1 is limited. Fortunately, we can move forward along the numerical analysis perspective as more advanced numerical methods can alleviate this issue.
The Linear Multistep Method Compared with the Euler method, the linear multistep method previously obtained "solutions" estimate the current one, leading to more accurate results. Formally, multistep method could be defined as: Vt+1 Ur Σi=1 0g F,, where F, F(y,, 0z)-
Another family of numerical methods is high-order ODE solvers by repeatedly refining the solutions within single step. Previous work [29] employed the Runge- Kutta methods [50, 23, 6, 2] for a higher-order solution to ODEs, where Runge- Kutta is a classic family of iterative methods with different orders of precision. More formally, the explicit Runge-Kutta methods of n-order solution is defined to be: Vt+1 Ye Σ Fi. where F F(ye, 0c). f. F Σ 3ig F 0.). Note that F the intermediate approximation to the solution at an inner step. 3 and 7 are coefficients to model the scale of the input and the output of F.. This kind of method can be adapted to Transformer blocks by reusing F--) within a block.
The High-Order ODE Method
3 Predictor-Correcte Transformer
In this section, we first show the core design of Predictor-Correcto paradigm to more accurately solve ODEs. Then propose alternative coefficient learning strategy that could be applied to arbitrary orders using the merit of the exponential moving average. At last, we show some additional training techniques for stable and well pelforerfo training.
3.1 Predictor-Correcta Method
A genuine problem-solving process involves the repeated use of available information to initiate exploration, which discloses, in turn, more information until a way to attain the solution is finally discovered. - Newell et al. [43]
The Predictor- Corrrctor framework leverages an iterative process of using available information
to refine approximations continuously. Initially, the Predictor generates a rough estimate, which subsequently refined by the Corrector using newly available data. This cyclical process mirrors the

---

wroblem-solving strategy described earlier, where each iteration uncovers additional information that enhances the final solution's 's accuracy.
3.1.1 Adams-Bashhortt-MMoutton Methods
A peedictor-correcteo method typically uses an explicit method for the predictor and an implicit method for the corrector. Here we take the 4-step taees--Bssfoott-oooutie [1] methods as an instance, where
is the predictor and Adams-Moulton is the corrector. Adams- Bashforth is 4-step method, which defined as
1 (55F, - 24
Vi+1 +
where F; = F(ye, 0z). F(-) denotes the 1-th function, and 0g is corresponding parameters. Obviously, the BashfoBBshfroth methods are explicit since Ut++1 only depends on the "observed" statistics (Figa). Similarly, Adams-Moulton is also 4-step method as below
ye+1 = 3/2 +
(9F + 19F, 5F + Ft-2).
24
Formally, both Eq. and Eq. reused F,, F, Ft-2 improve the accuracy, but the corrector necessitates an approximate current "solution" F++1 to substitute Ft-3. which is an implicit method. This is because that 3/++1 the value to be solved, thus we cannot compute F++1- To solve this, the methods utilize Eq. 3 to obtain the approximate value (P++1) for Vr+1- Then F++1 could be approximated following F++1 = F(P++1, rough approximation, which is the combination of the preceding four layer representations, And the corrector then improves the approximation, offering a more precise sample derived from the data.
(12++1,0,, 0z). Concretely, the predictor provides
However, applying the Adams-Baat Basutorth-Moulton method directly to Transformer architecture design leads to unstable training and limited benefits due the difficulty in optimizing constant coefficients. Similar issues have been observed in training a Runge- Kutta (RK4) network with numerically suggested coefficients [29]. To address these challenges, Wang al. [62] proposed Dynamic I. Linear Combination of Layers (DLCL) method. This approach utilizes learnable coefficients and adjusts steps based on layer depth, effectively transforming it into a variable multistep method. Additionally, the Adams-Bashforti method is not the only choice for the predictor; other numerical methods, such as high-order methods, are also considered strong alternatives as they often provide more accurate solutions [29, 72].
3.1.2 High-order Predictor and Multistep Corrector
The aforementioned discussion motivates us to design a more powerful and stable architecture based on the above principles. Our preliminary experiments indicate that more accurate predictors indeed improve the performance, thus we choose a high-order method to serve as the predictor. A 2-order method could be defined
1t+1 = + ( fe)
(5)
where A = Fly,, 0,), and F. - Fly + Ê, 0.). Rather than utilizing the previously obtained representations in multistep methods, high-order methods iteratively estimate the approximations upon the last timestep. Similarly, a 4-order method is:
11+1 = +
(A
+ 2F + 2F3 + Fa)
6
where Fi = F(y,, 0,), F. = Fly, + ff.. Fs F(y, + f22 0.), and F4 = F(( + F 9,). To break the limit of constant coefficients, Liet al. [29] employed a gated network to dynamically compute the coefficients of Fi and F2. however, this method cannot applied higher-order methods, e.g., RK4, To facilitate higher-order optimization, we design a more flexible coefficient learning method via an exponential moving average strategy.
Predictor with Exponential Moving Average Coefficient Learning The Exponential Moving Average (EMA) method [20] is widely used for estimating time-series data by assigning variable weights to past observations, giving more importance to recent data compared to simple weighted averaging methods. We hypothesize that high-order approximations each step should have larger

---

impact on the final output, as they provide a more accurate initial state than previous ones. To support this claim, we replaced Eq. 6 by Ut+1 = F,, where E [1....: 4]. We used single-layer decoder to compute the perplexity (PPL) on the validation set to simulate truncation errors. Our expectation that fewer truncation errors, the larger the coefficient it should own.
Figure displays the perplexity comparisons. It shows that a 4-th order approximation, used replacement for the linear aggregation in Eq. 6, delivers comparable results and outperforms other cases. This observation motivates us to combine the benefits of EMA with the coefficient learning method. EMA is more flexible to the order of ODE solvers, that is could be easily extended to 2 orders, 4 orders, or even larger. Figure 1(b) illustrates the design merit of our proposed PCformer with an EMA coefficient predictor and a parameterization multistep corrector. Here, we use the RK4-block as an example. It is apparent that the original scales have been replaced by y, 7 (1- 7. (1 - 7)², and 7 (1 7)8 from Fa to Fi, where is learnable and the initialization 0.5 empirically. In this way, our n-order predictor approximates Pt+1 as follows:
150
Truncation Errors (PPL)
142.33
140
136.33
129.91
130
126.89
127.89
120
Vanilla
RK4
1st
2nd
3rd
4th
Figure 2: Truncation errors with different intermediate approximations.
n
P+11 = 3/ Σ r. (1-1)) F,
Corrector with Parameterization Leveraging a robust predictor, our corrector is designed to be computationally lightweight, striking an optimal balance between performance and efficiency. Utilizing the Adams-Moulton method, we parameterize the coefficients of previous states with learnable parameters. These coefficients are initialized using an EMA value, where the newly estimated Fi+1 assigned larger weight (a 0.5), and the weights of previous states decrease in descending order. In this way, we rewrite the Eq. 4 by
yt+1 yt + 0:) + Σ a1-(1aa) F. 1=--
(8)
where P(+1 is obtained with Eq. 7. Empirically, we found that when the dataset is limited, an Backward Euler method [5] as the corrector is enough to provide precise correction, where ye+11 yt + F(P++1,0.) 0.). We will discuss this in the analysis for more insights.
3.2 Improving Training Stability
Step Normalization (RK-Norm) We
Algorithm 1 Predictor- Corrector Paradigm
built our PCformer following pre-norm architecture [62, 58], by rewritten Eq. to M/++1 = Yt + F(LN(y)), 0.), where LN(.) denotes the normalization. This ensures that the representation is nor- malized before computing the derivative F,. To achieve this, we normalize the obtained intermediate approximations F each inner step and then compute
1: procedure PREDICTORCORRECTORRY H)
St for + to do if = 1 then
> Initialize an empty list to store F;
4:
Fi F-y,,, 0c)
Compute A
else
F, + Fly,, Sli 11,0,)
Compute A
end if
9:
LF(F;) + LayerNorm(F) S.append(LNN(F:))
P. Apply RK-Norm
the offset, e.g., Vr + (L)( to obtain
10:
b Store LN(F,)
11:
end for
the Fg for the next timestep. Mcanwhile,
12:
Compute P.41 using S via Eq.
Predictor Compute F++1
the F; in Eq. 7 is rewritten by LN(F.).
13:
Fe+1 f- F(P++)
If not, this oversight can cause insta-
14:
Compute yt+1 using via Fq.
Corrector
bility when computing the final ODE solution, where we will make ablations
15:
H.add(Fe+1)
> Store Fe+1
16:
return V++1
> Return the layer output
17:
end procedure
in the analysis section. The algorithm
(right part) presents a more detailed computation flow single layer in our PCformer, where H
stores the previously obtained F++1-
5

---

Table 1: Comparison with the Wfate-of-the-ars the WMT En-De and WMT En-Fr tasks. We both report the tokenized BLEU and SacreBLEU scores for comparison with previous work.
WMT En-De
WMT En-Fr
Model
Layers
#Param
Steps
BLEU SBLEU #Param Steps BLEU SBLEU
Transformer [59]
6-6
213M
100K
28.40
222M
300K 41.00
MacaronNet [38]
6-6
30.20
Transformer-DLCC [62]
30-6
137M
S0K
29.30
28.6
Transtormer-Bat
6-6
61M
50K
27.89
26.8
69M
TO0K
41.05
39.1
RK2-block (Gated) [29]
6-6
61M
50K
28.89
27.7
69M
100K
42.31
40.3
RK2-block (EMA)
6-6
61M
S0K
29.11
28.1
69M
100K
42.44
40.4
RK4-block [29]
6-6
S0K
29.03
27.9
69M
42.56
RK4-block (EMA)
6-6
61M
50K
29.43
28.4
69M
100K
42.72
40.7
Transformer-Big
6-6
211M
100K
29.21
28.1
221M
100K
42.89
40.9
RK2-block (Gated) [29]
6-6
211M
100K
30.53
29.4
221M
100K
43.59
RK4-block [29]
6-6
211M
100K
30.39
29.3
221M
100K
43.51
41.6
PCformer (2-order)
6-6
211M
100K 30.90
29.8
221M
100K
43.85
41.8
Transformer-Bit Big
12-6
286M
100K
29.91
28.9
297M
100K
43.22
41.2
RK2-block (Gated) [29]
12-6
286M
100K
30.77
29.6
297M
100K
43.96
42.1
RK4-block [29]
12-6
286M
100K 30.55
29.4
297M
100K
43.81
41.8
RK4-block (EMA)
12-6
286M
100K
30.66
29.5
297M
100K
44.17
42.2
PCformer (2-order)
12-6
286M
100K
30.95
29.8
297M
100K
44.27
42.4
Table 2: Results on the En-Ro task.
Table Average Bac on the OPUS-100.
Model
Params BLEU
Models
Layers Hidden Params
X X-En En-X Avg
Transformer in [39]
62M
34.30
DeepNet [61]
200
33,2
31.1
Dellight [39]
53M
34.70
1000
512
3.8B
33,9
30.2
32.1
Transformer (Our impl.)
69M
33.49
200
512
863M
34.2
28.5
31.4
BranchNorm [33]
RK2-block (gated) [29]
34.94
1000
512
3.8B
35.0
29.6
32.3
PCformer (2-order)
69M
35.43
1024
466M
34.0
27.6
30.8
PCformer (4-order)
69M
35.49
Transformer
24
1024
618M
34.9
28.1
31.5
RK4-block [29]
226M
35.28
1024
466M
36.0
32.6
PCformer (2-order)
226M
35.55
PCfermer
24
1024
618M
36.9
30.5
33.7
PCformer (4-order)
226M
35.80
24
1536
1.2B
37.7
32.2
35.0
Sublayer Dropping Additionally, we observe that our models benefit from the rich information brought by high-order predictor and subsequent implicit multistep corrector. To prevent from overfitting (settling into sub-optimal solutions) as the learning ability is quite strong, we borrowed the sublayer dropping technique [28, 34]. The drop rate is empirically results in previous studies.
0.1 which delivers robust
4 Experimental Results
We mainly evaluated the proposed method on machine translation, abstractive summarization, lan- guage modeling, and language understanding benchmarks. The details of datasets, and corresponding hyper-parameten please refer to Appendix C. For a clear comprehension, note that RK2-block (gated) 2-order method with learnable coefficients in [29]'s work. And RK2-block (EMA) denotes our EMA strategy.
Results of En-De and En-Fr Table compares the proposed PCformer with state-of-the-aee systems base and large configurations. As ODE Transformer a strong baseline to ours, we implemented their results for a fair comparison. We can see that the proposed EMA coefficient learning method can further strengthen high-order methods, leading to better results than the gated fusion method in Li et al. [29]'s work (comparisons in RK2-block). And EMA can facilitate RK4-block to deliver a further gain of 0.40 BLEU points. The performance gains are more obvious for wider models, that PCformer sets matches the new state-of-then tte-art with fewer parameters. Notably, 6-yaerr PCformer (2-order) achieves a BLEU score of 30.90, surpassing the previous best of 30.77 by 12-layer RK2-bles b22bblock with gated fusion [29]. For En-Fr, PCformer outperforms the standard Big model

---

Table 4: ROUGE results on CNN//ailyMail summarization dataset.
Table 5; Perplexity results on Wikitext- 103, Adap- tive refers to Adaptive Input Transformer [3].
Model
RG-1
RG-2
RG-L
Model
Layers Params Valid Test
Surface Connection [32]
41.00
18.30
37.90
Adaptive
8L
146M
21.11 11 21.00
Transformer [59]
40.47
17.73
37.29
RK2-block (gated) [29]
8L
146M
20.02 19.98
RK22block (gated) [29]
41.58
18.57
38.41
PCformer (2-order)
81.
146M
19.50 19.21
PCformer (2-order)
41.96
18,99
38.74
Shortformer
8L
146M
19.04 19.78
RK4-black [29]
41.83
18.84
38.68
RK2-block (gated) [29]
8L
146M
18.67 19.23
PCformer (4-order)
42.10
19.13
38.87
PCformer (2-order)
8L 146M
18.01 18.55
Table 6: PCformer results against Transformer++ [58] on various configurations. All models are trained the same subset of the SlimPajama dataset (from 6B to 100B) with the Mistral tokenizer [21]. The last column shows the average over all benchmarks that use (normalized) accuracy as the metric.
Model
Param Tokens
Wiki.
LMB.
LMB. acc
PIQA
Hella.
SciQ
ARC-c [ec_norm=
Wino, acc
Avg.
ppl
ppl
acc f
acc noom
acc
acc
acc f
Transformer++
340M
6B
38.5
96.1
21.4
60.3
29.1
69.2
21.5
50.4
41.9
PCformer
340M
6B
35.3
78.8
23.6
61.6
30.1
71.6
22.9
51.8
43.6
Transformer++
340M
16B
28.3
65.3
29.8
63.2
33.9
73.2
23.1
51.4
PCformer
340M
16B
25.6
39.7
34.5
65.2
36.9
79.6
23.2
52.2
48.6
Transformer++
1.3B
16B
23.8
26.2
37.3
65.7
37.6
78.6
23.7
51.5
49.0
PCformer
1.3B
16B
20.9
23.2
42.5
68.3
43.4
81.5
25.1
52.4
52.2
Transformer++
1.3B
100B
16.3
11.8
51.6
71.0
51.7
86.7
28.1
54.6
57.2
PCformer
1.3B
50B
16.2
9.4
55.1
71.9
54.8
88.6
29.6
57.2
59.5
PCformer
1.3B
100B
14.0
7.4
59.6
73.8
60.0
90.7
31.7
61.7
62.9
PCformer
3B
50B
13.6
6.5
62.1
74.4
61.9
90.6
32.4
61.9
63.9
PCformer
3B
100B
12.1
5.8
64.3
76.3
66.7
92.6
35.3
64.0
66.5
by 1.00 and 1.05 BLEU points with 2-order and 4-order configurations. This demonstrates that the predictor-corrector paradigm is a more parameter-efficient option than pure high-order methods.
Results of En-Ro Table exhibits a similar phenomenon on the En-Ro task. Our predictor-correctt paradigm with EMA method achieves much better performance (35.49 v.s. 34.70) with DeLight within much less training cost. For bigger model (line 7), it obtains a BLEU score of 35.80. A much higher performance (36.00) could be achieved by carefully designed corrector which would be discussed in the subsequent analyses.
Results of OPUS Table 3 provides the comparison of PCformer against existing state-of-the the models [69, 61] on the OPUS-100 testset. The findings here are three aspects: 1) Across all configurations, PCformer delivers significant BLEU gains over vanilla baselines. 2) Our 12-layer EMA Pre-Cor model attains an average SacreBL.EU score of 32.6, which not only outperforms the 3.8B DeepNet but also beats its further optimized variant, BranchNorm, with only 1/8 model parameters. 3) PCformer can benefit from the enlarging width and depth. Notably, our 1.2B model shows an average SacreBLEU of 35.0, thereby setting a new state-of-the-are on the OPUS- 100 testset.
Abstractive Summarization Table presents the results of the abstractive summarization task. As shown, our PCformer consistently improves upon pure the high-order method [29], terms of three rouge scores. Notably, PCformer (2-order) even beats RK4-block which consumes less computation cost. Additionally, PCformer (4-order) sets a new state-of-the-ee
on the summarization task which excludes models based on pre-trained models. This result strongly supports our hypothesis that an appropriate coefficient learning schedule essential for the effectiveness of higher-order methods, thereby enhancing the performance of our PCformer model.
Language Modeling Table 5 presents a comparative analysis of our PCformer against vanilla Trans- formers Adaptive Input Representation [3] and Shortformer [48] settings. Our 2nd-order configu- ration achieves significant reductions in perplexity (PPL), outperforming Adaptive and Shortformer by 1.79 and 1.23 PPL. respectively, even within identical model capacity constraints. Remarkably, PCformer surpasses the high-orde method (RK2-block) by substantial margin in both settings both validation and test sets, demonstrating the superiority of PCformer.
7

---

Table 7: Comparison results the GLUE development set.
CoLA
QQP
MNLI-m/mm Acc
SST-2
STS-B Corr
QNLI Acc
RTE Acc
MRPC
Avg.
Model
Mcc
Acc
Acc
Acc
BERT
60.6
91.3
86.6/-
93.2
90.0
92.3
70.4
88.0
84.0
PCformer
65.9
92.0
87.3/-
93.6
90.8
92.8
74.7
91.5
86.1
LM Evaluation Harness In response to the increasing significance of attention mechanisms in LLMs, we conducted a comprehensive evaluation of PCformer using established benchmarks, LM Evaluation Harness [13], focusing diverse range of downstream tasks including common-sense reasoning and question-answerrng. Table 6 presents our findings, where we utilized llama-like model Transformer++, the foundation for PCformer. We trained models with parameter sizes ranging from 340M to 3B, using datasets comprising 6B to 100B tokens from Slimpajama. The experimental results indicate that PCformer consistently surpasses the performance of a well-tuned Transformer of equivalent capacity. Notably, PCformer achieves an average score improvement of 1.7 points for the 340M model and 5.7 points for the 1B model across six challenging subtasks. When scaled to a 3B parameter size, PCformer demonstrates even greater gains, achieving an additional 3.5 average score improvement compared the IB model, underscoring scalability and potential with larger model capacities and richer training datasets.
Language Understanding We also validate our method on the widely used natural language understanding benchmarks, namely GLUE, which consists of 8 sub downstream tasks. The evaluation metrics are follows: The result for STS-B is the Pearson correlation; Matthew's 's correlation is used for CoLA; Other tasks are measured by Accuracy. The results are presented in Table We see that PCformer achieves 2.1 points (on average) improvement over the BERT-large, which demonstrates the effectiveness of PCformer.
Analysis
In this section, we explore several significant issues, comprising the visualization of truncation errors,
COMET results, and set of essential ablation studies. We primarily conducted ablation studies on machine translation tasks, but the conclusions are generalizable.
Quantization of the Truncation Error Following the suggestion in [29]'s work, we use the perplexity between the single-layer Transformer decoder output and the ground truth to approximate the "truncation error". The results of Table were conducted on the Penn Treebank dataset. We see that the proposed EMA method achieves lower perplexity than the learnable coefficient (gated) learning method, similar observation 4-order (EMA Rk4-block), Additionally, the Predictor-Corrector paradigm can further reduce the truncation error, which demonstrates the effectiveness of our method.
Table 8: Comparison of PPL PTB.
Model
1-Layer
2-Layer
Residual-Blood
142.33
136.07
RK2-block
131.80
123.12
RK2-block (gated) [29]
128.48
121.02
RK2-block (EMA)
124.01
119.65
PCformer (2-order)
120.91
118.37
RK4-block
126.89
119.46
Corrector
RK4-block (EMA)
121.82
116.77
PCformer (4-order)
119.27
114.32
Evaluation by COMET Our PCformer con- sistently outperforms baselines, showing an even larger gap in COMET than BLEU, as shown in Table 9. Both metrics exhibit sim- ilar performance trends, highlighting our ap- proach's effectiveness. Increasing model depth from 6 12 layers does not improve BLEU for the En-De task but results in 0.64 COMET gain. A similar pattern is observed in the En- Fr task, where PCformer (4-order) achieves comparable BLEU to its 2-rrerr counterpart but
Table 9: COMET (%) v.s. BLEU (%) results.
En-De
En-Fr
Model
BLEU COMET BLEU COMET
Transforer-hig (61.)
29.21
51.87
42.89
71.21
PCformer (RK2)
30,90
54.74
43.85
73.96
PCformer (RK4)
44.10
74.76
Transformer-bii (12L)
29.91
52,90
43.22
72,33
PCformer (RK2)
30.95
55.38
44.27
75.09
PCformer (RK4)
44.21
75.33
gains 0.24 COMET.
we followed the setting Gu and Dao [14]'s work.

---

Table 10: Ablation the several choices of the predictor and corrector four translation tasks.
Predictor
Corrector
En-De
En-Ir
En-Ro
OPUS
First-order Baseline
29.91
43.22
34,20
31.5
ODE Transformer
30.77
43.96
35.28
32.3
RK2-block with EMA
Multistep Method
30.70
44.27
35.55
33.7
RK2-block with EMA
Backward Euler Method
30.95
43.68
36.00
33.2
Multistep Method
step Method
30.30
43.92
35,30
33.0
Multistep Method
RK2-block with EMA
29.78
42.68
34.40
32.5
Multistep Method
Backward Fuler Method
30.30
43.62
35.27
32.8
Table 11: Ablations on the PCformer design on the machine translation task. The evaluation metric is
BLEU (%).
Model
BLEU
Model
BLEU
T Transformer-aas
27.89
Transformer-big
29.21
RK4-block (EMA)
0.25
28.90
RK2-block (EMA Pre-Cor)
30.95
RK4-block (EMA)
0,50
29.43
w/o RK-Norm
failed
RK4-block (EMA)
0.75
28.99
Layer-wise
30.63
RK4-block (EMA)
0.99
29.20
Vector
30.88
(a) Coefficients: the effect of different y for the EMA coefficient learning.
(b) Other techniques: Figuring out several
key components for high-order solutions.
Ablation Study on Predictor-Corrette Framework The predictor-corrector framework is cru- cial in our work, with the choice of ODE solutions for each component significantly impacting performance. We experimented with various combinations of predictors and correctors, including high-order methods, linear multistep methods, and the Backward Euler method. Table 10 summarizes these results. We chose RK2-block with EMA as the default for high-order solutions due to its performance and efficiency. Key insights include: 1) The predictor must be highly accurate, as it sets the performance lower bound. High-order predictors outperform the multistep method (DLCL) and the Euler method. 2) complex corrector isn't 't always optimal: Backward Euler method suffices for small and medium datasets (e.g., En-Ro and En-De), while more complex methods may cause overfitting. 3) Combining multi-step method predictor with a high-order corrector performed worse than other combinations, highlighting the importance of predictor choice.
Ablation Study on Core Design Technique Table 11 presents the performance of our PCformer with various initial coefficients for the EMA method and stable training techniques. Our default, with Y set to 0.5, performs best because smaller values disrupt the numerical bound, and larger values overly focus recent approximations. As detailed in Section 3.2, RK-Norm is essential for training stability, as shown by the BLEU score drop without it. Testing different layer-wise coefficients and replacing the learnable scalar with learnable matrix vector showed no significant performance difference, so these were excluded from our default settings.
Inference Speed and Memory Con- sumption Table 12 presents a detailed comparison of inference speed and mem- ory consumption across various large model configurations, revealing that the proposed PCformer models achieve sat- isfactory inference performance. This is primarily because the computational overhead is concentrated on the decoder side rather than the encoder, as demon- strated in our experiments, Additionally, efffciiett memory-efficient, shown by the memory usage comparison between the baseline and the ODE Transformer. Despite the fact that our PCformer models are more than twice as slow as the vanilla baseline in encoder-only and decoder-only configurations, they deliver significantly superior performance, making the trade-off worthwhile. These performance gains are clearly illustrated in Ta- ble 6, where the substantial improvement in model effectiveness justifies the increased inference time. We acknowledge that further optimization to accelerate PCformer's 's inference speed is promising direction for future research.
Table 12: Comparison of inference speed (sentences/s) and memory consumption (GB) between the vanilla Trans- former and numerical Transformers. Model
Layers Inference Memory BLEU
Transformer
6
98.7
13.2
29.2
Transformer
12
94.5
18.7
29.7
Transformer
24
87.3
23.5
29.8
ODE Transformer (RK2)
6
93.5
15.1
30.7
PCformer (RK2 predeitor)
6
90.3
16.2
30.9
ODE Transformer (RK4)
87.1
17.3
30.5
PCformer is memory

---

More Analyses Due to the limited space in the main content, we summarized more detailed analyses in Appendix D, including the parameter efficiency (Figure 3), illustration of training and validation curves (Figure 4), and visualization of coefficients during the learning procedure (Figure 5). We anticipate that these analyses will offer a deeper and more comprehensive understanding of our method.
6 Related Work
Ordinary Differential Equations The connection between ResNet and ODEs was first proposed by Weinan [63], while Neural ODENet [8] introduced new perspective on neural architecture design. Several architectures [71, 24, 37, 17, 72, 38, 53] can be interpreted from the ODE perspective. Recent studies leverage ODE benefits for Transformers. Lu et al. [38] proposed MacaronNet using the Strang-Marchuk Splitting Scheme, and Zhang et al. [70] introduced continuous self-attention models. Dutta et al. [12] redesigned Transformer architecture for efficiency from mulli-pprticcl dynamic system view. Li et al. [29] showed that first-order ODE blocks could cause error accumulation, and high-order methods were suggested as solutions. In this work, advance the Transformer design with a more accurate Predictor Cooreecco paradigm and general coefficient learning strategy inspired by the exponential moving average, showing significant performance improvements on NL.P benchmarks.
ODE and Diffusion Models ODEs and numerical methods are also popular in diffusion models, reducing prediction errors in denoising processes. Text-to-image generation typically uses a two-stage model, including text-to-ine text-to-image diffusion model and super-resolution models. The standard diffusion model, DDPM [19], requires up to 1000 iterations to recover images from Gaussian noise. Subsequent work accelerated DDPMs using denoising equations [57] or scheduled variance [44], though often at the cost of performance. Liu al. [31] proposed treating DDPMs as solving differential equations on manifolds, introducing a pseudo linear multi-step method for efficiency and performance. Further diffusion acceleration efforts [35, 36] were motivated by ODE benefits. More recently, Xu al. [67] presented deep generative model solving the Poisson equation, opening new possibilities in text-to-image generation. ODEs also show promise in discrete diffusion models, as demonstrated by Lezama et al. [26], who used predictor-corrects paradigm to enhance the accuracy.
7 Conclusions
This paper advances the design of parameter-efficient neural network backbones through a numerical analysis perspective. Previous work has utilized high-order ODE solutions for more accurate approxi- mations at each block, yielding promising results on various sequence generation tasks. However, challenges remain, such the scalability of learnable coefficients to RK4-blocks and the lack of exploration into implicit ODE methods. To address these issues, we introduce a predictor-correcte framework to improve estimation precision. Additionally, we proposed EMA coefficients learning strategy to promote coefficients learning for high-order methods with high flexibility, Experimental results across 8 benchmarks demonstrate the general ability and strong effectiveness of our approach. More concretely, Our PCformer achieves 30.95 and 44.27 BLEU scores on the WMT" 14 En-De and En-Fr, setting a new state-of-the-an result on both testsets without considering data augmentation methods. Also, it delivers an average BLEU of 35.0 on the OPUS multilingual dataset, beating Deep- Net and other variants with much fewer model parameters. Notably, PCformer demonstrates strong potential in large language model scenarios, outperforming vanilla LLama models by a considerable margin across various configurations on LM Harness Evaluation. Our codebase could be found at https:/ gi thub. http//ibiinnu/Pole ormer.
Acknowledgments
This work was supported in part by the National Science Foundation of China No.62276056), the Natural Science Foundation of Liaoning Province of China 2022-KFF26-011 the Fundamental Re- search Funds for the Central Universities (Nos. N2216016 and N23160022 the Yunnan Fundamental Research Projects (No. 202401BC070021), and the Program of Introducing Talents of Discipline to Universities, Plan 111 (No.B16009 Jingang Wang is funded by Beijing Nova Program (Grant NO. 20220484098).
10

---

References
[1] Roger Alexander. Solving ordinary differential equations i: Nonstiff prohlems (c. hairer, sp norsett, and g- wanner). Siam Review, 32(3))485, 1990.
[2] Uri M Ascher and Linda R Petzold. Computer methods for ordinary differential equations and differential- algebraic equations. Siam, 1998.
[3] Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling, In Proc. of ICI.R, 2019.
[4] Yonatan Bisk, Rowan Zellers. Jianfeng Gao, Yejin Choi, Piqa: Reasoning about physical common- sense in natural language. In Proc. of AAAI, volume 34, pages 7432-7439 2020.
[5] John C. Butcher. Numerical methods for ordinary differential equations: early days. In Adhemar Bultheel and Ronald Cools, editors, The Birth of Nunerical Analysis, pages 35-44. World Scientific, 2009.
[6] John Charles Butcher. A history of runge-kutta methods. Applied numerical mathematics, pages 247-260, 1996.
[7] Bo Chang, Lili Meng, Eldad Haber, Lars Ruthotto, David Begert, and Elliot Holtham. Reversible architectures for arbitrarily deep residual neural networks. In Proc. of AAAI, pages 28 1-2818, 2018.
[8] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Proc. of NeurIPS, pages 5572-6583. 2018.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 180.00550, 2018.
[10] Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidircctional transformers for language understanding. In Proc. NAACL, pages 4171-4186, 2019.
[11] Kai Diethelm, Neville Ford, and Alan D) Freed. A predictor-corrector approach for the numerical solution of fractional differential equations. Nonlinear Dynamics, 29(1):3-22, 2002.
[12] Subhabrata Dutta, Tanya Gautam, Soumen Chakrabarti, and Tanmoy Chakraborty. Redesigning the transformer architecture with insights from multi-particle dynamical systems. pages 5531-5544, 2021.
[13] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPoti. Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Fric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024.
[14] Albert Gu and Tri Dao. Mamba: Lincar-time sequenec modeling with selective state spaces. CoRR, abs/2312.00752,, 2023.
[15] Eldad Haber, Lars Ruthotto, Elliot Holtham, and Seong-Hwan Jun. Learning across scales multiscale methods for convolution neural networks. In Proc. of AAAI, pages 3142-3148, 2018.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 2016.
[17] Xiangyu He, Zitao Mo, Peisong Wang, Yang Liu, Mingyuan Yang, and Jian Cheng. Ode-inspired network design for single image super-resolution. Proc. of Proc. IEEE Conf. Comput. Vis. Pattern Recognit. pages 1732-1741, 2019.
[18] Karl Moritz Hermann. Tomás Kocisky. Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proc. of NeurIPS, pages 1693-1701, 2015.
[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Proc. of NeurlPS, pages 6840-6851, 2020.
[20] Stuart Hunter. The exponentially weighted moving average. Journal of quality technology, 18(4):203-21 1986.
11

---

[21] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Manie-Anne Lachaux, Pierre Stock, Teven Le Scao. Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, CoRR, 0a0223100068255 2023.
[22] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. ICLR, 2015.
[23] Wilhelm Kutta. Beitrag zur naherungsweisen integration totaler differentialgliichunge.. Z. Math. Phys., pages 435 453,,,3,110 1901.
[24] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. In Proc. of ICLR, 2017.
[25] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization, ArXiv preprint, 2016.
[26] José Lezama, Tim Salimans, Lu Jiang, Huiwen Chang, Jonathan Ho. and Irfan Essa. Discrete predictor- corrector diffusion models for image synthesis. In Proc. of ICLR, 2023.
[27] Bei Li, Ziyang Wang, Hui Liu, Yufan Jiang, Quan Du, Tong Xiao, Huizhen Wang, and Jingbo Zhu. Shallow-to-deep training for neural machine translation. In Proc. of EMNLP, pages 995-1005, 2020.
[28] Bei Li, Ziyang Wang, Hui Liu. Quan Du, Tong Xiao, Chumliang Zhang, and Jingbo Zhu. Learning light- weight translation models from deep transformer. In Proc. of AAAI, pages 13217-13225, 2021.
[29] Bei Li, Quan Du, Tao Zhou, Yi Jing, Shuhan Zhou, Xin Zeng, Tong Xiao, JingBo Zhu, Xuebo Liu, and Min Zhang. Ode transformer: An ordinary differential cquation-inspired model for sequence generation. In Proc. of ACL, page 8335-8351, 8351, 2022.
[30] Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, 2004.
[31] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. 2022.
[32] Xuebo Liu, Longyue Wang. Derek Wong, Liang Ding, Lidia Chao, and Zhaopeng Tu. Understanding and improving encoder layer fusion in sequence- to se-uennence learning. In Proc. of ICLR, 2020.
[33] Yijin Liu, Xianfeng Zeng, Fandong Meng, and Jie Zhou, Branchnorm: Robustly scaling extremely deep transformers. In Findings of ACL, pages 11675-1 1687, 2024.
[34] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proc. of CVPR, pages 10012-10022, 2021.
[35] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling around 10 steps. Pruc. of NeurlPS, 2022.
[36] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu, Dpm- Fast solver for guided sampling of diffussion probabilistic models. arXiv preprint arXiv:2211.01095, 2022.
[37] Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In Proc. of ICML, pages 3282-3291, 2018.
[38] Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong. Tao Qin, Liwei Wang, and Tie- Yan Liu. Under- standing and improving transformer from multi- dynamic system point of view. ArXiv preprint, 2019.
[39] Sachin Mehta, Marjan Ghazvininejad, Srinivasan Iyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Delight: Very deep and weiht transformer. ArXiv preprint, 2020.
[40] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, In Proc. of ICLR, 2016.
[41] Tomáš Mikolov, Anoop Deoras, Stefan Kombrink, Lukáš Burget, and Jan Černocký. Empirical evaluation and combination of advanced language modeling techniques. In Proc. of INTERSPEECH, 2011.
[42] Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gulçehre, and Bing Xiang. Abstractive text summarization using eequence-to-squueee RNNs and beyond. In Proc. of CoNLL, pages 280-290. 2016.
12

---

[43] Allen Newell, J. C. Shaw, and Herbert A. Simon. Report on general problem-solving program. In Proc. of ICIP, pages 256-264, 1959.
[44] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proc. of ICML,, pages 8162-8171, 2021.
[45] Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 1-9, 2018.
[46] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: fast, extensible toolkit for sequence modeling. In Proc. of NAACL, pages 48-53, 2019.
[47] Denis Papemo, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lamhada dataset: Word prediction requiring a broad discourse context. arXiv preprint wrXiv:1600.00033 2016.
[48] Ofir Press, Noah A. Smith, and Mike Lewis. Shortformer: Better language modeling using shorter inputs. In Proc. of ACL, pages 5493-5505 2021.
[49] Ricardo Rei, Craig Stewart, Ana Farinha, and Alon Lavie. COMET: neural framework for MT evaluation. In Proc. of EMNLP. pages 2685-2702 2020.
[50] Carl Runge. Über die numerische auflösung von differentiallllcicunge.. Mathematische Annalen, pages 167-178, 1895.
[51] Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision volume, pages 352 364, 2019.
[52] Keisuke Sakaguchi, Ronan Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proc. of AAAI, pages 8732-8740, 2020.
[53] Michael E. Sander, Pierre Ablin, Mathicu Blondel, and Gabriel Peyré. Momentum residual neural networks. In Proc. of ICMI, pages 9276-9287, 2021.
[54] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proc. of ACL, pages 1725, 2016.
[55] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations, In Proc. of NAACL, pages 464-468, 2018.
[56] David R. So, Quoe V. Le, and Chen Liang. The evolved transformer. In Proc. of ICML, pages 5877-5886, 2019.
[57] Jiaming Song, Chenlin Meng. and Stefano Ermon. Demoising diffusion implicit models. In Proc. of ICLR. 2021.
[58] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, (iss/302..1997.. 2023.
[59] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Proc. of NeurlPS, pages 5998-6008, 2017.
[60] Alex Wang, Amanpreet Singh, Julian Michacl, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proc. of ICLR. OpenReview.net, 2019.
[61] Hongyu Wang. Shuming Ma. Li Dong. Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers 1,000 layers. IEEE Trans. Pattern Anal. Mach. Intell., 46(10)66761-777 2024.
[62] Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In Proc. of ACL, pages 1810-1822, 2019.
[63] E Weinan. A proposal on machine leamning via dynamical systems. Communications in Mathematics and Statistics, pages 1-11, 2017.
13

---

[64] Johannes Welbl, Nelson F. Liu, and Mat Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop Noisy User-generated Text, NUT @ EMNLP 2017, Copenhagen. Denmark, September 7, 2017, pages 94-106, 2017.
[65] Haixu Wu, Jialong Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Flowformer: Linearizing transformers with conservation flows, Proc. of ICML, pages 24226-24222 PMLR, 2022.
[66] Zhanghao Wu, Zhijian Liu, Lin, Yujun Lin, and Song Han. Lite transformer with long-short range attention. In Proc. of ICLR, 2020.
[67] Yilun Xu, Ziming Liu, Max Tegmark, and Tommi Jaakkola. Poisson flow generative models. arXiv preprint Xiv::22009.111 111199111111.111 2022.
[68] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:1905.07833 2019.
[69] Biao Zhang, Philip Williams, Ivan Titov, and Rico Semnrich. Improving massively multilingual neural machine translation zero-shot translation. In Proc. of ACL, pages 1628-1639, 2020.
[70] Jing Zhang, Peng Zhang, Baiwen Kong, Junqiu Wei, and Xin Jiang. Continuous self-attention models with neural ODE networks. In Proc. of AAAL, pages 14393-14401, 2021.
[71] Xingcheng Zhang, Zhizhong Li, Chen Change Loy, and Dahua Lin. Polynet: A pursuit of structural diversity in very deep networks. In Proc. of Proc. IEEE Conf. Comput. Vis. Pattern Recognit, pages 3900-3908, 2017.
[72] Mai Zhu, Bo Chang, Chong Fu. Convolutional neural networks combined with runge-kutta methods. Newral Compul. Appl. 12221166999611 2023.
14

---

Supplementary Material for "Predicto-Corrretor Enhanced Transformers with Exponential Moving Average Coefficient Learning"
A
Broader Impact
We do not anticipate any specific negative impacts from our work. However, as with any machine learning method, we recommend exercising caution. Our primary contribution is advancing neural model design from numerical perspective, aiming to facilitate models' parameter learning and enhance the performance, which believe be environmentally friendly. This approach encourages the community enhance neural models by integrating knowledge from various domains, such as physiology, physics, and mathematics, thereby further henefiting Al research.
B Limitations and Future Work
The proposed PCformer has demonstrated significant performance gains across a variety of tasks, including sequence generation, natural language understanding, and language modeling. While the encoder-decoder PCformer computationally efficient, only employs the predictor-correett paradigm on the encoder side, the primary computational overhead during inference remains the decoder. However, when our method applied to encoder-only models (for NI.U) or decoder-only models (for LM/LLLM), the additional computational complexity becomes non-negligible, Addressing how to further accelerate inference in these scenarios remains critical, and aim explore this in future work.
We plan address this issue from two primary aspects:
High-order Computation Latent Space: When adopting high-order methods, iterative computations among inner steps cannot be skipped. Therefore, we aim to investigate whether high-order computations can be performed in latent space with reduced dimensionality, such as 64 128 hidden dimensions, compared to the original 1024 dimensions. This approach presents significant theoretical challenge: maintaining the stability the ODE while computing higher-order intermediate approximations
latent space.
High-order Training and Inference in First-order Manner: Another alternative is to achieve high-order training and inference using first-order (Euler-like) approach. This might involve employing distillation techniques or treating high-order computations as a form of training regularization.
We will continue to advance the design of PCformers to balance performance gains with inference efficiency. particularly in the context of large language models. This research could provide valuable insights the community on whether such modeling methods can further enhance performance LLMs.
C Dataset and Evaluation
C.1 Machine Translation
We present experimental results across three WMT benchmarks, including WMT' 14 English- German (En-De), WMT' 14 Baglish-Frennde (En-Fr), and WMT 16 English-Romanian Romanian (En-Ro), well large-scale, challenging multilingual machine translation benchmark (OPUS).
Datasets
For the En-De task, the training data consisted of approximately 4.5M tokenized sentence pairs, as in [59]. All sentences were segmented into sequences of sub-word units [54] with 32K merge operations using shared vocabulary. We selected newstesr20]3 the validation data and newstes12014 the test data.
For the En-Fr task, we used the dataset provided within Fairseq, i.c., 36M training sentence pairs from WMT' 14, 2 was the validation data and newstest20/4 was the test data.
For the En-Ro task, we replicated the setup of [39], which used 600K/2K/2K sentence pairs for training, evaluation and inference, respectively.
Apart from the affrementioned three bilingual translation benchmarks, OPUS is multilingual benchmark that contains more challenges for the model to serve. We choose OPUS [69], an English-cente centric corpus covering 100 languages, which randomly sampled from the OPUS collection. For fair comparison with prior work.
15

---

Table 13: Statistics of the datasets and hyperparameters for sequence generation tasks. For the dataset, we both report the vocabulary size, sentence numbers of training, validation and test sets. For the training. Lr denotes the peaking learning rate and Warmup denotes the warmup step of the Adam optimizer. WD denotes whether we applied word dropout. For the inference, Beam and LP denote the beam size and length penalty, respectively.
Dataset
Training
Inference
Benchmarks
Vocab
Train
Dev
Test
Lr
Warmup
Batch
Steps
Beam
LP
WMT 14 En-De
34040
4.5M
3000
3003
0.002
16000
80K
50K
4 0,6
WMT 14 En-Fr
37284
35.7M
26,822
3003
0.002
16000
320K
100K
4
0.6
WM En-Ro
34976
602K
1999
1999
0.002
8000
80K
17K
1.3
OPUS
209816
109.3M
371068
376000
0.002
16000
320K
100K
5
1.0
CNii
32584
287K
13368
11490
0.002
8000
160K
50K
4
2.0
PTB
10008
908
73
0.002
2000
160K
3K
4 2.0
Wikitext-103
32584
180K
3760
11490
0.002
8000
160K
50K
2.0
we used the already sampled subset' and following the script provided by Zhang et al. [69] to pre-process the data, including the data filtering. sentence piece training and applying. After that, OPUS-100 contains approximately 55M sentence pairs. Following the same training strategy with [69, 61], we train single model for both to English (XE) and from English (EX), thus the total training data is 110M. Note that 2000 sentence pairs for each language to serve as validation and test sets.
Evalutation We measured performance in terms of BLEU. Both tokenized BLEU and SacreBLEU" scores were reported on the En-De and En-Fr tasks. For the OPUS task, we average the SacreBLEU scores among 94 languages for both X-En and En-X. The beam size and length penalty of each task are summarized in Table 13. In order to attain results that are more compelling, it is imperative to acknowledge that BLEU may not be suitable metric for evaluating model's 's performance. To supplement our evaluation, we have included summary of the COMET scores [49] of the top-performing model, utilizing reference-based model".
Training Details accordance with the recommendations provided by [27], we have incorporated relative positional representation [55], namely RPR for short, into our model architecture to establish stronger baselines. To ensure stable learning during training. we have also borrowed the merit of dense connections among layers [62] for stable optimization within FP16 training. Our models were trained 8 GPUs with 096 tokens per GPU. For the En-De, En-Fr and OPUS tasks, we have observed that larger batching schemas often result in better convergence [45]. Therefore, we accumulated gradients every 2 and 8 steps for En-De and En-Fr/OPUS, respectively. Adam optimizer [22] with (0.9, 0.997) for B1 and 82 is adopted. The hyperparamcters including the lcarning rate, the warmup step and the total training steps of three tasks could be found in Table 13. Note that we trained Base/Deep and Big models for 50K and 100K steps the En-De task. We regarded merging SAN and FFN as the default ODE block. addition, main results were the average of three times running with different random seeds (1, 42 and 2024), and we averaged the last several checkpoints towards the robustness. The detail of Basc/Decp/Wiin Wide configurations is as follows:
Base/Deep Model. The hidden size of self-attention was 512, and the dimension of the inner-layer in FFN was 2, 048. used 8 heads for attention. For training, we set all dropout to 0.1 as default, including residual dropout, attention dropout, ReLU dropout. Lahel smoothing fls = 0.1 was applied to enhance the generation ability of the model. For deep models, we only enlarged the encoder depth considering the inference speed.
Wide (or Big) Model. We used the same architecture as Transformer- Base but with larger hidden layer size 1, 024, more attention heads (16), and larger feed forward inner-layer (4, 096 dimensions). The residual dropout was set to 0.3 for the En-De task and 0.1 for the En-Fr Fr task.
C.2 Abstractive Summarization
We also tested the models' ability to process long sequences on the Call summarization task [42, 18]. The preprocessed method was the same as in [46]. We used shared BPE with 30K operations, resulting in vocabulary of 32, 580 entries. The evaluation metric was Fl-Rouge [30] (Rouge-1, Rouge-2 and Rouge-L).
'hups://object http://bbbccc.ppou..oo esc. wwweff/OPUS-0000 100vv1 10tpp/////u/////oo Woopus-100-oono wat.oorpus-l....rarr.... S EUCceee Www.FFUucse.mixd--nmuee mumee-- ++moothh #smooth.expttok. 13a+version.2.3.
'The default setting which uses reference-basee regression model built top of XLM-R R (Large).
16

---

29.0
Transfumerr
Evolved Transformer
Deliihh
2-order Gated
BLEU
2-order EMA,
28.0
4-order EMA
order PClomeer
27.0
30
40
50
60
Model Parameters
Figure 3: The comparison of BLEU as well as model capacities and training costs against previous state-of-the-art deep transformers.
C.3 Language Modeling
Datasets mentioned above, the truncation error analysis conducted on Penn Treebank [41], which a widely-used language model dataset. It contains 88K, 3, 370 and 3, 761 sentences for training, validation and test. The vocabulary size is 10K. We set the layer depth of the language model to or 2 to make a fair comparison. Assuming the layer depth is 1, then the loss between the model output and the true label could be regarded the truncation error. In this way, we alleviate the influence of the error accumulation across different layers. Apart from PTB, we also evaluate our approach on another widely acknowledged language modeling dataset, Wikitext- 103 dataset [40], which is the largest available word-level language modeling benchmark with long-term dependency. WikiText- 103 consists of 103M training tokens from 28K articles Wikipedia. and the average length of tokens per article is about 3.6K. The data can be easily obtained and preprocessed following Baevski [3]'s work.
Setups For the PTB dataset, we used the transformer. the filter size of the FFN is 2, 048. All the dropout rates are 0.1, including the residual dropout, attention dropout and ReLU dropout. Each model validation set when the epoch 10. Then the validation PPL began increase, though the training PPL still declining. is due to the small model capacity. The warmup step was 2, 000 and the batch size was 4, 096. The max learning rate was set to 0.0007. For the Wikitext-103 dataset, all models were based on the original open-source Fairseq toolkit, and the corresponding configuration is transformer 1m baevaki wiki10s It inherits the configuration of transformer __g with 1, 024 hidden size and 096 filter size. Besides these common hyper-panee parammetrsssss, it adopts the adaptive input representation to reduce the embedding matrix by decreasing the embedding size of low- frefrrquency words or sub-words. This is also the most common choice when building large-scale language models.
bass base configuration, whose hidden size 512, and
trained up to 20 epochs, while most models arrived at convergence the
C.4 Language Understanding
Our proposed PCformer by continue training the BERT model using the provided pretrained model, BERT- Large-cased, And evaluated the General Language Understanding Evaluation (GLUE) [60]. Concretely, the GLUE benchmark has different text classification regression tasks including MNLI, MRPC, QNLI, QQP, RTE, SST-2, SST-B, and CoL.A.
C.5 LM Evaluation Harness
also evaluate our PCformer on wide range of downstream tasks covering common-sense reasoning and question-answering, including LAMBADA (LMB. [47]), PIQA [4], SciQ [64]. Winograde (Wino. [52]), HellaaSag (Hella. [68]). and ARC-challenge [9]. We mainly evaluate language models via zero-shot evaluation. We randomly sampled 100B tokens from Slimpajama dataset for training.
D More Analyses
Parameter Efficiency Figure summaries results several efficient Transformer variants, including Lite Transformer [66], DeLight [39], a light version of the Evolved Transformer [56], and our PCformer. It is clear to see that PCformer is significantly more parameter efficient than others. We make detailed comparisons of PCformer within different hyper- parameters, comprising of hidden size and model depth. Concretely, RK2-block
17

---

6.20
3.50
Trinsfommer 6L
3.45
Transformer OL
Trainin PPL
5.80
Trmnfoomes1111 12.
3.40
Transfonnte 12
5.40
PR Ma 121
PCfomme--a
Validatioo
3.30
PCTomms-00:
PCRomes-111 122
3.25
6.00
3.20
4.60
3.15
12
16
20
24
28
32
36
40
8
12
16
20
24
28
32
36
40
Epoch
Epoch
(a) En-De
3.10
Training PPL
3.30
3.50
Transormer-&l 6l
Transfomer-112 121.
Validation PPL
2,90
Transformer 6L
Transforme-112
PCfommer 6L
2.70
PCfognge 6k
2.90
PC
PC
2.70
2.00
2.50
2:30
2
10
14
14
Epoch
Epoch
(b) En-Fr
Training PPL
4.60
Transfommes
Transfolmer-244.
Validation PPL
4.80
4.70
Trandfemer-
Transfoemer-241
4.30
Polommer 1g
4.60
PCfomme
PCfomler-341.
4.00
PC
4.40
3.70
3.40
4.10
10
12
16
10
12
16
Epoch
Epoch
(c) OPUS
Figure 4: The comparison of training and validation PPL base and wide models.
with gated fusion coefficient learning strategy delivers stronger performance than DeLight within the same model parameters. And is on par with DeLight in terms of BLEU, but having 9M fewer parameters. Moreover, as expected, our newly proposed EMA coefficient learning method slightly improves the performance of gated fusion almost all scenarios. A notable bonus it brought higher-order solutions, e.g., RKo-block is superior RK2-block with the help of EMA. It may offer a new choice for deploying NMT systems on edge devices.
Training and Validation Perplexity Apart from the BLEU scores, also compare our methods with baselines regarding perplexity on both training and validation scts. Figure plots the training and validation PPL curves of the PCformer (RK2-block (EMA Pre-Cor)) and the baseline on two representative translation tasks. All models were in big configurations for more convincing conclusions. The proposed RK2- block with EMA coefficient learning and Predictor-Corree within the configurations. More specifically, 12-layer model outperforms the 6-layer one. For both the En-De and En-Fr tasks, we observed that our 6-layer method even shows lower PPLs than a 12- laylryer Transformer. This phenomenon is more evident in the WMT En-Fr task, which again demonstrates the high parameter efficiency of our PCformer.
framework delivers much lower training and validation PPLs
method still benefits from increasing model depth, as the
Visualization of the Coefficient Learning Procedure We also collect the learning process of learnable coefficients during training. As we discussed above that the inspiration of EMA coefficient learning method comes from the prior that the most current approximation is more precise. This assumption has already been clarified by the comparisons of truncation errors. Here, we want figure removing the constraint. To achieve this goal, we set all coefficients to be independently initialized by the mean of 1, c.g., 2-order block with y1 = 0.5, 72 = 0.5. Figure 5 plots the learning curves of RK2- blocklock and RK4-block within these two learning strategies. As we can see that, for the independent initialization scenarios, coefficients vary dramatically within the first several epachs, and then show convergence in small range. Both two figures show that the contribution of the most current coefficient larger than others.
how cocfficients Iearned
To our surprise, 72 RK22block (Independent) converges to large than and oppositely, ? is even goes to a negative value. Meanwhile, " in the RK4-block (Independent) shows negative impact to the final solution, but the other 3 coefficients vary within our expectation. We notice the order relation among these 4 coefficients consistent with the numerical suggested coefficients in the Adams- Bashfoothho method (-9/44, 37/24, 59/24,
18

---

1.00
72
0.80
non
Coeflicienss
0.60
Coefficionts
1.00
0.10
0.60
0.20
0.00
0.20
0.20
13
21
25
1
21
25
Epuch
Epuch
(a) RK4-block (Independent), BLEU=29.00
(b) RK2-block (Independent), BLEU=28.78
1.00
1.00
71
72
78
74
71
O-
72
0.80
0.80
Coefiiients
Coefficient
000033060000
0.60
0.60
0.40
0.40
0.20
0.20
0000
+004
0.00
9
13
17
21
25
5
13
17
21
25
Epoch
Epoch
(c) RK4-block (EMA), BLEU-29.31
(d) RK2-block (EMA), BLEU-29.11
Figure The coefficient learning curves of independent initialization and EMA oin both 2-order and 4-order scenarios. The experiments conducted on WMT En-De.
55/24). This indicates the underlying relationship between numerical analysis neural network optimization. While, after employing our EMA method, coefficients are optimized along our expected direction, and these models are empirically better than those without constraints.
Results on Time-Series Forecasting We also followed the reviewer's suggestion to evaluate PCformer on time-series forecasting tasks. We selected 10 multivariate datasets from UEA Time Series Classification Archive following the setting and the codehase provided by Flowformer [65]. Thus we choose the Flowformer the baseline, which is also strong model on these testsets. For details, we build the PCformer upon Flowformer and report the 2-order predictor and Euler corrector as the training data is very small. Also, we use RK-Norm to avoid the model suffering from the overfitting problem as the authors of Flowformer trained their models upon 100 epochs (or 400 epoch some tasks.). The results evaluated the best accuracy. We can that PCformer can beat the Flowformer by average score on 10 testsets, which demonstrates the effectiveness on serics forecasting tasks.
Table 14: Comparison of Flowformer and PCformer different datasets.
Dataset
Flowformer
PCformer
ETHANOLLCONCENTTAATIO
30.3
33,9
FACEDETECTION
67.0
68.2
HANDWRITING
29.1
33.5
HEARTBEAT
77.0
78.5
JAPANESE
98.4
99.2
PEMS-SF
87.2
87.9
SELFREGULATIONSCP
89.0
92.2
SELFREGULLTTIONSCP2
55.0
56.1
SPOKENAARAIIDIIGITS
98.0
100.0
UWAVEGESTURELIIRAAR
85.3
86.3
Average Score
71.6
73.6
19

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have highlighted our contributions in Section 1. We proposed a predictor-correcto paradigm for Transformer architecture design, and also an EMA coefficient learning algorithm to augment the learning of high-order predictor. Experimental results have also shown the superiority of PCformer.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Please refer to Appendix B
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings. model well- sell-specification
asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice
what implications would The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect on the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with datasct size.
applicable, the authors should discuss possible limitations of their approach to address problems privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
20

---

All the theorems, formulas, and proofs in the paper should be numbered and cross cross-reference.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can cither appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data provided not)?
Question:
Answer: [Yes]
Justification: We have summarized all details including the datasets, the model configurations and training details in the appendix. Please refer to Section C of the Appendix. And we also make clear algorithm to illustrate the detailed computation flow of the proposed PCformer. And we will release the code after re-organizing the codebase, to provide detailed scripts for reproduce results.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice,
example, if the contrihution the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., the case of a large language model), releasing model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., . a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g.. with open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-soure, models, it may be that access to the model is limited in some way (e.g.. registered users). but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: will open-source the code as soon as we collect our scripts an easy way to reproduce our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines
gut.dess/odeeuuuniesssoooooooo for more details.
21

---

While we encourage the release of code and data, we understand that this might not be possible. "No" acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g.. for a new open-soure benchmark).
The instructions should contain the exact command and environment needed the results. See the NeurlPS code and data submission guidelines (httpe: / /nwps.ppp.... cc cc/pulic//
reproduce
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental Setting/Detain
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: have summarized the training and test details in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have conducted experiments times and reported the average result. The seeds are 1, 42, 2024, respectively.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 96% CI. if the hypothesis of Normality of errors
a 2-sigma error than state that they not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric emror bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Answer: [Yes]
22

---

Justification: We mainly conducted experiments A100 and H800, single machine with 8 GPUs. Each GPU has 80G memory. When training PCformer IB and 3B models, used 128 11800 gpus. Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code Ethics https: /ttprr//eeuupppeeeeee ctcplpp..ccpputiee cc/ppbliccEhiicccutellli...
Answer: [Yes]
Justification:
Guidelincs:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special considcration duc laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts the work performed?
Answer: [Yes]
Justification: Please refer to Appendix
Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g.. deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example. it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
23

---

The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should he released with necessary safeguards to allow for controlled use of the model, for example hy requiring that users adhere to usage guidelines restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are creators or original owners assets (e.g.. code, data, models), used the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: This paper only uses publicly available open- sourcs code and datasets, and the authors have properly credited the original creators and respected the licenses and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or datasct.
The authors should state which version of the asset is used and, if possible, include a URL.
name of license (e.g.. CC-BY 4.0) should be included each asset. scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi wwwwwthcon thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should he provided.
If this information is not available online, the authors are encouraged to reach out to the asset's
creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: publication.
We will provide detailed documentation in our open source code repository after
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, rememher to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer means that paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
24

---

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval. you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
initial submissions, ble), such as institution conducting the review.
include any information that would break anonymity applica-
25


Reviews:
[{"review_id": "bK6lV5yFm2", "reviewer": "Reviewer_1pKC", "summary": "The paper presents advancements in Transformer architecture to minimize errors in approximating solutions to Ordinary Differential Equations (ODEs). \n\nThe contributions are:\n- Introducing a learning paradigm with a high-order predictor and multistep corrector to reduce truncation errors.\n- Proposing an exponential moving average-based method to enhance the predictor's learning ability and stability by replacing constant coefficients with dynamic ones.\n- Demonstrating superior performance across various benchmarks, including machine translation, abstractive summarization, language modeling, and natural language understanding, achieving notable improvements in BLEU scores and parameter efficiency.\n\nThe work shows improvements in translation tasks and highlights the general applicability of the proposed methods across different natural language processing domains.", "questions": "Can you provide more details on the computational cost of the proposed predictor-corrector framework and EMA coefficient learning method? Specifically, how do these methods impact training time and resource utilization compared to traditional Transformer models?\n\nHow does the proposed method scale with increasing model size and dataset complexity? Have you encountered any challenges in scaling up your approach, and if so, how did you address them?", "limitations": "Provide a detailed analysis of the computational complexity and resource requirements of the proposed methods. This should include a comparison with standard Transformer models and an explanation of any trade-offs between performance improvements and computational costs.\n\nDiscuss the scalability of the proposed methods in more detail. Explain any challenges encountered when scaling up to larger datasets or models and how these were addressed. Provide insights into potential limitations in terms of scalability and how future work could overcome these challenges.", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper introduces a new predictor-corrector framework within Transformer architectures, which is a fresh and innovative approach to addressing errors in approximating solutions to ODEs. The integration of a high-order predictor and multistep corrector, combined with an exponential moving average (EMA) coefficient learning method, represents a creative combination of established numerical analysis techniques with modern neural network architectures. \n\nThe paper successfully applies the method across various natural language processing tasks. The paper is well-structured and clearly written, making complex concepts accessible.", "weaknesses": "The complexity of implementing these methods might be a barrier for practical adoption. The paper could benefit from providing more detailed guidelines or code to facilitate easier implementation and replication of the results.\n\nThe experiments primarily focus on natural language processing tasks. While the results are impressive, it remains unclear how well the proposed methods generalize to other domains, such as time series forecasting. Including preliminary results or discussions on the potential applicability to these other areas could strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your recognition on our writting and the style to present our idea. We would like to answer your questions as follows:\n\n> W1: The complexity of implementing these methods might be a barrier for practical adoption. The paper could benefit from providing more detailed guidelines or code to facilitate easier implementation and replication of the results.\n\nWe open-source our core code at https://anonymous.4open.science/r/Neurips-PCformer/. This allows others to easily reproduce our results and apply PCformer to additional benchmarks. We will also release the complete codebase after finalizing the README.md to facilitate better reproducibility.\n\n\n> W2: The experiments primarily focus on natural language processing tasks. While the results are impressive, it remains unclear how well the proposed methods generalize to other domains, such as time series forecasting. Including preliminary results or discussions on the potential applicability to these other areas could strengthen the paper.\n\nWe mainly evaluate our PCformer on NLP tasks, including both the natural language understanding and natural language generation. While, our method is quite general that it could be applied to other Transformer variants in other areas, e.g., Swin-Transformer in computer vision. For example, our PCformer can also improve Swin-Transformer in tiny and base configurations. The results were conducted on 224*224 image size, and 2-order predictor with a multistep corrector.\n\n\n| Model                  | Params. | Image size | Top1-acc |\n| ---------------------- | ------- | ---------- | -------- |\n| Swin-Transformer(tiny) | 29M     | 224*224    | 81.3     |\n| PCformer(tiny)         | 29M     | 224*224    | 82.0     |\n| Swin-Transformer(base) | 88M     | 224*224    | 83.5     |\n| PCformer(base)         | 88M     | 224*224    | 84.0     |\n\n\nBeside this, we also follow your suggestion to evaluate PCformer on time-series forecasting tasks. We select 10 multivariate datasets from UEA Time Series Classification Archive following the setting and the codebase provided by Flowformer [1]. Thus we choose the Flowformer as the baseline, which is also a strong model on these testsets. For the details, we build the PCformer upon Flowformer and report the 2-order predictor and Euler corrector as the training data is very small. Also, we use RK-Norm to avoid the model suffering from the overfitting problem as the authors of Flowformer trained their models upon to 100 epochs (or even 400 epoch on some tasks.). The results are evaluated by the best accuracy. We can see that PCformer can beat the Flowformer by 2 average score on 10 testsets, which demonstrates the effectiveness on time-series forecasting tasks. We hope these results can address your concern, and we'd like to add them into the updated version of this paper.\n\n\n| Dataset              | Flowformer | PCformer |\n| -------------------- | ---------- | -------- |\n| EthanolConcentration | 30.3       | 33.9     |\n| FaceDetection        | 67.0       | 68.2     |\n| HandWriting          | 29.1       | 33.5     |\n| Heartheat            | 77.0       | 78.5     |\n| JapaneseVowels       | 98.4       | 99.2     |\n| PEMS-SF              | 87.2       | 87.9     |\n| SelfRegulationSCP1   | 89.0       | 92.2     |\n| SelfRegulationSCP2   | 55.0       | 56.1     |\n| SpokenARABICDIGITS   | 98.0       | 100.0    |\n| UWAVEGESTURELIBRARY  | 85.3       | 86.3     |\n| Average Score        | 71.6       | 73.6     |\n\n[1] Flowformer: Linearizing Transformers with Conservation Flows, ICML 2022.\n\n> Q1: Can you provide more details on the computational cost of the proposed predictor-corrector framework and EMA coefficient learning method? Specifically, how do these methods impact training time and resource utilization compared to traditional Transformer models?\n\nOur proposed PCformer is parameter-efficient as we share the parameters between the predictor and the corrector, especially for the F functions in the high-order predictor. Please refer to the general response W1 for more details.\n\n\n> Q2: How does the proposed method scale with increasing model size and dataset complexity? Have you encountered any challenges in scaling up your approach, and if so, how did you address them?\n\nOur PCformer could be easily scaled from both the model capacity and training tokens. We summarize the results of larger model capacity (from 340M to 1.3B) and training tokens (from 6B to 100B) in the general response W2, where PCformer can beat LLama-like models in all scenarios. Additionally, the results in Table 3 have already shown the superiority of PCformer in a 1.2B setting on OPUS multilingual translation tasks.\n\n We think there is only a major challenge: the computational overhead of PCformer is somewhat higher than that of the vanilla Transformer (e.g., LLama models). This limitation has been discussed in our paper. We are actively working to address the training and inference overhead in our ongoing research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9okyLYGHXb", "reviewer": "Reviewer_Rc6M", "summary": "This paper presents an approach to improve the performance of Transformer models for conditional natural language generation (machine translation and summarization).   The authors introduce a predictor-corrector framework, inspired by numerical methods for\nsolving ordinary differential equations (ODEs), to enhance the accuracy and\nstability of the  models. The proposed model is evaluated against standard Transformer models on multiple benchmark datasets. The presented results show improvements in prediction accuracy and efficiency when compared to standard models.", "questions": "- How does the predictor-corrector framework perform in comparison to\nstate-of-the-art pretrained Transformer models?\n- Do you think that all the 72 references are necessary? \n- The quotation of Newell 59 is maybe a bit too much. Don't you think?", "limitations": "The authors should discuss the potential computational overhead introduced by the predictor-corrector framework.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The application of the predictor-corrector paradigm to Transformer models is innovative and offers a new perspective, extending the ODE transformer proposed in (Li et al., 31).\n\nThe integration of high-order ODE solutions into the Transformer architecture is a novel contribution. The authors use EMA (Exponential\nMoving Average) coefficient learning to enhance training stability.\n\nThe experimental results look comprehensive, with evaluations on multiple benchmark datasets demonstrating the efficacy of the proposed model.\n\nThe paper provides a detailed description of the methodology.\n\nThe paper is well-organized and clearly written, with each section leading to the next.", "weaknesses": "The paper does not compare its results with state-of-the-art  models, which could have provided a more comprehensive evaluation of its effectiveness.  For instance for WMT14 En to FR, we can reach 43 in BLEU, instead of 41 for Attention is all you need.  Moreover, the difference with ODE Transformer is very limited. \n\n\nThe theoretical justification for the predictor-corrector framework could be more detailed, particularly in explaining how it relates to and improves upon existing methods. \n\nThe computational overhead induced by the proposed method is only lightly discussed in the Appendix and would require more in depth discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive advice and we think all the conerns would be well addressed in our improved version. We would like to address your concerns as follows:\n\n> W1: The paper does not compare its results with state-of-the-art models, which could have provided a more comprehensive evaluation of its effectiveness.\n\nThank you for pointing out this issue. Due to page limitations, we focused our comparisons on the most closely related work, particularly those designed based on numerical methods, such as Transformer-DLCL, MacroNet, and ODE-Transformer. Notably, the ODE Transformer is a strong baseline on the WMT En-De and En-Fr benchmarks, used to achieving state-of-the-art results in these tasks. In addition, for the OPUS task, we compared PCformer with SoTA models like DeepNet and BranchNorm, which are leading models on this benchmark. To the best of our knowledge, PCformer indeed achieves SoTA results on WMT En-De (without pretrained models), En-Fr, En-Ro, and OPUS in these machine translation tasks. As for other tasks, such as summarization, PCformer shows competitive results compared to baselines learning from scratch. The current SoTA models in these tasks are primarily finetuned versions of BART and other advanced pretrained models. We will aim to make a fair comparison with these models in our next version. Thank you for your understanding and your valuable feedback.\n\n\n> W2: The theoretical justification for the predictor-corrector framework could be more detailed, particularly in explaining how it relates to and improves upon existing methods.\n\n- Compared to existing high-order methods such as Runge-Kutta (ODE Transformer), our PCformer leverages a predictor-corrector paramdigm to estimate intermediate approximations more accurately. Notably, we introduced an EMA-based coefficient learning method to enhance the robustness and stability of our high-order predictor, and the corrector then applies a multistep method to further reduce truncation errors. \n\n- Theoretical Improvements: Truncation error is a crucial factor in numerical solutions. In Section 3.1.2, we provide a theoretical analysis demonstrating that higher-order intermediate approximations tend to be more accurate, leading to more reliable final results. And in Table 8, we show that PCformer can achieve lower ppl (analogous to truncation errors) than the 1-step and high-order methods.\n\n- Practical Superiority: As demonstrated in our experimental results section, our framework significantly outperforms existing models, including the robust 3.8B DeepNet, on several large-scale datasets while using only one-third of the parameters.\n\n\n> W3: The computational overhead induced by the proposed method is only lightly discussed in the Appendix and would require more in depth discussion.\n\nThank you for your advice. We regret that we were unable to include this discussion in the main body of this paper due to time constraints. We would like to provide the following summary on the complexity and computational overhead induced by our proposed method, the details please refer to the general response W1!\n\nIn a nutshell, PCformer only introduces slight inference overhead as we mainly applied the predictor-corrector in the encoder side, which takes a very small amount of computation in the process of auto-regressive decoding.\n\n\n> Q1: How does the predictor-corrector framework perform in comparison to state-of-the-art pretrained Transformer models?\n\nWe have conducted extensive experiments across several benchmarks. Our proposed PCformer not only significantly outperforms the vanilla Transformer but also shows a substantial improvement over the strong ODE Transformer. We believe your question may be directed towards understanding how PCformer performs in comparison to other large pretrained language models (LLMs). If our interpretation is incorrect, please let us know, and we will address the specifics accordingly.\n\nDue to the limited number of response characters, please refer to the general response W2 for the new proposed results on LM harness evaluation. We can see that PCformer beat the Transformer baseline across all settings, we will further scale the model to a much larger capactiy in the future, e.g. 7B/8B model to make a fair comparison with LLAMA models.  While training even larger LMs from scratch is costly and time-consuming, it is a promising direction for further research.\n\n> Q2: Do you think that all the 72 references are necessary?\n\nWe have carefully reviewed all the references cited in our work and believe that almost all reference are necessary. The large number of references is due to the comprehensive nature of our study, which spans five distinct tasks: machine translation, abstractive summarization, language modeling, natural language understanding, and LM harness evaluation. For each task, we report results on several widely used benchmarks, such as WMT En-De, En-Fr, En-Ro, OPUS, nine test sets for BERT, and eight test sets for LM harness evaluation. To ensure a fair comparison and to validate our results, it was essential to cite the datasets and the results from related work. Consequently, the extensive referencing reflects the breadth and depth of our study rather than an excess. We will futher check the details to avoid unnecessary reference included!\n\n> Q3: The quotation of Newell 59 is maybe a bit too much. Don't you think?\n\nThank you for pointing this out. Our intention was to illustrate that the predictor-corrector paradigm aligns with the seminal ideas of Newell, as also mentioned in the Tree-of-Thought paper. Specifically, the predictor first provides an initial estimate. This estimate is then refined by the corrector for greater accuracy, mirroring the problem-solving process proposed by Newell. We appreciate your feedback and will carefully reconsider and reorganize this section in the next version of our manuscript to ensure it is succinct and relevant.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qj56b0ZQc6", "reviewer": "Reviewer_MB44", "summary": "This paper takes inspiration from established high-order approaches in numerical analysis for solving differential equations to improve the architectural design of Transformers. Prior work has shown that residual networks can be seen as discrete approximations of Ordinary Differential Equations (ODE) and explored methods to improve the quality of the solution. Specifically, this paper introduces a predictor-corrector learning framework to minimize approximation errors and an exponential moving average-based coefficient learning method. These advancements are used within Transformer architectures that are evaluated on several tasks such as translation, summarization, language modeling and language understand tasks, improving over standard and previous ODE-based Transformers.", "questions": "- Q1. Where does the parameter efficiency of the model stems from and how a fair comparison with other models was ensured? In Appendix D, there is a section that discusses this but the comparison does not seem to be apples-to-apples (i.e. same config and #params) and whether the result holds in all the tasks in the experiment section.\n- Q2. The method description wasn't entirely clear to me and I have the following questions: \n   - The parameterization of $\\mathcal{F}(P_{t+1}, \\theta_t)$ is not clear from the textual description. What kind of transformation is used here? \n   - How are the coefficients are parameterized exactly? Do you parameterize a single coefficient for each order for an $n$-order predictor or it is the same coefficient shared?  \n    - The text mentions that a larger weight is assigned ($a=0.5$) to the estimated $F_{t+1}$, is this value fixed or you have experimented with different values? It would be useful to clarify which coefficients among $\\alpha$ and $\\gamma$ are hyper-parameters and which ones are learned.\n- Q3. Is the observed improvement worth the the additional computational cost for obtaining a better approximation? It would be useful to show what is the difference compared to a standard or a simpler ODE Transformer. \n- Q4. Do you mean \"ROUGE results\" instead of \"Rough results\" in Table 4? In the same table, what is the number of parameters used by each model? \n- Q5. What is the size of the models in Table 8 and is it the same for all variants? \n- Q6. In the ablation experiment and other results, there is lack of comparison with a simple 1-order single/multi-step method or high-order 1-step method. Is the proposed method substantially better than them?\n- Q7: Did you perform continued training of BERT directly on language understanding tasks or on some pre-training data first in Table 7? Please also report the number of parameters for both models and the exact configuration for PCformer.", "limitations": "The limitation section covers the shortcomings when the proposed methods are applied to different model designs but lacks discussion about the experiment shortcomings related to model size, computational cost, and comparison with simpler methods in a controlled setting.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well motivated and clearly described for the most part. It provides sufficient background to make the reading self contained and explain what is the novel contribution of the work. It's also well situated and compared to prior work on ODE Transformers as it discusses prior first order and high-order methods with a single step. \n- Builds on top of predictor-corrector methods from numerical analysis and extending them to improve their stability when training ODE Transformers. The key novelty lies in the selection of the predictor and corrector methods, and the proposal to use exponential moving average method to learn the coefficients in high-order methods instead of using constant values. \n- The proposed method uses a high-order method as predictor and a multi-step method as corrector and aims to provide a better approximation to the implicit ODE problem in Transformers than previous studies. \n- Presents empirical results on a variety of tasks are better than the original and previous ODE Transformers. This makes the work of interest to the researchers working in ODE Transformers.", "weaknesses": "- W1. There is lack of emphasis in the experiments on the computational overhead introduced by the high-order predictor and multi-step methods. Achieving better quality is not sufficient for adoption in practical settings. I'd suggest quantifying the training and inference cost compared to standard and ODE Transformers.\n- W2. In terms of parameter efficiency, the proposed model achieves better performance with 1/3 of parameters only on one of the examined datasets. It's not clear if this is by chance or if the result holds on other settings. It would be useful to report performance with a smaller model size on the rest of the datasets to better establish the underlying claim. \n- W3. The impact to research communities beyond the ones focusing on ODE transformers is somewhat limited because the results are with relatively small model sizes. Showcasing that the results hold on larger architectures such as 7B Mistral would make the results more convincing. \n- W4. The improvement of the proposed multi-step high-order method compared alternative predictor-corrector methods is small (Table 10) and the benefit compared to simpler first-order methods from the predictor-only paradigm in a controlled setting is not provided (similar to the ablation in Table 10).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thanks for your recognition for our motivation and the organization of our method.\nWe also appreciate for your constructive feedback towards the shortcomes of the current manuscript, and we think all the concerns would be well addressed in our improved version. Here, we would like to show the details of your main concerns:\n\n> W1: Computational overhead: e.g., training and inference cost.\n\nWe hope you can refer to the details of the general response W1. We report the training and inference cost of machine translation tasks and give a detailed analysis of large language models. \n\n\n> W2: Parameter efficiency: performance with a smaller model size on the rest of the datasets.\n\n- Besides OPUS, actually, we have already done simialr experiments in Figure 3 (Appendix), where a 28M PCformer can beat other models larger than it. \n- In our general response W2 for our newly proposed results on LLMs, we found that **Pcformer can beat the vanilla Transformer by only training on half of the training data! (1.3B models and 50B vs 100B tokens)**; More excited, our 340M PCformer (16B tokens) can achieve comparable results with 1.3B models with only 1/4 parameters!\n- For summarization tasks, a baseline with 12/18 encoder-layer configuration achieves RG-1, RG-2, and RG-L scores of 41.18, 18.23, 38.03 and 41.33, 18.31, 38.20, respectively. In comparison, our 2-order method yields scores of 41.96, 18.99, 38.74, **outperforming the baseline while utilizing only 1/3 to 1/2 of the layers.** \n\nWe'll add more comprehensive comparisons next.\n\n> W3: Results on larger models such as Mistral 7B.\n\n- We appreciate your suggestion and completely agree. Our current results demonstrate that our model performs exceptionally well on smaller model sizes (less than 1B), indicating that PCformer has significant potential to scale up to larger model sizes. Given the interest from reviewers in seeing how PCformer performs in LLM evaluations, we have scaled up the model size from 340M to 1.3B parameters and increased the training data from 6B tokens to 100B tokens. These efforts have pushed the limits of our current computational resources. The summarized results are provided in the general response W2. We hope these results address your concerns.\n\n- We recognize the importance of showcasing results on even larger architectures like the 7B Mistral. However, achieving this would require additional computational resources. At present, we plan to train a PCformer based on the Mistral checkpoints and restart the training process with significantly less data compared to what was originally consumed in their pretraining. We are committed to conducting these more challenging experiments as soon as we have access to the necessary computational resources. Thank you for your valuable feedback.\n\n\n> W4: More details of the ablation (Table 10)\n\n\nVery good suggestions. Here we add the simpler first-order methods and ODE Transformer for each setting. The results are below:\n\n| #    | Predictor            | Corrector             | En-De | En-Fr | En-Ro | OPUS |\n| ---- | -------------------- | --------------------- | ----- | ----- | ----- | ---- |\n| 1    | First-order baseline | -                     | 29.91 | 43.22 | 34.20 | 31.5 |\n| 2    | ODE Transformer      | -                     | 30.77 | 43.96 | 35.28 | 32.3 |\n| 3    | RK2-block with EMA   | Multistep Method      | 30.70 | 44.27 | 35.55 | 33.7 |\n| 4    | RK2-block with EMA   | Backward Euler Method | 30.95 | 43.68 | 36.00 | 33.2 |\n| 5    | Multistep Method     | Multi-step Method     | 30.30 | 43.92 | 35.30 | 33.0 |\n| 6    | Multistep Method     | RK2-block with EMA    | 29.78 | 42.68 | 34.40 | 32.5 |\n| 7    | Multistep Method     | Backward Euler Method | 30.30 | 43.62 | 35.27 | 32.8 |\n\nWe observe that all settings, except for the Multistep Predictor and High-order Corrector, show consistent BLEU improvements over the baseline (#1). This phenomenon is explained in our current manuscript: the predictor needs to be concise enough, otherwise, the initial value may cause the subsequent corrector computations to deviate. Additionally, both multistep and backward Eulere correctors show substantial improvements to ODE Transformer. Our proposed PCformer is a framework that you can choose a proper corrector according to the training date complexity, and RK2-block with EMA as the predictor always behaves well. We hope the results here can address your concern.\n\n> Q1: About fair comparison in experiments.\n\nWe mainly want to compare the PCformer with the vanilla Transformer in two settings. \n- Firstly, as we only introduce quite small parameters, just a 1D tensor for EMA coefficient learning and several layernorm to achieve RK-Norm, we compare these two models in similar (near the same) parameters. The PCformer beats the vanilla Transformer by a quite large margin nearly in all scenarios.\n- Secondly, we can compare them in the same FLOPs, for example, a 6-layer PCformer with much fewer parameters can beat a 12-layer vanilla Transformer, meanwhile fewer computations.\n- Thirdly, for LLMs, we find that  PCformer also beats Transformer even the former trained with 50B tokens and the latter trained with 100B tokens. As we all know, increasing the training data is the most effective way and also the simplest way to improve performance. But at this serve setting, PCformer still behaves strongly.\n\nDue to the limited time and page, we will add more comparisons in all experiments for a much stronger claim.\n\n> Q2.1 The parameterization of  is not clear from the textual description. What kind of transformation is used here?\n\nAs we emphasized that $\\mathcal{F}_(P_{t+1},\\theta_t)$ is the function to compute the derivate upon the input. Thus, either a self-attention network or an FFN, or even the whole encoder block could be regarded as an F function, also they could be seen functions in different granularities. In this work, for a fair comparison with ODE Transformer, we choose the whole block as the function.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x0v1f76F1h", "reviewer": "Reviewer_hvMK", "summary": "This work presents a transformer architecture inspired by predictor-corrector methods for solving ODE problems. It adopts the Adams-Bashforth-Moulton method and utilizes an exponential moving average (EMA) method to compose the predictor, discussing two correctors (the EMA-based and the simple backward Euler method). Without the background of ODE, the proposed method could be seen as a special case of a transformer with cross-layer connections. Consider that the residual connection only links two layers, while the proposed method links multiple layers in a high-order way.", "questions": "What's the efficiency of the proposed method, such as inference time and GPU memory consumption? I saw that inference efficiency has been mentioned in the limitations; it's better to have quantitative results and also memory states.", "limitations": "Agree with the limitation section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work provides theoretical results to enhance the connections between residual connections and the ODE solver, providing an interesting perspective on viewing cross-layer connections within neural networks.\n\nUsing EMA as a high-order predictor is a simple and flexible solution, and the experiments demonstrate its effectiveness.", "weaknesses": "The experiments are insufficient and out-of-date. As the main contribution is the new transformer architecture, scaling law style experiments and testing on modern LLMs' benchmarks are recommended. I saw that Appendix D has some results, but it's better to have a formal experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Following your suggestion, we have scaled up our large language model (LLM) experiments, focusing on how our PCformer performs in the LLM setting. We aim to demonstrate the capabilities of PCformer from both model capacity and data volume perspectives. The updated results are as follows:\n\n| Model(Params & Tokens)     | Wiki.(ppl) | LMB.(ppl) | LMB.     | PIQA     | Hella.   | SCIQ     | ARC-c    | Winograde| Avg.     |\n| -------------------------- | ---------- | --------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| Transformer++ (340M & 6B)  | 38.5       | 96.1      | 21.4     | 60.3     | 29.1     | 69.2     | 21.5     | 50.4     | 41.9     |\n| PCformer (340M & 6B)       | 35.3       | 78.8      | 23.6     | 61.6     | 30.1     | 71.6     | 22.9     | 51.8     | 43.6     |\n| Transformer++ (340M & 16B) | 28.3       | 65.3      | 29.8     | 63.2     | 33.9     | 73.2     | 23.1     | 51.4     | 45.8     |\n| PCformer (340M & 16B)      | 25.6       | 39.7      | 34.5     | 65.2     | 36.9     | 79.6     | 23.2     | 52.2     | 48.6     |\n| Transformer++ (1.3B & 16B) | 23.8       | 26.2      | 37.3     | 65.7     | 37.6     | 78.6     | 23.7     | 51.5     | 49.0     |\n| PCformer (1.3B & 16B)      | 20.9       | 23.2      | 42.5     | 68.3     | 43.4     | 81.5     | 25.1     | 52.4     | 52.2     |\n| Transformer++(1.3B & 100B) | 16.3       | 11.8      | 51.6     | 71.0     | 51.7     | 86.7     | 28.1     | 54.6     | 57.2     |\n| PCformer (1.3B & 50B)      | 16.2       | 9.38      | 55.1     | 71.9     | 54.8     | 88.6     | 29.6     | 57.2     | 59.5     |\n| PCformer (1.3B & 100B)     | **14.0**   | **7.46**  | **59.6** | **73.8** | **60.0** | **90.7** | **31.7** | **61.7** | **62.9** |\n\n\n- In our initial submission, we reported results on a 340M parameter configuration trained with 6B Slimpajama data. Here we included additional perplexity metrics on Wikitext and Lambada to provide more comprehensive results. Furthermore, for more convincing results, we scaled the model size from 340M to 1.3B parameters, which is the maximum size feasible within the limited time. Scaling to larger models requires additional engineering efforts, such as implementing tensor or pipeline parallelism for robust and efficient training (using Megatron codebase). We plan to report results on 7B or larger models in our next version and we believe PCformer can indeed perform strongly on those settings.\n\n- For the data, we scaled from 6B to 16B and 100B tokens. Our findings show that with more training data, the average performance improves significantly. This demonstrates that our model benefits from increased data size, without experiencing diminishing returns. This indicates that model bias persists and plays a crucial role in our setting (Larger settings still worth to be explored in future work). As the model size increases (from 340M to 1.3B), PCformer shows substantial performance gains, both in terms of accuracy on the LM harness evaluation and lower perplexity on Wikitext and Lambada tests.\n\n- We trained the 1.3B model on a cluster of 256 A100 GPUs. Note that 1.3B models consist of 24 layers, where the hidden size is 2048 and the FFN size is 5432 （8/3 * hidden size, 16 attention heads and SiLU activation functions. The baseline (1.3B + 100B tokens) was trained up to 20 hours and nearly 40 hours for our PCformer. Thus PCformer (1.3B + 50B tokens) was trained within similar time.\n\n- Additionally, given that PCformer consumes more FLOPS per forward pass, we compared models with similar total FLOPS consumption. Our PCformer trained on less than 50B data outperformed Transformer++ trained on 100B data by a significant margin. Moreover, PCformer (340M & 16B) achieved results comparable to Transformer (1.3B & 16B) with nearly 1/4 of the parameters. These results demonstrate that PCformer remains competitive in settings with larger model capacity and data volumes, highlighting the potential for further research in model designs that fully utilize available data.\n\n- These results demonstrate that PCformer remains competitive in settings with larger model capacity and data volumes, highlighting the potential for further research in model designs that fully utilize available data. While the increased inference and training costs in the decoder-only paradigm are notable, the substantial performance gains justify continued exploration of PCformer, including parameter-efficient training algorithms.\n\n- We believe these updated results address concerns about the relevance and sufficiency of our experiments. Thank you for your valuable feedback. Please let us know if you have any further concerns.\n\n\n\n> Q1: What's the efficiency of the proposed method, such as inference time and GPU memory consumption? I saw that inference efficiency has been mentioned in the limitations; it's better to have quantitative results and also memory states.\n\nThis question is a common issue among reviewers, we hope you can see the global response W1 for more details!\n\nDue to the limited page in global response, we provide a comparison of the total inference time between PCformer and the baseline in the 1.3B setting (trained on 100B tokens). Using 8 A100 GPUs with CUDA 12.2, we measured the inference time across all benchmarks listed in the table. The PCformer took 1428 seconds, while the Transformer++ took 1037 seconds. Although PCformer's inference time is longer, the performance gains make the additional time investment worthwhile.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you to all four reviewers for your efforts and instructive comments on our paper. We believe these updated results address your concerns regarding the efficiency and effectiveness.

> W1: about the computation overhead, inference and training cost comparison.

| Model                    | Layers | Inference | Memory | BLEU |
| ------------------------ | ------ | --------- | ------ | ---- |
| Transformer              | 6      | 98.7      | 13.2   | 29.2 |
| Transformer              | 12     | 94.5      | 18.7   | 29.7 |
| Transformer              | 24     | 87.3      | 23.5   | 29.8 |
| ODE Transformer (RK2)    | 6      | 93.5      | 15.1   | 30.7 |
| PCformer (RK2 predcitor) | 6      | 90.3      | 16.2   | 30.9 |
| ODE Transformer (RK4)    | 6      | 87.1      | 17.3   | 30.5 |


1. The table above compares the inference speed (sentences/s) and memory consumption (GB) for various models in a big configuration on WMT En-De. The experimental results show that the proposed PCformer models achieve comparable inference speeds with the baselines. This is primarily because MT models typically follow an encoder-decoder paradigm, with the main computational overhead coming from the decoder side due to the auto-regressive decoding process, rather than the encoder on which we primarily experimented.

2. In terms of memory consumption, PCformer is also competitive. It consumes slightly more memory during the forward computation phase because we need to store previously obtained approximations for the multistep method (corrector), as well as iteratively generated inner step approximations for higher-order predictors. This slightly increases the memory consumption for encoder-decoder models.

3. However, if the proposed predictor-corrector paradigm is applied to encoder-only or decoder-only models, such as BERT for the former and LLMs for the latter, the additional overhead is nonnegligible. For example, our PCformer would consume about twice time compared the vanilla baseline in LLM training. Despite this, the performance gains are still noticeable. These observations motivate us to develop more efficient variants to overcome the computational overhead. 

4. Fortunately, we have already collected some promising results in this direction. We attempted to achieve the high-order approximations in latent space and current experimental results show there is only a quite small performance gap compared with the PCformer in most of tasks, but is computation friendly!


W2: More results on LLMs.

| Model(Params & Tokens)     | Wiki.(ppl) | LMB.(ppl) | LMB.     | PIQA     | Hella.   | SCIQ     | ARC-c    | Winograde| Avg.     |
| -------------------------- | ---------- | --------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |
| Transformer++ (340M & 6B)  | 38.5       | 96.1      | 21.4     | 60.3     | 29.1     | 69.2     | 21.5     | 50.4     | 41.9     |
| PCformer (340M & 6B)       | 35.3       | 78.8      | 23.6     | 61.6     | 30.1     | 71.6     | 22.9     | 51.8     | 43.6     |
| Transformer++ (340M & 16B) | 28.3       | 65.3      | 29.8     | 63.2     | 33.9     | 73.2     | 23.1     | 51.4     | 45.8     |
| PCformer (340M & 16B)      | 25.6       | 39.7      | 34.5     | 65.2     | 36.9     | 79.6     | 23.2     | 52.2     | 48.6     |
| Transformer++ (1.3B & 16B) | 23.8       | 26.2      | 37.3     | 65.7     | 37.6     | 78.6     | 23.7     | 51.5     | 49.0     |
| PCformer (1.3B & 16B)      | 20.9       | 23.2      | 42.5     | 68.3     | 43.4     | 81.5     | 25.1     | 52.4     | 52.2     |
| Transformer++(1.3B & 100B) | 16.3       | 11.8      | 51.6     | 71.0     | 51.7     | 86.7     | 28.1     | 54.6     | 57.2     |
| PCformer (1.3B & 50B)      | 16.2       | 9.38      | 55.1     | 71.9     | 54.8     | 88.6     | 29.6     | 57.2     | 59.5     |
| PCformer (1.3B & 100B)     | **14.0**   | **7.46**  | **59.6** | **73.8** | **60.0** | **90.7** | **31.7** | **61.7** | **62.9** |

1. Here we scaled the model size from 340M to 1.3B parameters, which is the maximum size feasible within the limited time. For the data, we scaled from 6B to 16B and 100B tokens. PCformer can significantly beats the baseline in similar parameters, and with more training data, the average performance improves significantly. This demonstrates that our model benefits from increased data size, without experiencing diminishing returns. This indicates that model bias persists and plays a crucial role in our setting (Larger settings still worth to be explored in future work). As the model size increases (from 340M to 1.3B), PCformer shows substantial performance gains, both in terms of accuracy on the LM harness evaluation and lower perplexity on Wikitext and Lambada tests.

2. We trained the 1.3B model on a cluster of 256 A100 GPUs. Note that 1.3B models consist of 24 layers, where the hidden size is 2048 and the FFN size is 5432 （8/3 * hidden size, 16 attention heads and SiLU activation functions. The baseline (1.3B + 100B tokens) was trained up to 20 hours and nearly 40 hours for our PCformer. Thus PCformer (1.3B + 50B tokens) was trained within similar time.

3. Additionally, given that PCformer consumes more FLOPS per forward pass, we compared models with similar total FLOPS consumption.
    - Our PCformer trained on less than 50B data outperformed Transformer++ trained on 100B data by a significant margin. Additionally, PCformer (340M & 16B) achieved results comparable to Transformer (1.3B & 16B) with nearly 1/4 of the parameters. These results demonstrate that PCformer remains competitive in settings with larger model capacity and data volumes, highlighting the potential for further research in model designs that fully utilize available data.
    - While the increased inference and training costs in the decoder-only paradigm are notable, the substantial performance gains justify continued exploration of PCformer, including parameter-efficient training algorithms.

Meta-review: This paper explores transformer architecture design to minimize the error compared to the solutions from ODE formulation. It introduces a predictor-corrector learning framework to minimize the truncation errors. It also proposes an exponential moving average-based coefficient learning method to further strengthen the higher-order predictor. Extensive experiments show the superiority of the approach on different benchmarks. All the reviewers acknowledge that this paper provides novel contribution and has valuable contribution to transformer architecture innovation from ODE perspective. However, the reviewers have also pointed out several weakness of the paper, such as restrictions of the application to natural language domain, relatively small model size in experiments, and potential complexity of the method. During the rebuttal, the authors have successfully addressed most of the major concerns by providing additional results on vision domain and time series domain. It also provides additional experiments on the scaling behavior of the method, which shows its potential effectiveness when the FLOPs go up. One suggestion from my side to present the scaling law results is to also show the results in the form of “performance vs FLOPs”, which is another standard visualization in scaling law study.
Decision: Accept (poster)