Applying Second Order Optimization to Deep Transformers with Parameter-Efficient Tuning
Authors: Ning Ding
Qiaosen Wang
Yulin Chen
Pengjun Xie
Zhiyuan Liu
Hai-Tao Zheng
Maosong Sun
Venue: Submitted to ICLR 2023 (2023)
Keywords: Pre-trained Models
NLP
Model Adaptation

Abstract: Despite the theoretical superiority in convergence issues, second-order optimizers are generally not among the top choices for training large-scale neural networks due to their high computational and memory cost. Nevertheless, introduced in recent progress of parameter-efficient tuning is a new paradigm that large-scale pre-trained models (PTMs) can be adapted to specific tasks by optimizing a tiny proportion of parameters, which might hopefully change the game. We associate this new paradigm with the computational tractability of second-order optimizers and succeed in applying them to large PTMs that are from hundreds of millions to billions in scale. Beyond verifying their tractability, we further investigate the stability-influencing factors in the optimization process and propose accordingly a Newton-step-clipping approach in which we clip the update tensors rather than the gradients. This approach stabilizes the convergence by gating the magnitude of Newton steps along the optimization trajectories through the rugged landscapes of deep transformers. 
We conduct extensive experiments across different downstream tasks, demonstrating that, when equipped with our Newton-step-clipping strategy, second-order optimizers, especially Kronecker-factored curvature approximation (K-FAC), can attain comparable and even superior results and faster convergence to those state-of-the-art bars implemented with AdamW.  Furthermore, we scale the model up to 3 billion parameters and validate the tractability and effectiveness of our method. This work is not only the first successful application of second-order optimization on such large-scale models but also sheds light on the possibility of further optimization-wise analysis on large-scale models in the future.

Under review as a conference paper at ICLR 2023
APPLYING SECOND ORDER OPTIMIZATION TO DEEP TRANSFORMERS WITH PARAMETER-EFFICIENT TUN- ING
Anonymous authors Paper under double-bling bbind review
ABSTRACT
Despite the theoretical superiority in convergence rate, second-order optimizers are generally not among the top choices for training large-scale neural networks due to their high computation and memory cost. Nevertheless, introduced in recent progress of parameter-effie models (PTMs) can be adapted to specific tasks by optimizing tiny proportion of parameters, which might hopefully change the game. We associate this new paradigm with the computational tractability of second-order optimizers and succeed in applying them to large PTMs from hundreds of millions of parameters billions in scale. Beyond verifying their tractability, we further investigate the stability- inflit--iffuuecingg factors in the optimization process and propose accordingly a Newton step clipping approach in which we clip the update tensors rather than the gradients. This approach stabilizes the convergence by gating the magnitude of Newton steps along the optimization trajectories through the rugged landscapes deep transformers. We conduct extensive experiments across different downstream tasks, demonstrating that, when equipped with Newton-ster clipping, second-order optimizers, especially Kronecker-factoonn factored curvature approximation (K-FAC), attain comparable and even superior results and faster convergence to those state- of-the-art bars implemented with AdamW. Furthermore, we scale the model up to 3 billion parameters and validate the tractability and effectiveness of our method. This work is not only the first successful application of second-order optimization on such enormous models but also paves the road towards the design and analysis of second-orde optimizers for the downstream adaptation of large-scale PTMs.
tuning is new paradigm that large-scale pre-trained
INTRODUCTION
Pre-Trained models (Bommasani et al., 2021; Han et al., 2021) (PTMs) based on deep transform- (Vaswani al., 2017; Devlin et al., 2019; Raffel et al., 2020; Brown al., 2020) yield remarkable performance on a wide range of tasks thanks to the tremendous capacity brought by numerous parameters. A prevalent paradigm is to firstly pre-train such models large-scale corpora in self-supervised manner and then adapt them to specific datasets (typically with supervision). Such adaptations are usually implemented by first-order gradient-based optimization. However, recent applications have witnessed an approaching bottleneck for first-order training - it is neither likely to be faster nor easy to attain higher scores (Pascanu et al., 2013; Goodfellow al., 2016). On the other hand, second-order optimizers which enjoy better convergence properties might have been an ideal alternative in theory (Nocedal Wright, 1999; Boyd et al., 2004). They are, nevertheless, almost unattended on the downstream adaptation of PTMs in practice. It is because they require quadratic storage and cubic computation time for each update, which is especially prohibitive given the enormous model scale, even though bunches of their simplified counterparts have been devised (Byrd et al., 1995; Martens & Grosse, 2015; Botev et al., 2017; Anil et al., 2020; Tang et al., 2021).
Hopefully, recent advances in PTMs have brought slight twist to the situation. Studies show that paramparrmeter optimization may not be necessary for task-specific adaptations, as updating a small portion (0.05%~199 1%) of parameters et al., 2019; Li & Liang, 2021; Lester et al., 2021; Hu et al., 2021). With the new paradigm largely shrinking the trainable parameter space, we find in experiments that second-order optimization is
achieve non-trivial performance in many datasets (Houlsby

---

Under review as a conference paper at ICLR 2023
now tractable on large-scale PTMs with up to billions of parameters. This successful implementation signals that the design and analysis of second-order optimization can start marching from relatively toy or medium models to pre-traannt pre-trained deep transformers that are formidably large in scale. Further triggered a sequence of interesting research topics including: How tractable are second-order optimizers on deep transformers? Are they capable of converging faster and more steadily and yielding better results? If not, what auxiliary skills are needed? And also, how does the ratio of trainable parameters influence the relative performance of second-order optimization?
In this application-oriente paper, we answer all the aforementioned questions through theoretical justification and experimental verification. Taking K-FAC as the major example of the second-orde optimizer, we first experimentally verify that its training time and memory cost on extremely large PTMs are totally affordable. In addition, we point out that clipping the gradients before or after Hessian (or Fisher) preconditioning makes a non-negligible difference. We propose accordingly a Newton-step clipping strategy which is indispensable for second-order training because of its superior stabilizing effect. We then post comprehensive results to illustrate that with the assistance of our Newton-step clipping strategy, second-order optimization outperforms baseline first-order optimzers well as its non-clipping and traditionally gradient clipping counterparts in terms of both convergence speed and final test scores. Moreover, we scale the model scale up to 3 billion parameters evaluate the tractability and effectiveness. Observations of the relation between tunable ratio and convergence speed are also presented subsequently.
2 SECOND-ORDER OPTIMIZERS FOR LARGE- SCCLE TRAINING
Following the introductory part, we recap the essential background of second-order optimization. While first-order optimizers such stochastic gradient descent (SGD) and Adam (Kingma & Ba, 2014) have been more than ubiquitous in deep learning, second-order optimizers remain rela- tively under-explored in this field. Unlike their first-order counterparts, which take only first-order derivatives into account, second-order optimizers will in addition incorporate the loss function's second-order features, or, in other words, the curvature information.
2.1 NEWTON'S ' METHOD AND ITS VARIANTS
Newton's 's method and its variants are perhaps the most typical second-order optimization skills. To be specific, in Newton's 's method, we approximate the loss function with its Taylor expansion up to order
L(O + 60) L(0) + (o) 50 + 2
=5000 VOL(0)60
1(0) ( 50
50 H H0)500
in which g0) and (0) are the gradient and Hessian matrix of L at respectively. We then hope to move towards the direction in which the quadratic function on the right- side of Equation 1 is minimized. This condition implies that the standard Newton's method should proceed
0 + 0 - n H (0,) ()))).
(())))((O)))
(2)
where T/t is the learning rate for the t-th step. Here the Hessian /i is assumed to be invertible. The quantity u = H " is usually called a Newton step.
Despite the theory that fewer steps required for convergence, the original Newton's method has long been criticized for high computation and storage costs in calculating the Hessian matrix, inverting it, and other additional matrix manipulations (Nocedal & Wright, 1999; Boyd et al., 2004). To remedy this deficiency of speed, amazing increments of Newton's 's method have been proposed successively in recent centuries. Included are those reducing the cost by using an approximation of the Hessian matrix (Levenberg, 1944; Marquardt, 1963; Botev et al., 2017), and those quasi-Newton optimizers such BFGS (Broyden, 1970; Fletcher, 1970; Goldfarb, 1970; Shanno, 1970) and its limited-memory version L-BFGS (Byrd et al., 1995) which approximate the inverse of Hessian matrix directly. Those aforementioned methods can partly alleviate the intractability of full Newton's method on deep neural networks.
2.2 NATURAL GRADIENT DESCENT AND K-FAC
In a common situation of supervised learning, a class of parametric conditional distribution {p(v x, 0) 0 () is assigned to the model to fit the underlying distribution q(v æ)) of the ob-
2

---

Under review as a conference paper at ICLR 2023
served data. Kullback-Leiblee divergence (Kullback & Leibler, 1951) L( Dxzzlg2, y)lp((z, 0)] between the two usually selected the loss function to be minimized. Natural gradient descent suggests update direction satisfying
(2.2l((,,))llp,, y joint distributions, or up to an additive constant, the negative log- bog-likelihood, is
(0)) lim c-> [l (ttp://)0000e_0-0000000- so
arg min
I(0)
x F.)
where k is positive constant and
(()) p v logp(a; 0) TV logp(z, 0)),
(5)
is known as the Fisher information matrix (FIM) of p(z, 0). Since the true FIM is generally not accessible, is a common practice evaluate the empirical FIM (Martens, 2020) instead, that is, to evaluate
(() Eq(r,s) Vo p(r, Ø) Vo logp(z, y 0)),
in which q is the empirical distribution of the observed samples.
Interestingly, when current distribution p(- 0) and target distribution q are close each other, we have (0)( = F()). This relation sheds light on the connection between Newton's 's method and natural gradient descent, in the sense that natural gradient descent can be conceived as special Newton's 's method with approximate Hessian matrices. For this reason, we do not distinguish natural gradient descent from Newton's method in the rest of the paper and also call the update in equation Newton step Newton update. Further discussion on the connections and distinctions among FIM, empirical FIM and Hessian matrix are provided in Kunstner et al. (2019).
Vanilla natural gradient descent suffers from the same inefficiency with full Newton's method. Compared to the numerous increments Newton's 's method, few improvements for natural gradient descent have ever been devised, among which is the enlightening work of Kronecker-factoree approximate curvature (K-FAC) (Martens & Grosse, 2015). K-FAC specializes in optimizing the weights only involved once in linear mapping, including the weights of fully-connected linear projection and convolutional operation (Grosse & Martens, 2016; Martens et al., 201 8; Ba et al.,
2016). The core idea of K-FAC is the approximate evaluation of empirical FIM F(0) taking advantage of the properties of Kronecker products. To be specific, it approximates the expectation of the Kronecker product with the Kronecker product of expectation. In this way, the computation and storage burden is greatly relieved without simplifying the second-order structures too much. K-FAC has been proved promising optimizer across various deep learning models and tasks (Martens & Grosse, 2015; Grosse & Martens, 2016; Martens et al., 2018; Wu et al., 2017; Osawa et al., 2019). and will be the major second-order optimizer we implement and analyze in this paper. More details about natural gradient descent and K-FAC can be found in appendix A.
3 PARAMETER-EFFICNE
TUNING ENABLES SECOND-ORDER TRAINING
In the foregoing section, we have introduced typical second-order optimizers together with incremen- tal methods which ameliorate their efficiency in large-scale settings. In spite of that amelioration, second-order optimization still requires generally N2 order of storage space and 3 order of com- puting operations for every full parameter block with NV parameters, and is therefore intractable on extremely large models, especially those foundation language models based on deep transformers. For example, it is unlikely a possible mission to fine-tune T5XL with either L-BFGS K-FAC directly. The sad truth informs us that, beyond making second-order optimizers lighter and faster, more modifications should be made to the way we train. While hardly any further improvements could be made to the optimizers, cutting down the volume of the trainable parameters might be another way out. This straightforward thought coincides with new paradigm in the spotlight carameter-efficenn tuning- including adapter, LoRA, and other useful methods, which we will explain in detail in the following passage.
3.1 BACKGROUND OF PARAMETER EFFICCEET TUNING
Parameter-efficient tuning aims to adapt PTMs, especially large ones, via optimizing a tiny proportion of parameters. This intuition could also be found in previous works under other scenarios (Tajbakhshh
3

---

Under review as a conference paper at ICLR 2023
Table 1:
Peak memory consumption of different models with first- and second-order optimization. #P denotes the number of total parameters, and #T denotes trainable parameters. All the numbers are measured on the RTE dataset with LoRA with single NVIDIA A100 GPU, batch size = 16.
Adai K-FAC
30
Memory (G)
15-
Model
#P
#T
Optim.
Mem.
10
AdamW
4.22 GB
BERT
336M
0.75M
K-FAC
4.99 GB
AdamW
4.32 GB
RoBERTa
355M
0.75M
K-FAC
5.08 GB
500
1000
1500
2000
Time (6)
AdamW
T5e
220M
0.56M
K-FAC
2.49 GB
Figure 2: An example of GPU memory varia- tion during training the STS-B dataset with adapters. The used model RoBERTa and the batch size is 128. Both experiments are run on a single NVIDIA A100 GPU.
AdamW
6.00 GB
T5L
770M
1.50M
K-FAC
7.53 GB
AdamW
20.94 GB
T5XL
3B
3.75M
K-FAC
34.00 GB
al., 2016; Guo et al., 2020; 2019). The most practical advantage of such a paradigm is that we do need to update all the parameters and produce separate fine-tuned instances for every downstream task. By training such lightweight parameters, we are able flesh out the abstract ability of large-scall scale models to solve specific problems. Houlsby et al. (2019) injects small neural modules to each layer of the Transformer model and only optimizes the adapters training. Subsequently, series of variants of adapters have emerged (Pfeiffer et al., 2020; Sung et al., 2021; Mahabadi et al., 2021; Sung et al., 2022). Prefix and prompt tuning (Li & Liang. 2021; Lester et al., 2021) prepends tunable parameters to the input layers. LoRA (Hu et al., 2021) injects low-rank trainable decomposition matrices to the weights and successfully applied to GPT-3 (Brown et al., 2020) with 175 billion parameters. Apart from introducing additional parameters, experiments show that optimizing designated proportion of the inherent parameters produce a similar effect (Zhao et al., 2020; Zaken et al., 2021; Guo et al., 2021). This line of work implies that after massive pre-training, the adaptation of large-scale PTMs may be a "simple" process and is worth further exploring (He et al., 2022; Ding et al., 2022).
Adapter. The adapter (Houlsby et al., 2019) method inserts lightweight neural modules (adapter layer) to each layer of the Transformer model. Given the input hidden state hip E Rd each adapter layer comprises a down-projection D Rdxr non-linear activation function a--), and an up-projection U R Bxxd There is also residual connection from the input to the output of an adapter layer.
hou (hin
hin-
The position of adapter layers is optional. Houlsby et al. (2019) add them after the multi-head attention layer and the feed-forward layer of each Transformer block. There are also variants that add adapter modules to other positions like the LayerNorm layer (Pfeiffer et al., 2020). Depending on the choice of the size of the bottleneck dimension r, the tunable parameters of the adapter approach account for roughly 0.5%~~8% of the total number of parameters.
LoRA.
The LoRA method (Hu et al., 2021) assumes that the change of each weight matrix is intrinsically low-rank, thus it injects tunable low-rank decomposition matrices D E pddxr U e Br to estimate the weight changes AW DU. Hence, the output hidden state
hour + hin W + D)U,
(8)
where W is the initial weight matrix. Depending on the intrinsic rank r, the optimizable parameters of LoRA are normally less than 1% of the total parameters.
3.2 PARAMETER-
TUNING ENABLES SECOND-ORDER TRAINING
Having introduced the background second-order optimization and pframeter-efficient tuning and clarified our motivation for integrating them, move verify through experiments that
4

---

Under review as a conference paper at ICLR 2023
parameter-efficient tuning can indeed enable second-order optimization on large-scale PTMs. As shown in Table 1, PTMs with different architectures and scales could be tractably adapted with a single NVIDIA A GPU. It is worth noting that the excess memory consumption resulting from second-order optimization is relatively low compared to the whole memory consumption. As the model scales, the absolute amount of parameters to be fine-tuned increases, and the memory footprint of the second-order optimization increases considerably. For example, in T5XL. the second-order optimization takes up 13.06 GB more video memory than the first-order optimization. However, this is still acceptable in practice. We report an example of the GPU memory variation in Figure 2, which has a jagged shape over the training time. At low points, all the data is sent to the CPU for processing. leaving only the parameters of the model itself and the states in the optimizer in GPU. The gap between the first- and second-order optimization's 's memory usage at the low point reflects the volume difference in states of optimizers. We also conduct an analysis of time efficiency in Appendix C.3.
4 NEWTON-STED CLIPPING STABILIZES SECOND-ORDER TRAINING
The combination of second-order optimization and parameter-efficient tuning is not enough for training smoothly (illustrated in Figure in Section 5.2). We reveal both in theory and by experiments that new auxiliary skill, which we call Newton-step clipping. has to be implemented to achieve satisfactory results. In this section, we first introduce a more traditional stabilizing strategy named gradient clipping. We will afterward clarify the divergence between gradient clipping and Newton-step clipping and justify our initiative of devising and applying this new approach.
4.1 GRADIENT CLIPPING
One crucial point for a smoother training experience is to manage your step-sizes (norms of update tensors) wisely. However, unlike the cases in toy tasks, it is of low efficiency to apply delicate step-size schedules, such as line search with Wolfe (Wolfe, 1969) and Armijo-Goldstein (Armijo, 1966) conditions, to large models like deep- taap--ransfforr--bbee based PTM. Such low efficiency is partially ascribed to high computational cost, and, more substantially, to the nonconvex landscape that invalidates the theoretical benefit of those schedules. Mostly adopted in deep learning is set up fixed learning rate schedule for all layers at the beginning of a training process. But, this solution may not be flexible enough and may result in gradient explosion problem since the optimal step-sizes for different layers can typically be different.
Two strategies adaptive gradient and gradient clipping have been devised in previous works to tackle this issue. AdaGrad (Hinton et al., 1), RMSProp (Duchi et al., 2012), Adam (Kingma & Ba, 2014), and other optimizers adopting the adaptive gradient strategy divide the first-order term by the square root of the second-orde term to ensure that the magnitude of each layer's step-size automatically remains in a rather stable range highly irrelevant to the gradient norm, Compared the adaptive gradient methods, gradient-clippnng (Pascanu et al., 2013; Goodfellow et al., 2016) is more straightforward approach to set off the influence of extremely rugged landscapes. In this approach, a gradient vector is clipped whenever its norm exceeds a fixed threshold, that is,
g clp
min
9-
[gll
This approach is designed mostly by intuition to conquer the high-curvature landscape in deep learn- ing, especially indispensable for some NLP-specialized models (Pascanu et al., 2013). Apart from the stabilizing effect, more of its advantages are subsequently explored and proved, such as faster con- vergence (Zhang et al., 2019) and prevention of convergence to stationary points (Chen et al., 2020).
4.2 NEWTON-STER CLIPPING
In fact, the term "gradient clipping" can be misleading. We claim that the underlying purpose of per- forming "gradient clipping" is restrict the scale of the update tensor, NOT the gradient as is literally suggested. It is noteworthy that gradient clipping is usually applied to first-order methods, especially SGD and its variants. For these optimizers, the stepsize is proportional to the gradient norm, and hence clipping the gradient is equivalent to restricting the scales of updates. Nevertheless, for second-order optimizers, this rule may not hold, For instance, in natural gradient descent, the update is chosen to be
Elgg J-11 EEg]]
(10)
5

---

Under review as a conference paper at ICLR 2023
Algorithm I: Generalized procedures of pre-clipping and post-clipping
Data:
Total iteration current iteration E, current learning rate 1):
while : 7 do
Compute and store gradient g: and other required intermediate quantities hy; it pre-clipping then
9:
min
To
9ti
Compute other required quantities rt;
Transform the gradient to the final update by: us t- Tig. hi), where T is given transform; post-clipping then
ue min
||aell uei
Update the parameter 0 <- 0 1/
Hence, we may expect that approximately || (11
The above relation indicates that K-FAC can take an arbitrarily large step at low curvature point, therefore, problematic to associate large gradient norm with large step size natural gradient descent. similar problem persists among general second-order optimizers due to the precon- dition gradients by either Hessian matrices or FIMs. In addition, though traditional gradient clipping (that is applied before preconditioning) indeed shrinks the updates of a second-order optimizer to certain extend, the precondi- tioned updates remain poorly controlled since the eigenval- ues of Hessian matrices or FIMs can be arbitrarily small.
1
H
igcpp
For the above reasons, gradient clipping, though intuitively correct, may fails improve the stability in second-order optimization. To address this issue, we propose Newton-step- clipping in which we clip the Newton update u -H' g (or u F- (g) rather than the original gradient g. that is,
Figure 3: An illustration of the Newton-step clipping strategy.
slp
min [1, 7
|||]]] u.
(11)
т, standing for the maximum update norm, is a hyperparameter. This approach has previously appeared as an exclusive skill for K-FAC training (Ba et al., 2016), but it would hopefully work for wider range of second-order optimizers. We examine this approach on K-FAC, on which we discover that r = 0.1 generally works well.
4.3 GENERALIZATIONS PRE-CLIPPING AND POST-CLIPPING
We can generalize our approach as follows. In gradient- based optimization, gradient plays major role. However, the weight (or parameter) seldom updated directly by its corresponding gradient; most optimizers will further transform the gradient into final update tensor. Newton's method makes a case point. It left-multiplies the gradient with the inverse of Hessian to yield final Newton step. Then divergence emerges in the question that when is the appropriate time for clipping. The vanilla gradient clipping strategy clips the gradients at the stage when the gradient backpropagation completed. This sounds reasonable for SGD, feasible for Adam, but problematic for K-FAC. different measure not to clip anything until the final updates have been transformed from the original gradients, such the two afforementoneed approaches pre-clipping and post-clipping by convenience. Procedures of pre-clipping and post-clipping summarized in algorithm
Newton-stey clipping. According to their time of clipping, name
In a word, we have explained the above passage that the choices of when to clip and what to clip can lead to entirely different training results. Although second-order optimizers favor later clipping upon the Newton step, for most of the rest of the optimizers that have ever been proposed, the choices between pre-clipping and post- post-clipping remain a mystery to be explored in the future.

---

Under review as a conference paper at ICLR 2023
5 EVALUATION
In this section, evaluate and analyze approach with corresponding baselines across widely used natural language understanding (NLU) tasks and different backbone PLMs.
5.1 EXPERIMENTAL SETTINGS
The evaluated benchmarks in our experiments include SST-2 (Socher et al., 2013) for sentiment analysis, MRPC (Dolan & Brockett, 2005) for paragraph detection, CoLA (Warstadt et al., 2019) for inference acceptability, RTE (Dagan et al., 2005) for inference, QNLI (Rajpurkar al., 2016) for inference, STS-B (Cer et al., 2017) for textual similarity, Choice of Plausible Alternatives (COPA) (Gordon et al., 2012) for commonsense causal reasoning, CommitmentBank (CB) (Marneffe et al., 2019) for inference, and Winograd Schema Challenge (WSC) (Levesque, 2011) for commonsense reasoning.
Datasets,
Setup. We adopt RoBERTatage (Liu et al., 2019) with 350 million parameters for QNLI, SST-2, RTE, COLA, MRPC, and STS-B; and T5-3b (Raffel et al., 2020) with 3 billion parameters for COPA, CB, RTE, and WSC. The project and second-order optimizers are implemented by PyTorch (Paszke et al., 2019), and the used models are loaded from the Huggingface Transformers (Wolf et al., 2019) library. We use NVIDIA Tesla A100 with 40GB memory for all experiments. Most datasets use accuracy evaluation metrics, except for MRPC, CoLA, and STS-B. MRPC uses F1 scores. CoLA uses Matthew correlation coefficient (Matthews, 1975). Spearman correlation and Pearson correlation scores are reported on STS-B. More experimental details reported in Appendix B.
5.2 EFFECT OF NEWTON-STEP CLIPPING
We first directly assess the effect of the Newton-step clipping method in the second-order training of deep transformers. When straightforwardly applying K-FAC to deep transformers without any clipping or with traditional pre-clipping. the training process is exceedingly unstable. Series of experimental results shown in Figure 4 verifyy that for large-scale second-order optimization, the strategy of Newton-step clipping significantly stabilizes the training process and outperforms its non- clipping counterpart, while pre-clipping cannot deliver similar effectiveness. The results indicate the indispensability of appropriate clipping when applying second optimization in the parameter- efficient paradigm, which is consistent with the discussion in Section 4.2.
SST-2
CoLA
RTE
MRPC
Newton proelipping
Newtun pre-dipping dipping
Newton pec-dipping cliping
pre-elipping clipping
Training loss
cloping
10a
10a
10a
50
40
Epoch
Figure 4: Training loss curve of K-FAC optimizer on RoBERTaange + Adapter with and without our clipping approaches. All settings and hyperparameters are the same except for clipping strategy.
5.3 EXPERIMENTAL RESULTS
Experimental results of natural language understanding are reported in Table We mainly compare our approach with AdamW (Loshchilov & Hutter, 2017), which is broadly considered as the most powerful optimization method for deep transformers. Most of the parfmeter-efficient adaptations achieve on-par performances to full parameter fine-tuning. With both the adapter and LoRA approaches, second-order optimization with our Newton-step clipping considerably outperforms the AdamW counterparts. Specifically, in direct comparisons, with the adapter method, the average performance in six tasks NewtwonClip outperforms Adam by 1.06%. And with the LoRA method, the absolute improvement is 1.25%. We also observe that the performance gap is mainly reflected in RTE, COLA, and MRPC datasets, which are also generally considered to the more difficult natural language understanding task than the other three tasks.
7

---

Under review as a conference paper at ICLR 2023
Table 2: Results on NLU tasks. indicates the results are from Hu et al. (2021), and a, b, and indicate different amounts of trainable parameters. best performances of each column.
Blue and orange represent best and second
#Trainable params
Method
QNLI
SST-2
RTE
COLA
MRPC
STS-B
Avg.
RoBERTabuse
125M (100%)
92.8
78.7
63.6
85,22
RoBERTajuge
355M (100%)
94.7
96.4
86.6
68.0
90.9
RoBERTauage Adapter
Adam Wa
6.0M (1.69%)
10.2
96.2
83.4
66.5.14.
88.7
91.0
86.75
AdamWT
3.0M (0.84%)
10.8
96.1
83.8
68.3 11.0
90.2
91.5
87.45
AdamWie
0.8M (0.23%)
94.7
96.3
72.912.9
66.3 12.0
87.7
91.510.5
84.90
AdamW
2.4M (0.67%)
44.5
95.6.0.2
84.642.2
48.2
91.8
91.94
87.22
NewtonClip
2.4M (0.67%)
94.3
96.3+0.4
86.8+1.5
08.4
92.3
91.640.3
88.28
RoBERTaurg
+ LoRA
AdamWt
0.8M (0.23%)
94.8
96.210.8
85.2
68.2
90.211.0
92.3
87.82
AdamW
0.8M (0.23%)
94. 10.66
95 95.4
82.5
69.1 69.1
91.1
91.6
87.30
NewtonClip
0.8M (0.23%)
94.4 94.410.1
96.210.3
85.7
70.411.8
92.710.3
91.940.3
88.55
As illustrated in Figure 5, equipped with Newton-step clipping, K-FAC demonstrates faster and more stable convergence than the first-order optimizer Adam Despite the effectiveness, the hy- perparameter tuning of K-FAC is slightly more costly than that of AdamW since the second-order optimizer itself more sensitive to the learning rate and also involves more hyperparameters, such as the scale of clipping. We carry out the ablation study of hyperparameters in Appendix C.1. We will investigate adaptive techniques for Newton-step clipping in future work to better deploy second-order optimization PTM adaptations.
QNLI
0.6
0.6
80
0.4
X-FAC
0.2
0.2
60
0.2
60
K-FAC
Validation metric
Training loss
0.9
0.0
20
10
15
10
20
50
Adamt
0.6
CoLA
MRPC
STS-B
AdoenW
60
0.6
75
0.4
0.2
20
0.2
0.9
0.0
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
Epoch
Figure 5: Training loss and validation metric curves of RoBERTa Tatage + Adapter NLU tasks with AdamW optimizer and K-FAC optimizer equipped with our Newton-step clipping strategy.
5.4 IMPACT OF THE NUMBER OF TRAINABLE PARAMETERS
To explore the impact of trainable parameters our method, we train a RoBERTaunge + LoRA with different LoRA intrinsic ranks (i.e., the bottleneck dimension of D. U in equation 8) on the MRPC dataset. When linearly choosing r in {8, 12, 16, 20, 24, 28, 32). the amount of trainable parameters becomes {0.8M, 1. 1.5M, 1.9M, 2.3M, 2.6M, 3.0M, 3, 4M). respectively (we conduct 3 runs with different random seeds for each LoRA rank). However, as illustrated in Figure 6 and Figure 7, the convergence speed is observably inversely proportional to the number of trainable parameters, ie., the smaller the number trainable parameters, the faster the convergence speed. Meanwhile, observe that the test performance does not change significantly the number of trainable
8

---

Under review as a conference paper at ICLR 2023
parameters changes. For PTM and specific task, the adaptation process is "simple" that it can be accomplished with very few parameter optimizations, but is difficult to make a leap in adaptation performance by changing the number of training parameters. In other words, the pre-traine mre-training, model structure, and scale of the model itself seem to determine the upper limit of practical adaptations. The fact that using fewer parameters leads to faster convergence is also testament to the effectiveness of our Newton-step clipping approach.
100-
Truinable porameters (M)
0.7
LoRA rank
3
0.6
Training loss
95
0.5
32
Metric
0.4
90
0.3
0.2
85
0.1
80
25
30
35
8
12
16
20
24
28
32
36
Epoch
LoRA rank
Figure 6: Training loss curve of models with different LoRA ranks. Experiments are con- ducted on MRPC with RoBERTatage and batch size is 128.
Figure 7: The change of test metrics (FI) and number of trainable parameters with different LoRA ranks. Experiments conducted on MRPC with RoBERTaaage batch size is 128.
5.5 SCALING 3 BILLION PARAMETERS
We scale our experimentation T5xI, sequence-to-sequence model with 3 billion pa- rameters. Evaluated datasets in this part relatively small due to high training costs. As shown in Table 3, K-FAC with the proposed Newton-step clipping can achieve comparable performance with AdamW. For some datasets like RTE and WSC, our method even outperforms its first-order counterpart by a considerable margin. We also empirically find that larger models tend favor larger maximum norms for Newton-step clip- ping due to the vast capacity. The success of K-FAC on T5xL. further demonstrates its tractability under parametcr-efficcie tuning paradigm, and its potential efficiently stecring large pre-trained models.
Table 3: Results on COPA, CB, WiC, and WSC datasets of T5x Adapter.
Method
COPA
CB
RTE
WSC
Avg.
Adam NewtonClip
82.00
89.29
79.86
67.31
79.62
80.00
89.29
82.01
69.23
80.13
DISCUSSION
Theoretical analysis and experimental results are presented in this paper, illustrating that the parameter- efficient paradígm can vivify second-order optimization on extremely large-scale PTMs with the assistance of the Newton-step clipping strategy. Although the application of second-order optimization on enormous PTMs is promising, the exploration is yet far from closed in the sense that pieces of dark clouds are still hanging over this topic. (1) To begin with,
have observed in our experiments, second-order optimizers exhibit higher sensitivity to the choices of hyper-pane parameters compared to their first-order counterparts. While second-order optimizers tend to introduce more pareme-arameters, many of these newly-added hyper-parameters are more obscured in mathematical meaning and their experimental influence is more elusive. It remains unclear whether there theories and methods to make the hyper-parametee tuning of second-order optimization no longer sort of dark art. (2) Another uncertainty lies in the question that whether the design architecture-speeife optimizers feasible. We notice in current work that both adapter and LoRA add to the original model fully-connected feed-forward branches which coincide with K-FAC's strength. But, for the other parameter-efffcient methods like modifications of each head of attention output in Prefix Tuning, no similar conclusion has yet been drawn. (3) Moreover, it appears promising to study the combination first-order or second-order optimization instead of sticking merely to one. Observed in our experiments is the rule of thumb that first-order optimizers, though with slower loss descent and lower test scores, enjoy better numerical and convergence stability. It is a natural deduction to run first-order steps as the warm-up period for second-orde -odder optimization. Seemingly trivial at first glance, the design of first and second-order compounds is bound to be arduous journey of in-detail technical implementation.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.090/8 2020.
Larry Armijo. Minimization of functions having lipschitz continuous first partial derivatives. Pacific Journal of mathematics, 16(1))1-3, 1966.
Jimmy Ba, Roger Grosse, and James Martens. Distributed second-orde optimization using kronecker-
factored approximations. 2016.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.072 07.558 2021.
Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical gauss-newton optimisation for deep learning. In International Conference on Machine Learning, pp. 557-565. PMLR, 2017.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. university press, 2004.
Cambridge
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Charles George Broyden. The convergence class of double-rank minimization algorithms 1. general considerations. IMA Journal of Applied Mathematics, 6(1):76-90, 1970.
Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained optimization. SIAM Journal on scientific computing, 16(5):1190-1209 1995.
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of SemEval-2017, pp. 1-14, August 2017.
Xiangyi Chen, Steven Z. Wu, and Mingyi Hong. Understanding gradient clipping in private sgd: A geometric perspective. Advances Neural Information Processing Systems, 33:13773-13782. 2020.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pp. 177-190. Springer, 2005.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACI. June 2019.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv preprint arXiv:2203.06900 2022.
William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of IWP Workshop, 2005.
John Duchi, Nikhil Elad Srivastava, and Kelvin Swersky. Lecture 6d separate, adaptive learning rate for each connection. Slides of Lecture Neural Networks for Machine Learning. 2012.
Roger Fletcher. A new approach to variable metric algorithms. The computer journal, 13(3):317-322, 1970.
Donald Goldfarb. A family of variable-metric methods derived by variational means. Mathematics of computation, 24(109):23-66, 1970.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
10

---

Under review as a conference paper at ICLR 2023
Andrew Gordon, Zornissa Kozareva, and Melissa Roemmele. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. *SM 2012: The First Joint Conference on Lexical and Computational Semantics Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop Semantic Evaluation (SemEval 2012), pp. 394-398, 2012.
Roger Grosse and James Martens. A kronecker-factored approximate fisher matrix for convolution layers. In Proceedings of ICML, pp.
PMLR, 2016.
Demi Guo, Alexander Rush, and Yoon Kim. Parameter-efficien transfer learning with diff pruning. In Proceedings of the ACL, pp. 4884-4896, Online, 2021. Association for Computational Linguistics.
doi: 10. 12011.all-ooooo..111. lall--ong..3...
Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of CVPR, pp. 4805-4814, 2019.
Yunhui Guo, Yandong Li, Liqiang Wang, and Tajana Rosing. Adafilter: Adaptive filter fine-tuning for deep transfer learning. Proceedings of the AAAI, volume 34, pp. 4060-4066, 2020.
Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Zhang, Liang Zhang, et al. Pre-trained models: Past, present and future. Al Open, 2:225-250, 2021.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor KirkgKKikaatrick,, and Graham Neubig. Towards unified view of parameter-effictte transfer learning. In Proceedings of ICLR, 2022.
Geoffrey Hinton, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP, In Proceedings of ICML, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. In Proceedings of ICLR, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXivv1412.6980 2014.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathemati- cal statistics, 22(1):79-86, 1951.
Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approx- imation for natural gradient descent. Advances in neural information processing systems, 32, 2019.
Brian Lester, Rami AI-Rfou, and Noah Constant, The power of scale for parameter-efffiient prompt tuning. In Proceedings of EMNLP, 2021.
Kenneth Levenberg. A method for the solution of certain non-linear problems in least squares. Quarterly of applied mathematics, 162):164-166, 1944.
Hector J. Levesque. The winograd schema challenge. In Logical Formalizations of Commonsense Reasoning, Papers from the 201 AAAI Spring Symposium, Technical Report SS-1 1--1, Stanford, California, USA, March 11 1223, 201 2011.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, et al. Datasets: A community library for natural language processing. arXiv preprint wxX2:2100.02846, 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of ACI., pp. 4582-4597, Online, 2021. Association for Computational Linguistics.

---

Under review as a conference paper at ICLR 2023
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907. 16692, 2019.
yya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of ICLR, 2017.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. ArXiv preprint, abs/2106.04647, 2021.
Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Inves- tigating projection in naturally occurring discourse. In Proceedings of Sinn und Bedeutung 23, 2019.
Donald W Marquardt. An algorithm for least-squares estimation of nonlinear parameters. Journal of the society for Industrial and Applied Mathematics, 11(2):331-441, 1963.
James Martens. New insights and perspectives on the natural gradient method. The Journal of Machine Learning Research, 21(1):5776-5811 2020.
James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pp. 2408-2417. PMLR, 2015.
James Martens, Jimmy Ba, and Matt Johnson. Kronecker-factored curvature approximations for recurrent neural networks. In Proceedings of ICLR, 2018.
B.W. Matthews. Comparison of the predicted and observed secondary structure 14 phage lysozyme. Biochimica et Biophysica Acta (BBA) Protein Structure, 405(())442 451, 1975.
Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer, 1999.
Kazuki Osawa, Yohei Tsuji, Yuichiro Ueno, Akira Naruse, Rio Yokota, and Satoshi Matsuoka. Large-scale distributed second-order optimization using frottceer-factored approximate curvature for deep convolutional neural networks. Proceedings CVPR, pp. 12359-12367, 2019.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318. PMLR, 2013.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning. arXiv preprint 2rXiv:2005.00247, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 66, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP, pp. 2383-2392, Austin, Texas, 2016. Association for Computational Linguistics. doi: (0.18653/v1/D16-11 1264.
David F Shanno. Conditioning of quasi-newton methods for function minimization. Mathematics of computation, 24(111):447656, 1970.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of EMNLP, pp. 1631-1642, 2013.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efffiient transfer learning for vision-and-language tasks. arXiv preprint arXiv:2112.06825, 2021.
12

---

Under review as a conference paper at ICLR 2023
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. arXiv preprint (ttb:2206006522,, 2022.
Nima Tajbakhsh, Jae Shin, Suryakanth R Gurudu, R Todd Hurst, Christopher B Kendall, Michael B Gotway, and Jianming Liang. Convolutional neural networks for medical image analysis: Full training or fine tuning? IEEE Iransactions on medical imaging, (565):1299-1311 2016.
Zedong Tang. Fenlong Jiang, Maoguo Gong, Hao Li, Yue Wu, Fan Yu, Zidong Wang, and Min Wang. Skfac: Training neural networks with faster kronecker-factored approximate curvature. In Proceedings of CVPR, pp. 13479-13487, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Alex Warstadt, Amanpreet Singh, and Samuel Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019. doi: 10.1
tacl 00290..
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of-the-arr natural language processing. arXiv preprint xrXiv:191000377,, 2019.
Philip Wolfe. Convergence conditions for ascent methods. SIAM review, 1 11(2):26-235, 1969.
Yuhuai Wu, Elman Mansimov, Roger B Grosse, Shun Liao, and Jimmy Ba. Scalable trust-region method for deep reinforcement learning using kronecker-factoree approximation. Advances in neural information processing systems, 30, 2017.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg, Bitfit:
Simple parameter-efficien fine-tuning for transformer-based masked language-models. ArXiv preprint, abs/2106. 10199, 2021.
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. arXiv preprint /905. 2019.
Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. Masking an efficient alternative to finetuning for pretrained language models. arXiv preprint arXiv:2004.12406, 2020.
13

---

Under review as a conference paper at ICLR 2023
A MORE ABOUT NATURAL GRADIENT DESCENT AND K-FAC
A.1 CONNECTIONS BETWEEN NEWTON'S METHOD AND NATURAL GRADIENT DESCENT
The Hessian matrix of the Kullback-Leible divergence (the loss function mentioned in section 2.2) reads
VOL(O) VOD ( y))p,(r, y 0)1
(12)
- - (e V p( y [)]
(13)
(x,,)) Ve log ( y) V log 0)]
(14)
In case that the current density p(- 0) is close the target density q, we could that V F.
A.2 UPDATE RULES OF K-FAC
In accordance with the symbols adopted in Martens & Grosse (2015), layer of linear mapping can be written as
8; W;
(15)
then the W,-related sub-block of FIM can be expressed as Define g: Us,"
F,, = E |vec V Vw,
1 (T,-1 & (a, g
15 -E @ 999.
(16)
The core idea of K-FAC is to approximate the expectation of Kronecker product in equation 16 with the Kronecker product of expectation, that is,
Fi, Eg a. a7 @ Eg Bggg _ A. G.
(17)
= Furthermore, in light of the properties that (B @ C)-1 B @ C-1 and that (B & CC)vec (V) vec(CV B ) the natural gradient update in K-FAC can be formulated in tensors as
W W. 17 G, (Vw,
(18)
B ADDITIONAL EXPERIMENTAL DETAILS
Hyper-parameters, For experiments in Table we perform hyper-parameter grid search for both AdamW and K-FAC optimizers to select better-performing models. For AdamW, the search space for learning rate is (0.01, 0.001, 0.0001, 0.00003) and for maximum gradient norm clipping scale is {0.1, 1.0, 10). For K-FAC, in addition to the learning rate and the Newton-step clipping scale (T), we also set the damping factor to stabilize the matrix inverting operation (Levenberg, 1944; Marquardı, 1963). The search space for learning rate, Newton-step clipping scale, and damping factor are {0.01, 0.05, 0.1, 0.5), {0.1, 1.0, 1.5, 2.0}. and {le-2, 1e-3, 1c-4, 1e-5, lc-6), respectively. The search space of K-FAC is larger for AdamW for two reasons. (1) K-FAC has more parameersss than AdamW, and we find that the damping factor has a considerable impact on the training.
(2) Using Adam to optimize deep transformers extensively practiced in the community and our previous empirical studies, and we choose the set of reasonable learning rates and simply use commonly used values for other parame-arseeters like weight decay. However, there is little empirical evidence to provide reasonable search space for K-FAC on large language models. The second-order update interval for K-FAC is 500 in our experiments, which could simultaneously yield promising performance and time efficiency. The batch size is 128 for Table 2 and Figure 5, and the training epochs and steps for each dataset shown Table 4.
The reported results in Table and Table 3 use the following hyper-pane papem-parreeter settings in Table 5 and Table For results in Figure 4, we adopt K-FAC the optimizer and set the learning rate as 0.01, and set the damping factor as le-4 for both experiments. For all experiments, weight decay is set to le-4, and a linear scheduler with warm-up adopted.
14

---

Under review as a conference paper at ICLR 2023
Datasets. All datasets are loaded with huggingface datasets (Lhoest et al., 2021). Since labels are not ac- cessible for the test data for testing. For small datasets (#samples < 10K), randomly divide the validation set into halves, and use one half as the test set and one half as the validation For larger datasets (#samples 10K), we randomly take IK samples from the original training set as valida- tion and the rest as the training set, keeping the original validation set as the test set. All models are trained on the training set and evaluated every 200 steps on the validation set, the checkpoint with the best performance on the validation set is kept for evaluation on the test set.
Table 4: Total training epochs and steps for cach dataset for Table 2 and Figure 5.
manually split part of the
Dataset
Epochs
Total Steps
QNLI
20
16220
SST-2
30
15570
RTE
50
1000
CoLA
50
3350
MRPC
50
1450
STS-B
50
2250
When dealing with MRPC, RTE, and STS-B datasets, some works use the best model checkpoint on the MNLI dataset for initialization to boost the performance. In our empirical study, we NOT use this strategy, and
usual initializations for all the models. In our experimentation, the main parameters of PTMs are frozen, and we only optimize the trainable modules.
GPU Memory. The GPU memory statistics in Table and Figure 2 come from PyTorch API and NVIDIA, respectively, so there might be a small inconsistency caused by NVIDIA's extra calculation of cache memory.
Table 5: The training hyper-parameters for K-FAC with the Newton-step clipping strategy.
Model
Dataset
Method
Learning Rate
Max. Norm
Damping Factor
Adapter
0.1
1.0
le-4
QNLI
LoRA
0.05
1.0
le-3
Adapter
0.1
1.0
le-3
SST-2
LoRA
0.1
1.0
lc-4
Adapter
0.05
1.5
le-3
RoBERTatnn
RTE
LoRA
0.1
1.5
le-5
Adapter
0.01
1.0
le-2
CoLA
LoRA
0.01
1.0
le-2
Adapter
0.5
0.1
le-2
MRPC
LoRA
0.05
1.0
le-3
Adapter
0.1
1.0
lc-2
STS-B
LoRA
0.1
1.0
le-6
CОРA
Adapter
0.1
10.0
le-2
T5XL
CB
Adapter
0.1
10.0
le-2
WSC
Adapter
0.1
10.0
le-3
RTE
Adapter
0.1
10.0
le-2
C ADDITIONAL EXPERIMENTAL RESULTS
This section reports additional experimental results Section 5. We study the effect of hyper- parameters, time efficiency, and clipping occurrences in the training process. We find that the training will be more sensitive when dealing with small datasets. Hence, to take a closer look the training procedure, we choose larger dataset (STS-B B and MRPC, respectively) and small dataset (RTE) for analyses Appendix and Appendix C.3,
C. EFFECT OF PARERR-RR
Hyper- pareee-pr could considerably influence the performance of optimization. In case, find that the learning rate, maximum norm, and damping factor could simultancously affect the training process. Hence, conduct an ablation study explore such impacts. When studying the effect of
15

---

Under review as a conference paper at ICLR 2023
Table 6: The training hyper-parameters for AdamW,
Model
Dataset
Method
Learning Rate
Max. Norm
Adapter
lc-4
0.1
QNLI
LoRA
le-3
0.1
Adapter
lc-4
0.1
SST-2
LoRA
le-3
0.1
Adapter
le-4
0.1
RoBERTape Jatage
Auarge
RTE
LoRA
le-3
0.1
Adapter
le-3
0.1
CoLA
LoRA
le-4
1.0
Adapter
le-4
1.0
MRPC
LoRA
le-3
0.1
Adapter
le-4
0.1
STS-B
LoRA
le-4
0.1
CОРA
Adapter
lc-4
0.1
T5XL
CB
Adapter
lc-4
0.1
WSC
Adapter
le-4
RTE
Adapter
le-4
0.1
Damping factor-0.01, Max norm-1.0
Leaminng rate-D.1, Max norn-1,0
Learning rate-0.1, 1, Damping faccoor-..0
Leaming rale
Damping factor
norm
0,01
0.1
le-6
1e:3
0,1
1.5
0-
0,05
le-5
0.01
1.0
2.0
le
-2
-2
log[Training loss)
-4
20
Damping factor-le.3 1e:3, nomm-1.5
Learning rale nomm-1.5 Learning Damping factor-le-3
Damping factor
le.6
2
le-5
0.01
le-4
-4
Learning rale
-4
Max
0.01
0.1
0.1
1.5
0.05
1.0
2.0
20
40
40
Epoch
Figure 8: Impact of different hyper-paramente parameters on test performance with STS-B dataset (upper) and RTE dataset (lower). Experiments are conducted with RoBERTa Adapter and K-FAC + NewtonClip, and is trained with batch size of 128 for 50 epochs.
one hyper-parameter we fix the other two RTE datasets and illustrate the training loss in Figure and the corresponding performance on the test set in Table 7. could seen that with a small learning rate and small maximum norm value, the model may fail to converge within reasonable number of training steps. It indicates that, generally, the second-order optimizer is compatible with a relatively large update value, which is in line with its theoretical precision in loss landscape estimation. In terms of performance, it is relatively stable for STS-B, BB, but observes a larger variation on the RTE dataset. probably because the scale RTE much smaller and thus the performance
We run an ablation study on STS-B and
vulnerable small perturbations in model parameters.
16

---

Under review as a conference paper at ICLR 2023
Table 7: Impact of different hyper-parameters on training loss with STS-B and RTE datasets. Experi- ments are conducted with RoBERTajarg + Adapter and K-FAC + NewtonClip, and are trained with batch size 128 for 50 epochs.
Dataset
Learning Rate
Damping Factor
MaxNorm
Performance
0.01
le-2
1.0
91.4
0.05
le-2
1.0
91.7
0.1
le-3
1.0
91.3
0.1
le-4
1.0
91.4
0.1
le-5
1.0
91.6
STS-B
0.1
le-6
1.0
92.1
0.1
lc-2
0.1
91.3
le-2
1.5
92.0
0.1
le-2
2.0
91.9
0.1
lc-2
1.0
92.0
0.01
le-3
1.5
71.2
0.1
le-3
1.5
83.5
0.05
le-2
1.5
80.6
0.05
lc-4
1.5
77.0
0.05
le-5
1.5
79.9
RTE
0.05
le-6
1.5
81.3
0.05
lc-3
0.1
57.6
0.05
le-3
1.0
83.5
0.05
lc-3
2.0
77.7
0.05
le-3
1.5
87.1
C.2 CLIPPING TRAINING PROCESS
We investigate how the proposed clipping tech- nique affects the training procedure. Figure 10 shows how many times the Newton-step clipping actually occurs in whole training procedure. It can be seen that clipping happens in almost ev- ery step in the early training stage, testifying the importance of clipping. After considerable iterations of training, some steps could obtain a steady amount of updates without Newton-step clipping. The phenomenon persists through dif- ferent datasets. It should also be noted that clip- ping is not equivalent to shrinking the learning rate. The functionality of clipping depends on the scale the update norm so that the scaling effect can be adjusted dynamically according the current update scale. Learning rate adjust- ment, however, applied in pre-defined manner that happens independent of the current update. Designing suitable learning rate schedule for a second-orde optimizer would require the knowl- edge of the change in update norm, and thus ex- tremely difficult given the rugged loss landscape of deep neural networks. Figure further proves the point where simply decreasing the learning rate without clipping technique will not achieve satisfactory training. where reduce the learning rate to K- K-FAC without Newton-ster clipping and find that none of them could steadily converge.
Irainninng loss)
Leaming rale
Clipped
0.001
0.01
0.5
False
30
Epoch
Figure Comparison of training loss curve be- tween small learning rate without post clipping and large learning rate with post clipping. Ex- periments conducted MRPC dataset with RoBERTaass + Adapter and K-FAC. Weight decay is le-4, and linear scheduler with warm-up is applied. The solid line denotes the training process with Newton-step clipping and dotted lines denote processes without Newton-step clipping across different learning rates.
17

---

Under review as a conference paper at ICLR 2023
MRPC
5.0
2.5
-2.5-
-5.0
200
400
600
800
1000
1200
1400
RTB
2.5
0.0
log( Total norm)
-2.5
-5.0
200
400
600
800
SST-2
10.0
7.5
2.5
-0.0
-25
2000
4000
6000
8000
10000
QNLI
10
2000
4000
6000
8000
10000
12000
clipped
not clipped
Step
Figure 10: Total parameter norm and status of post clipping throughout the training process with RoBERTaaage + Adapter and K-FAC NewtonClip. For all datasets. models trained with the optimum hyper-parameter setting.
C.3 TIME EFFICIENCY
After analyzing the memory tractability of second-order optimization in Section 3.2, find that second-order optimization is tractable for large language models under the parameter-efficient paradigm. In this section. we further conduct experiments for time efficiency. We conduct experiments on the MRPC and RTE datasets with single NVIDIA A100 GPU and compute the average wall clock time over 100 epochs. For second-order optimization, the value of second-order update interval could particularly affect the training time. Thus we conduct different runs with intervals in (1, 50, 200, 500). The results are reported Table 8, from where we observe that K-FAC reaches comparable speed with AdamW when the update interval is set to 50 since there are not lot of trainable parameters in the parameter-effice outspeeds AdamW with satisfying performance (we did not fine-tune the hyper- pperrpprraeetess for each setting in this section). In the spirit making second-order optimization as practical as possible on large models, set to 500 in all other experiments.
paradigm. When the value is above 200, K-FAC
18

---

Under review as a conference paper at ICLR 2023
Table 8: Wall clock time of AdamW and K-FAC NewtonClip on the RTE dataset with RoBERTalare Values in the parentheses denote the relative time over W.
Dataset
Optimizer
Update Interval
Wall Clock Time
Performance
AdamW
31.3s (1.00x)
91.3
K-FAC + NewtonClip
56.3s (1.80x)
92.7
MRPC
K-FAC NewtonClip
50
33.6s (1.07x)
90.7
K-FAC + NewtonClip
27.9s (0.89x)
91.5
K-FAC + NewtonClip
500
28.0s (0.89x)
92.3
AdamW
18.4s (1.00x)
84.6
K-FAC NewtonClip
31.4s (1.70x)
81.3
RTE
K-FAC + NewtonClip
16.3s (0.89x)
81.3
K-FAC + NewtonClip
200
17.8s (0.96x)
86.3
K-FAC + NewtonClip
500
16.9s (0.92x)
86.8
C.4 PROLONGING TRAINING PROCESS
In this section, we take a closer look at the training process by prolonging the iterations. Specifically, we extend the training procedure to 500 epochs on STS-B dataset, and the training loss and validation metric curves are shown in Figure 11. It can be seen that with both optimizers, the training loss keeps decreasing, while K-FAC observes an even smaller loss scale and converges faster, especially in the later training stage. At the final stage of training, both optimizers reach very small orders of magnitude terms of training losses (10 and 10-6 for AdamW and K-FAC, respectively). The validation metric curve of comparable performances also shows that there overfitting issues for both two optimizers, which could be regarded as an advantageous characteristic of parameter-efficient tuning.
Adan K-FAC
Validation metric
AdamW K-FAC
Training loss
10a
10-2
10-8
100
300
400
500
Epoch
Figure 11: Training loss and validation metric curves on STS-B dataset with RoBERTanng over 500 training epochs. The hyper-parameter settings follow Table 5 and Table 6.
19


Reviews:
[{"review_id": "tr4k8Gs8OD", "reviewer": "Reviewer_tTAh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes using second-order optimizers (K-FAC) to train large-scale Pre-Trained Models (PTMs). The key idea is to only apply the second-order update on adapter layers to make the storage and computation of the gradient covariances in K-FAC feasible in large PTMs. To further stabilize the training, the authors introduce a clipping scheme to control the size of the K-FAC update. Empirically, the proposed method can adapt additional layers in PTMs with faster convergence and improved performance compared to the AdamW baseline. ", "summary_of_the_review": "While I believe that the key idea proposed in the paper is interesting, the paper has critical weaknesses in both technical and empirical aspects. At the moment, I recommend a score of 3 (reject).  ", "strength_and_weaknesses": "Strengths:\n- The general idea of only applying second-order optimizers to adapter layers in Pre-Trained Models (PTMs) is interesting. \n- The paper is generally clearly-written and well-motivated. The topic of the paper is relevant to the ICLR community. \n\nWeaknesses:\n- The justifications for gradient clipping (pre-clipping and post-clipping) are non-theoretical and weak. Moreover, the authors do not mention the existing works on damping and clipping, which I further address below. \n- Algorithm 1 contains several undefined variables and functions, and it isn't easy to understand the exact algorithm. \n- The empirical results in the paper are not convincing due to two reasons: (1) the authors do not include the results of (a) no clipping, (b) only pre-clipping, (c) only post-clipping, and (d) the proposed modification, and (2) the search space for hyperparameters such as learning rate seem to be much larger for K-FAC and I believe that the comparison to weakly tuned AdamW is unfair. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "K13AVHF9GIm", "reviewer": "Reviewer_fdsh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This study applies second-order optimization (K-FAC) in parameter-efficient tuning (with adapters and LoRA) of Transformer models. Measurements show that the memory cost of second-order optimization can be relatively small due to the small number of trainable parameters, suggesting a second-order method is more suitable in this setting. This study proposes \"Newton-step clipping,\" in which gradient clipping is applied after preconditioning, and shows that the training with K-FAC is stable. In the parameter-efficient tuning of the RoBERTa large model, K-FAC achieves lower training loss than AdamW for a given number of epochs.", "summary_of_the_review": "The use of second-order information in parameter-efficient tuning is more reasonable than in full-parameter tuning, but its practical benefit is not clearly demonstrated in this study. The novelty of the proposed clipping procedure is limited, and its justification is more of an intuitive interpretation than a \"theoretical\" one. Based on the above, I do not believe that this study at this time is ready for publication.\n", "strength_and_weaknesses": "Strengths\n- Second-order optimization methods, which have computational and memory overhead, are suitable for parameter-efficient tuning. The numbers of the tunable parameters and the memory overhead of a second-order optimization method (I assume it is K-FAC) in fine-tuning pre-trained Transformer models presented in this work (Table1) help highlight this point.\n\nWeaknesses\n- The technical significance is limited.\n    - With parameter-efficient tuning, the training time could already be short (this is what parameter-efficient tuning is for), even with first-order optimization. The low computational and memory cost of second-order optimization is attractive, but (as I also describe below “the validity of the comparison”) it is questionable whether the benefit of the second-order information is worth the cost in parameter-efficient tuning. \n    - The proposed “Newton-step clipping” merely shifts the timing (from pre- to post-preconditioning) at which gradient clipping is applied. Post-preconditioning clipping of K-FAC gradient based on the KL-divergence of the model’s predictive distribution before and after the update (rather than the Euclidean norm of the update as in the gradient clipping) has already been proposed (https://jimmylba.github.io/papers/nsync.pdf). How one should determine the timing (pre- or post-) and \"distance\" (e.g., KL-divergence, Euclidean norm) is an interesting question, but that is not studied in this work.\n\n- The validity of the comparison is questionable.\n    - It is not clear whether the number of epochs shown in Figure 5, which depends on the task, is reasonable since the timing at which training stops (e.g., when training loss reaches a threshold, a fixed number of epochs) is not stated. In this case, there appears to be room for AdamW to achieve better performance when the number of epochs is further increased, and its difference from K-FAC could be more negligible.\n    - Furthermore, there is no mention of how the model is selected for evaluation (e.g., using a validation set) or how the training set, validation set, and test set are partitioned, so it is doubtful that hyperparameters are tuned on the test set. (Just because training loss converges faster, as shown in Figure 5, does not necessarily mean that the model generalizes well.) \n    - It is unclear whether K-FAC is \"faster\" than AdamW since there is no comparison regarding the wall-clock time for training. K-FAC is not necessarily faster than AdamW in wall-clock time (= number of steps x time per step) because of the per-step computational overhead compared to first-order optimization methods.\n    - Finally, optimization methods should be compared not only for best results but also for their sensitivity to the choice of hyperparameters. It is difficult to consider an optimization method practical for other tasks if it is too sensitive. \n\nOther comments\n- Table1: what are the “first-order optimization” and “second-order optimization”? A reader can guess they are SGD (or AdamW) and K-FAC, but it is unclear.\n- Eq (19) 2nd line: I believe it should be (a \\times g)(a \\times g)^T. The shape of each matrix and vector should be described to avoid confusion.\n- p6 the last paragraph, “derivative-based optimization”: I believe this should be “gradient-based optimization.”", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FXozufGHJ7j", "reviewer": "Reviewer_xAvz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper makes two observations regarding training large language models:\n1. During fine-tuning large transformers, several authors have previously shown that it is necessary to train a small number of parameters. This paper proposes using K-FAC to fine tune such models. The authors say that second order methods are very expensive for full training, but since fine-tuning involves few parameters, K-FAC is tractable, and achieves better results compared to first order methods.\n2. Gradient clipping is a common method to prevent training blowups due to large gradients. The authors show that for K-FAC, this is not sufficient due to preconditioning, instead clipping should be done both before and after preconditioning.\n\nThe paper contains several experiments showing that a network fine-tuned with K-FAC achieves slightly better results than with AdamW.", "summary_of_the_review": "The paper is currently not strong enough for ICLR. It can be strengthened by addressing some of the theoretical questions above, and slight enhancements to the experimental section.", "strength_and_weaknesses": "The main ideas in the paper are very easy to understand and well explained. The experimental section is good, with some caveats.\n\nK-FAC is a carefully designed algorithm with extensive theoretical justification. How does NewtonClip affect this theory? Does the convergence rate change? Does the bias introduced by NewtonClip mess the independence assumptions?\n\nGiven that second order methods are more expensive computationally, the authors should include a comparison of the runtime and memory cost of AdamW with NewtonClip. Also, please state the results of training with K-FAC only (without clipping).\n\nThe statement \"In spite of that amelioration, second-order optimization still requires at least N^2 ∼ N^3 order of storage space and computing operations\" is not really correct for K-FAC --- in fact the storage is at most 2mN, where m is the largest dimension of any layer, so this is much less than N^2. Similarly, compute operations are at most m^2 N. Still large, but not N^3.\n\nPage 7: NewtwonClip --> NewtonClip", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2bv3KEFTxvv", "reviewer": "Reviewer_Cqu9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper exploits a trend of low-rank finetuning or finetuning tiny number of parameters by applying a second order optimizer instead of typically used AdamW.", "summary_of_the_review": "While the paper proposes an interesting idea of leveraging potentially more effective optimizers in the case of finetuning small set of parameters, the paper's contribution of clipping Newton step poses several unresolved questions and not quite fully supported through evidence.", "strength_and_weaknesses": "Strengths\n- The general idea of leveraging expensive (but more effective) optimizers in the case of finetuning small number of parameters seems novel and practically useful.\n\nWeaknesses\n- It is unclear if tuning the learning rate more heavily may be able to subsume post-clipping, it would be interesting plot how many iterations clipping is actually applied. If it is applied in all iterations, then it may be corrected by just tuning the learning rate more thoroughly.\n- It would also be interesting to apply clipping to the baseline AdamW method. The authors discuss a hand-wavy rationale for not doing it, but it does not seem satisfactory and I would be interested in well-tuned empirical results on clipping AdamW update.\n- Another interesting experiment would be to run adamw longer and see if it can recover performance with Newton steps.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject