Exploring Landscapes for Better Minima along Valleys
Authors: Tong Zhao
Jiacheng Li
Yuanchang Zhou
Guangming Tan
Weile Jia
Venue: NeurIPS 2025 poster ()
Keywords: optimization
landscape
exploration
local minimum
convergence
exponential moving average
Primary area: optimization

Abstract: Finding lower and better-generalizing minima is crucial for deep learning. However, most existing optimizers stop searching the parameter space once they reach a local minimum. Given the complex geometric properties of the loss landscape, it is difficult to guarantee that such a point is the lowest or provides the best generalization. To address this, we propose an adaptor "E" for gradient-based optimizers. The adapted optimizer tends to continue exploring along landscape valleys (areas with low and nearly identical losses) in order to search for potentially better local minima even after reaching a local minimum. This approach increases the likelihood of finding a lower and flatter local minimum, which is often associated with better generalization. We also provide a proof of convergence for the adapted optimizers in both convex and non-convex scenarios for completeness. Finally, we demonstrate their effectiveness in an important but notoriously difficult training scenario, large-minibatch training, where Lamb is the benchmark optimizer. Our testing results show that the adapted Lamb, ALTO, increases the test accuracy (generalization) of the current state-of-the-art optimizer by an average of 2.5\% across a variety of large-batch training tasks. This work potentially opens a new research direction in the design of optimization algorithms.

Exploring Landscapes for Better Minima along Valleys
Tong Zhao! 1.2
Jiacheng Li
Yuanchang Zhou'.2 1.2
Guangming 1,2,
Weile Jia 12,
State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences
UUniversity of Chinese Academy of Sciences.
(zhaotong, 1ijiacheng22s, zhouyuanchang23e, tgm, jiaweile)Oict.
Code
PyPI
Abstract
Finding lower and better- seater-generalizing minima is crucial for deep learning. How- ever, most existing optimizers stop searching the parameter space a local minimum. Given the complex geometric properties of the loss landscape, it is difficult to guarantee that such point is the lowest provides the best generalization. To address this, we propose an adaptor "E" for gradient-based optimizers. The adapted optimizer tends to continue exploring along landscape valleys (areas with low and nearly identical losses) in order to search for potentially better local minima even after reaching local minimum. This approach increases the likelihood of finding lower and flatter local minimum, which is often asso- ciated with better generalization. We also provide proof of convergence for the adapted optimizers in both convex and non-convex scenarios for completeness. Finally, we demonstrate their effectiveness in important but notoriously difficult training scenario, large-batch training, where Lamb is the benchmark optimizer. Our testing results show that the adapted Lamb, ALTO, increases the test accuracy (generalization) of the current state-of- oate-off-hhe-rt optimizer by an average of 2.5% across variety large-bated training tasks. This work potentially opens a new research direction in the design of optimization algorithms.
they reach
1 Introduction
Almost all gradient-based optimiz- aim to converge imum [13, 31] after being trapped by certain local minimum. Fig-
local min-
25
ure illustrates this phenomenon of
ne-
two most widely used optimizers (e.g. SGD [37] and Adam [22]) on typical optimization test functions. However, if an optimizer only relies on local in- formation (e.g. loss function values and their gradients), it can not ensure that the point finds the lowest or the one with the best generalization.
35
X0yrrad Poes
2. -15
Figure 1: Comparison between SGD/Adam and ES- GD/EAdam on 2D polynomial test functions, which are typ- ically representative of landscapes. Left (the square of the cardioid): The valley forms a cardioid shape, where loss and special points are marked by cross. Some of these points have flat neighborhoods. which means better generalization. Right (Rosenbrock finction): The valley is parabolic, and the optimum is marked with cross.
What happens if modify the opti- mizer to ensure continued exploration along the valley for potentially lower and flatter minima? We propose a
Cooresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a)
(b)
f(0)
0
Large-scale
Small-scale
0x Op
QK-1
1
On
Oue
On
0 p-11
Opening
sharp
01
flat
01
f" 0
Pa
(c)
Bc-1
(d)
P--1
Figure 2: (a) is an intersection of valley (large-scale minimum), which captures optimizers. (b) the enlarged solid line in (a) between two dots and shows the optimizer escaping from small-scale sharp minimum. ag accelerates the training and remembers the direction of the right arrow. Zooming in on point 0k. we obtain (c) and (d), which show how ALTO handles minimum by analyzing of directions of 0₂ - (or -gx and gi
1-
gradient-based optimizer adaptor, "E" (exploration and exploitation), for this purpose. Fig- ure 1 shows, the adapted SGD (ESGD) and Adam (Eadam) explore along the valley (points with loss near 0). The longer the valley an optimizer explores, the flatter and lower the best minimum in the explored valley which is chosen the solution to optimization.
Before implementing the aforementioned ideas, Section introduces necessary preliminaries and notations and then identifies that modifying traditional gradient-based optimizers for persistent exploration requires simultaneously addressing two fundamental issues: (i) being trapped by large- scale local minima (to ensure valley-following behavior), while (ii) escaping sharp small- scale minima (to maintain exploration capability). Beyond this exploration property, the adapted optimizer demonstrates two additional advantages: accelerated training (fast loss decay) and preferential convergence to flatter local minima. theoretical completeness, Section provides convergence analysis for both convex and non-convex scenarios. The remaining sections apply the adapted optimizer ALTO to resolve an important and challenging training problem, for which the adapted optimizer is very well-suited.
Our experimental results demonstrate the superior performance of ALTO across various datasets and tasks, such as CV [20, 50, 42] and NLP [32, 33] training, with 3-5 times hyperparameter tuning per task for all optimizers in large batch training. Compared the current state-of sfate-ofthee-ee the-art,, ALTO achieves better accuracy in all our 17 CV and NLP experimental tasks and can save 29.68% of computation time on a typical CV task while reaching the same accuracy. particular, ALTO can achieve better accuracy (70.83%) on ImageNet with batch size 4086 compared to SGD with a batch size of 256 (70.64%), achieve better test perplexity (78.37) than that of Lamb [51] (83.13, SOTA) in GPT-2 [34] with batch size 4096, and outperform Lamb in image classication task (ResNet50, ImageNet with the same setting as its original paper [5 [JI) with batch sizes (IK, 2K, 4K, 8K, 16K, 32K).
Method
Preliminaries and notations. Consider the non-convex stochastic optimization problem
f* = min f(O),
:= =E<pp ()),
BED
where we define f(0) as the landscape [43] or the entire landscape. To find the best parameter 0" arg mingep f() within domain D C R2d optimizers aim minimize the expectation of the loss function { <). This function measures the suitability of the parameter 0 for sample Ç drawn from a dataset C R*, subject a probability distribution P. To navigate the parameter 0₂ towards 0" at time step k optimizers usually iteratively update it. For simplicity, consider SGD, which contains the core idea of all gradient-based optimizer for neural network. It guides the parameter to move along the negative gradient direction g: 12 Σs,2,, V (0x, 5.) of the landscape

---

(0) :=
1. [2el Ec,E2, {(k,. C:) of the batch 2₁ a 17,- me-leggth step
0 0 T
where Σ is a batch sampled from E, gi is an empirical estimator of V (0), and V denotes computing the gradient with respect 8. In the rest of the paper, operations like x² and x/y involving any vectors X and y are elementwise, while we denote inner product, 12-norm, and bse -orm as (, Il- and llso respectively. If the activation function Lipschitz, the value of f(0) at infinity is bound by a power function. Further, if the activation function is ReLU (due to its positive homogeneity), we obtain the slice of landscape f(x) = f(@ + x6') - Cz", where O E D and for some C, c 2 0. For some directions, 0 and f(x) tends to C. Therefore, assume that fxx) is power function infinity.
0' E gd-1 as
Design. Designing an optimizer capable of con- tinuing exploration along valleys rather than stagnating in local minima requires simultane- ously addressing two key aspects:
Table 1: Comparison of the directions of -ge- V((s)( and gx 1a--g-1 in different stages of Figure 2 (c) and (d). "+" and "." represent positive and negative, respectively. The sign of (0 0g Ex gk-1) indicates whether the op- timizer accelerated (+) or decelerated (-).
1. Macroscopically, the optimizer should be captured by large-scale local min- ima (to ensure valley-following behav- ior) as shown in Figure 2 (a).
Direction/Sign
23
+B
2. Microscopically, it must be able to es- cape from small-scale local minima (to maintain exploration capability) Fig- ure (b).
0₁ P.-1
g
+
+
V (())
+
8+ BR-1
0 On gi
ga
Almost all traditional gradient-based optimizers address the first aspect, but none of them address the second. Considering - [ we find is similar to -ga == - Egg for optimization but with repulsion from sharp minima. Moving along - - /| f(xs)l converges and tends to escape the sharp local minimum. The stable point may not be local minimum, but usually flat. The reason hides in
stable point 0
V = = -HH
where He is the Hessian matrix of f parameter Oμ. The only difference between -Bs and ((x)(())]l"" is the Hessian matrix Hç. The larger ||H:|] || is, the sharper the minimum becomes. g& enlarges the stepsize around sharp minimum, which helpful for escaping sharp minimum. In contrast, small -H, means flat minimum, and g is helpful for convergence to the flat minimum. Therefore, combining Es with H, in an optimizer should converge to a flatter minimum than using only -@k- However, directly calculating this direction is not affordable. Fortunately, we have
~ 0 0x-1
which holds with the positive constant omitted, and then we have
VIVi((O)) ~ (0 01-1) 8k
(2)
In fact, - is better choice for escaping sharp minima than VOVV/ Hn 1. Figure 2 (c) and (d) shows the 4 different stages when an optimizer approximates a sharp minimum and flat minimum, respectively. Without loss of generality, we assume that all directions of 02 0 are negative in the different stages. From Table 1, we find that the inner product between 02 0x-1 and some direction should be positive in stage @3 and ($ for accelerating the optimizer to escape the sharp minimum. E g is the only direction satisfying the condition. Considering cradient-based optimizers updating parameters along -ga, we should correct this to -BA + az (gi: gx-1) for escaping local minima and then exploring landscape where a' > O. The adapted optimizer is more likely be captured by a flat minimum. If the minimum sharp, the optimizer only needs to pass through a small opening with large gx and g: g which means large step size and thus makes easier easy to escape. Conversely, if the minimum is flat, meaning a large opening but small g: and gı 1 the optimizer will be captured. For stability and noticing more informative gradients at early stage of training, we replace g: g with their exponential moving averages (EMA) [19].
3

---

Main Idea (just for emphasis, connected to the context)
Therefore, we propose a gradient-based optimizer adaptor that can adapt any gradient-based optimizer for exploring better minima along valleys via simple replacement:
gk + aaa gk: where as 3 Bjai-1 + (1 B1) (gx gg-1),, a a=-a'
(3)
The term gk decides whether the optimizer is captured by large-scale local minima. For caş, if a 0, helps with escaping small-scale local minima and thus exploring the landscape: if a > 0, it helps with exploiting (accelerating convergence to) the aforementioned large-scale local minima. |ox] represents the intensity of exploration and exploitation. 31 reflects the persistence of the memory of the gradients-decaying decattigddddddmoooss directions. Based on (2), (3), and the preceding analysis, the advantages and limitations of the two cases can be inferred as follows (Remark 2)1):
1. a < 0 converges slowly (exploring large parameter space) but tends to flat minima.
2. 0 converges fast (exploring small parameter space) but tends to sharp minima.
Since our motivation is to adapt an optimizer for exploring large parameter spaces and finding flat minima, we mainly focus on a < 0. which very suitable for large-batch training. In fact, choosing >> 0 usually, but not consistently, leads to some marginal advantages (accuracy, fast convergence) in the amall-batch training case, but the corresponding minimum is consistently sharp.
Taking SGD, Adam, Blased-Lambb and Lamb [51] as examples, we adapt them into ESGD (Algo- rithm 3), EAdam (Algorithm 4), ALTO Vanilla (Algorithm 1), and ALTO (Algorithm 2), respectively, Similar to Lamb, we also employ layerwise regularization [51] based on the integration with Adam, and obtain Algorithm 2, where g E RRd: is the parameter corresponding to the i-th layer, with i [h] = 21,2. ... .}. Ei de d. & is zero-pun vero-proof function such that Φ(x) 2 usually taken as ((z) == + E3. All are very small zero-proof term, and X is a weight decay term for parameter regularization.
Algorithm 1: ALTO Vanilla
Algorithm 2: ALTO
Input: initialize 0,, learning rate EMA fac- Inc-ln the same as that of Algorithm 1
tors (81, 82, fa) [0, 1)s, stable parameter Output: {0x} T
k-1 do
E1 >> 0, weight decay A > 0, acceleration 1: while
factor |a| < 1/(1 (1). ag 0, m, 0, 2:
ge -zl C,Ez, (e St):
0, and go = O.
3:
a B1a- + (1 31) (gi gg-1);
Output: [0x)
4:
m = 82m4 + - B2) (gi + oag);
1: while k < do
5:
Vk farA-1 + +11 (1- Bz) [gk + +aaa]?
2:
gk 2v Ec,ez, V4 (0xx Ci);
6:
m. = ms/ (1 Bh):
3:
a 31a + (1 B1) (g: gK-1)
7:
VE = Vn/ (1 53
4:
m = 82m + (2) (gA + oag);;
FE m (V E1) + AgOE
5:
VE 3-VA-1 + B3) [ge oak?
nerk a (10%²
6:
TE ma/ (v + +1) + 10=
9:
((i)
k+1 k (1r[2 (10
(i)
000) mer (1) (10
7:
10: end while
k+1
(1r ||
02
0
8: end while
Why is large-batch training important, what challenges does pose, and how can our adaptor help? As data scales up and GPU computing power increases, enlarging the batch size is the most direct way to fully utilize as many GPUs as possible (data parallelism) and then to accelerate pretraining, However, this accelcration occurs under the condition that the total number of training epochs remains almost unchanged. This condition means large-batch training involves far fewer parameter updates than small-batch training but expects nearly the same test accuracy(for more details, see Section A). To overcome this challenge, Krizhevsky [23]. Bottou et al. [5] proposed the linear scaling rule (7) & [2x] 1) and the square root scaling rule (7)- x Z These rules are effective when batch-size is not large enough. As the batch size increases to infinity, learning rate becomes bounded [30] by a task-specific critical value determined by the geometry of the landscape. Otherwise, the training will explode. With the same learning rate, the exploratory nature of the

---

Parsmeter Ivin, batehdize 128, ManW
Paramoter Connerganc, batchutte 128, AdamW
1210
600
g
300
a
-0.
300
0.
2000
a 0.6
111:
750
200
250
100
103
150 Epocks
200
250
300
0
100
134 Epochs
200
500
110
150
200
250
300
Epochs
Top Beeslaan Eigrovalor, buichsice 10384, kamb
1500
2
[H|:
1000
100
200
400
Spochs
Epochs
Epochs
Figure 3: The variances of |2 (the top eigenvalue of the Hessian matrix), |0, 0ol (parameter drift), and ||0; roll (parameter convergence) during the pretraining ResNer20 on CIFAR100 with AdamW (1 128), Lamb (| = 16384) and their corresponding adapted optimizers with different values of a.
adapted optimizer provides it a larger effective learning rate (faster training). Its underlying principle is using remembered informative gradient information at early training stage (the larger the 3,, the longer the memory) guidance during later training stage where gradient information is flooded with noise. Meanwhile, the adapted optimizer is more likely to be captured by flatter minima than the original optimizer. Since the fluctuations at the bottom of the landscape are relatively small, the improvement in generalization may not be substantial, but it is stable.
Why is ag incorporated into g: rather than m Incorporating a, into m. suggests that g: -8=-1 on the same scale as -gt, which causes the optimizer to either vibrate violently or have no effect. Incorporating it into gi: means the EMA of the EMA of g: gg-1 in the final momentum, whereas incorporating into gA means a EMA of ga g in final momentum. These two are intrinsically different, we can not identify the EMA with the EMA of EMA just by adjusting 3. In fact, if we define EMAg as the EMA EMAR-1 with common coefficient for any k, the weight of g: g:-1 in EMA,, at time step k is (1 - 3) 3h 3h
(k-itn-) (K-
k
11 -
The larger the n is, the more stable the EMA,, becomes. We use n 2 for affordable computational cost. This means Algorithm 2 moves along the EMA, of ---- ga-1 near the bottom of valley. The EMA2 of gs gi 1 heavily depends on the early stage training where gradients decay very fast. Remark 2.1. I. In order to verify the advantages and limitations of positive and negative a, we adapt Adam W and Lamb (a 0 Lamb, a -5 ALTO) as examples for the small and large batch cases respectively. Figure illustrates that, compared to adapted optimizers with a >> 0, those with < are more likely to find flatter minima in larger areas but converge slowly.
Remark 2.2. On the constraint |ae| 1/(1 B1). If replace with 1, set DD ai 1/t = 1 2 At, and treat 0 as a 1 I-dimensional dynamical system updating via m without considering v and layer regularization, we have
Ô 0, 0:-1 m,
1
7 i + m 9), 9
à
a
a =
- 1 B2
0=
(4)
At
31
According to g = f', = g and Equation (4), we get the differential equation that @ satisfies:
+ + (1=
+ (1) -1)(111))f' = 0.
(5)
If we regard/ f' and f" as constant in Equation (5), it becomes a linear ordinary differential equation. For stability and convergence, the real parts of the system's eigenvalues must be negative. Therefore,
1
1
- I- + nh
< < ( 31 nf2
where Si = max f",04), -min(mi (ming J", For simplicity, we use |a| < 1/(1 - -31).
5

---

3 Algorithm Convergence Analysis
Prior to conducting convergence analysis for both non-convex [7] and convex [36] cases, require some common assumptions that are widely used in related works [52, 9,3]
Assumption 3.1 (L-smoothness)) The function Assumption 3.4 (Bounded parameter). We sup- ((O, () is I-smooth if and only if there exists pose that at any time step k, parameter On is constant such that
bounded i.e., ||0x|| - D and ||0x|1oc S Doo.
Assumption 3.5 (Monotonicity). We assume E [d].
|V () V + 8, 8,() - 1|
for all 0 E R. 8 E R. and & 2.
Vk (VVV, + +1) (|(=² (
+ 520 (108/)) ())
Assumption 3.2 (Unbiased, independent, and vari- ance-hounded stochastic gradient). We assume:
(100
Yk E N, Sk == Egg = VE [e (0x. C)]
2. Vk + E N, EA and gr are independent
increases monotonically with respect to k.
[)) VE,(0, 1)1l z a², 40 E Rd. Assumption 3.6 (Convexity). Assuming E(: () is
convex meaning VÇ E RR R 0,0' E TRdd .7 E [0, 1]. that at any time step k. gradients are bounded we have i... [|ga loo / /16.
Assumption 3.3 (Bounded gradient). We suppose
oP (0, C)+(1 - (1(00)())00))))))))
Remark 3.1. Assumptions 3.1 to 3.3 are common for the analysis of stochastic first-order methods non-convex [11, 12, 35]. Assumptions 3.3 to 3.5 (or its analogue) and Assumption 3.6 usually appear in the analysis of convex optimization convergence [36, 52]. Except for bounded factor, Assumption 3.5 and its analogue are common for convex case convergence analysis [52, 8]. If without it, the proof can not be finished. In fact, the optimizers in above literature and ours, also violating this kind of condition, can eventually converge during training.
Theorem 1. !f Assumptions 3.1 to 3.3, μ = A1-0,Gre < 1, = b-
(Gx = Ave
X(1 n)* o 1/2 S
-- and T O (GLSE-2) -2) hold, Algorithm satisfies: GdL
E
VI (0x)l s 462,
T+1
where
A
fa( = Ec(((0 C)] +
00 v
1101 == (0, (v + +)
2
the above conditions, Assumption 3.4, and T 2 O Dc holds, Algorithm 1 satisfies:
T
M E (IVF(0W)I)) s 6c?.
T+1
Remark 3.2. The proof and detailed version considering more hyperparameters is given at Section C.2. There, we change 33 to 1 3, for all E [3] for brevity. Compared with Lamb [51], we use different analysis method and lead to similar result under weaker constraints in more hyperparameter cases which are omitted Lamb for simplicity (Table 2).
Table 2: Convergence analysis comparison (Lamb vs. ALTO)
T>
b b
n some fs = Or more general
Lamb ALTO
0
(
O
(
0
(
Theorem Suppose that Assumption 3.3, 3.4, 3.5, and 3.6 hold. Let mp
" VE
S
dE
S } Vi'
vk
and
< 1. Algorithm 1 achieves the following guarantee
R(T) < 0 0.5 Good
0 (VT).
where R(T): = Et_ (64 (0x) Ex (0)))

---

Remark 3.3. For the proof and more accurate estimation, see Section C.3. We obtain the same main term (VT) as Adam when T oo up to an additional factor vd without requirement
1, which is due to the factor (11+ tez0 0
F1, = B&A and a different requirement
in
ALTO.
Remark 3.4. Although these theorems are only proved for Algorithm 1, analogous consequences also hold for Algorithm 2 up to bounded factor of Bi, i E [3]. since the bias correction is bounded between 1 and // (1 3t), similar statement in [29, 8].
4 Experiments
We mainly focus on large batch training problem and evaluate the Lamb-adapted optimizer ALTO. In the small batch case, there is also improvement but it is marginal Table 8. We compare ALTO with typical optimizers, including SGD, Adam, W [28], Lamb, and AdaBellef [52] across various tasks (CV, NLP, reinforcement learning (RL)), diverse datasets (CIFAR-10, CIFAR-100 [24], Ima-
geNet [15], CoNLL-2003 [38], IMDB [2], MRPC [46], and GPT-2 Output Dataset [1]. various archi-
tectures and scales 1ResNet18. 18, ResNet20, ResNet34, ResNet50 [15], VGG16 [41]. DenseNet 169 [18]. LSTM [16], BERT [10], and GPT-2 [34]), distinct environments (Swimmer, Ant and Humanoid [44]), and different batch size (from 32 to 50K). The following analysis mainly focus on CV and NLP tasks in this section, and ALTO is particularly suitable for GPT-based generative models due to its large effective batch size. Additionnii, we also conduct experiments on RL and LSTM tasks(Section D.1). Finally, ablation study and hyperparameters tuning experiments show the influence of ALTO's s im- portant terms and hyperparameters on optimization results, in order to demonstrate the necessity of their introduction and the relevant statements or explanations former section. For fair comparison, we conduct experiments on the same conditions, which would be different from those in their own original papers. Therefore, this may cause a little difference in experiment results than what are reported in original papers (See Section D for more experimental details). Of course, we also conduct some typical experiments under the same conditions as their original papers. In all these experiments, the best results of different optimizers are in bold. Each result is the mean of three independently repeated experiments.
BATCHBIZE
BATCHSIZE 18384
Table 3: Test top-1 Acc. (%) on ResNer20 with CIFAR1O and CIFARI00 and on ResNet34 with ImageNet (for hyperparameters and architecture
01A610
details, see Section D.2)
SGD
Amm
Adam
Dataset
CIFARIO
CIFARI00
ImageNet
Adamri
Adadel
Batch Size
128
16384
128
16384
256
4086
Lamb
Lamo
SGD
91.85
80.86
64.93
44.20
70,64
49.35
ALTD
AITO
Adam
89.88
87.34
64.35
54.91
65.06
54.96
200
1.30
AdamW
90.54
82.29
64.62
52,95
68.40
Lamh
90.89
83.56
61. 29
56.06
69.17
70.34
AdaBelicf
91.12
88.03
64.44
52.94
70.12
70.18
ALTO
91.24
88.83
65.74
57.78
69.95
70.83
Table 4: Test top-1 Acc. (%) of different optimiz-
ers for ImageNet training with RESNET-50 using
Asam
different batch sizes in 90 epochs. is reported in [51]. (for hyperparameters and architecture de-
23
Lamo
Lamo
ACTO
tails, see Section D.2)
Ratch Size
2K
T6K
Figure 4: Test top-1 Acc. (%) on CIFAR-10 and
Adam
13.08
73.08
73.32
73.11
73.09
72.50
AdamW
74.93
74.65
74.40
74.10
73.57
CIFAR-100 with batch size 128 and 16384 on
Adahelief
73.32
73.48
73.41
73.14
73.00
72.89
ResNet-20. The x-axis is epoch (for hyperparame-
Lamb
77.06"
77.111
76.921
76,89'
76.66*
76.421
ALTO
77.22
77.25
77.35
77.10
76.87
76.70
ters and archetecture details, see Section D.2).
Machine configuration. All our experiments were conducted on single node equipped with 4 NVIDIA 80GB 1 GPUs interconnected with PCI-E3.0. We remark that multi-node experiments were not performed due to our limited hardware resource.
7

---

Table 5: (a) Test top-1 accuracy (%) of ALTO and Lamb with batch size scaling. (b) Training time (s) with batch size 16384 to achieve the same test top-1 accuracy. Both (a) and (b) involve training VGG-16 on CIFARIO0 (c) Comparative performance (train loss and test perplexity) of optimizers in training the GPT-2 (345M parameters) Model with batch size 4096 on Megatron-L.M Framework by the OpenAl's s
GPT-2 Output Dataset (1M).
(a)
Batch Size
(c)
Loss
PPL
200
500
TK
2K
5K
T0K
25K
50K
Adam
4.43
87.74
ALTO
66.9
63.35
59.79
70.43
67.55
64.51
59.19
49.69
Adam W
4.43
86.51
Lamb
67.3
63.3
59.82
70.24
67.05
62.92
57.79
39,91
Adabelief
5.17
182.80
(b)
20%
30%
40%
50%
60 %
Lion
4.66
110.54
ALTO
137.103
202.674
333.817
482.842
608.023
Lamb
4.39
83.13
Lamb
195,992
276.694
409.277
582.210
864.669
ALTO
4.33
78.37
Hyperparameters. Though ALTO introduces five extra hyperparameters compared with Adam, usually and only adjust parameter 31 and rj according to batch size. It is clear that the larger the batch size is, the larger the and B1 should be. Hence, we set a 0.5, B1 0.01 in small batch training (batch size <1K) and a = 5, 31 = 0.99 in large batch case (batch size 21 unless otherwise specified. If not mentioned, we set 32 0.9, B3 0.99. À 10_ E1 10 E2 10-6, E3 10 10, These parameters allow ALTO ample room for performance improvement. We only adjust 31 and r7 for ALTO, while for other optimizers, tune all hyperparameters.
4.1 Image Classification
We conduct experiments on a variety of convolutional neural networks [26, 25] (CNNs) and the datasets mentioned above. Due to space limitations, we only give a part of representative experiment results here (see Section D.1 I for more experiments and details). As ALTO is tailored for large-batch training (in case 16384), it not only outperforms all listed competitors on these three datasets, but also achieves a better (esult(70.83) than SGD (70.64) on a relatively small batch (256) on ImageNet. There is widely accepted view that if the same learning rate is used, small batch SGD typically converges more slowly than the Adam family (Adam, Adam W, AdaBelief), but often achieves superior convergence result [52, 28] (Figure 4). This may be because the loss of large batch better approximates the landscape. In contrast, ALTO beats them only on CIFAR-100 for small-batch (128) case (Table 3). The larger the batch size, the larger the advantage of ALTO over other optimizers (Table 4, Table 5 (a), Figure 12 and Figure 13 in Section D.I.. We visualize the training process in Figure 4. As ALTO uses history information as to guide the training in later stage (after epoch 50 and 100 for small batch size 128 or epoch 100 and 200 for large batch size 16384) where gradient information is noisy, outperforms other optimizers a greater extent when the batch size is larger, implying more accurate gradients-decun batch cases (16384), ALTO achieves the best accuracy of other optimizers in our experiments using only half the number of epochs. Honestly, due to the extra accelcration a, ALTO's epoch computation time is longer than that of Lamb. Considering these two factors, we find that ALTO leads to less training time to reach a given (ecuracy(Tablee 5b)).. For more details, see Section D. Finally, ALTO as an optimizer developed based on Lamb, it outperforms Lamb in different batch size cases (Table 4) at nearly any training stage (Figure 4). For additional experimental details, refer to Section D.2.
direction (gx - information a containing. In large
4.2 NLP
Transformer-basedd attention neural networks [45] are heavily used in natural language processing. To demonstrate ALTO's ability to train popular large language model (LLM). we have employed two of them. One model we mainly focus on is GPT-2 with 345M parameters on Megatron- Framework [40] for pre-training tasks in GPT-2 Output Dataset (Table 5(c)), and the other, introduced in Table (Section D.1) for experiment completeness, is BERT base with 110M parameters for fine- tuning tasks in three datasets, where we also observe that as the batch size increases, ALTO's advantages become more pronounced and stable. We choose these two relatively small LLMs, due to our limited available computational resources. In the pre-training task, ALTO achieves an obvious generalization advantage (78.3 test perplexity) over other optimizers (Table 5). This is due to the massive equivalent batch size, batch size (4096) sentence length (20-30), caused by the GPT
8

---

task, where every token needs to be predicted. Therefore, ALTO is suitable for pre-training current generative LLMs. Compared with recent proposed optimizer Lion [6], ALTO achieves its final perplexity using only one-third of its iteration (also epoch) count, though ALTO requires 332ms and lion requires 253ms per iteration. If ALTO is applied to L.L.M training. another concern the additional GPU memory overhead. Due to the lack of computational resource, directly measuring extra GPU memory overhead on real LLMs like GPT-4 is unrealistic, but we can estimate it via the undetermined coefficients method, about 2% more than Lamb, shown below, Compared with Lamb, 1 d=s da -1 cxbteeg ? where d cydbtegd
the memory consumption of ALTO increases Lamb menurry
ALTO memory
is the number of parameters, b is the batch size, and C1, C2 some coefficients to be determined. The reason is the extra term a leads to memory consumption d, the forward and backward process for every sample and parameter lead to a consumption e db, and intermediates like m, v, etc. lead to consumption C2d. According to the above formula cybtrz and Table 6 (b), we find this growth rate is irrelevant to d. Taking BERT model based on transformer, as example, we have Table 6 (a) with d fixed and Table (b) with b fixed. According to the data of batch size 32 and 1024 in Table 6 (a), we have C1 = 0.048 and = 0.7877, and find it fits well if batch size is 512. For experiment details, see Section D.2.
Table 6: The comparison of GPU memory consump- tion (MiB) of ALTO and Lamb (a) using BERT buse with different batch sizes. (b) using batch size 1024 on BERT base with varying degrees of parameter reduc- tion.
Mecin
(a) Batch Size
32
512
1024
Lamb
366366
109006
ALTO
7869 (+13,31%)
60822(+3.73%)
111228
(+2.019)
(h) WP MPremmm
44506 145
69789951
ST1515977
109117005
Velue Zyba Rates Soe 20182
Lanb
34776
59259
$4340
108966
Alph: Ratea
ALTO
36968 (x2.27%)
60605 (-2.999)
89946 (22022))
181220
Table 7: Results of ablation study (%) on CIFAR1O (batch size 16384) and MRPC (batch size 2048) the test dataset.
CIPARIO
MRPC
0301
Festure
Topo Acc.
Avo.
Recall
Avg.
ALTO
89.83
14.35
$3.85
75.48
12.32
$1.45
$7.00
-1.23
7066
$3.31
70.34
95.11
16..5
-135
97.55
-3.47
Figure 5: The distribution of top-1 accuracy with different (31, a) after ResNet- 18 trained
80.51
96.22
2.01
12:85
81.07
74.99
91.00
1801
-158
10 0..
88.11
0.72
68:44
8064
69
65.12
16.46
1.9
180 epochs on CIFAR-100.
72.55
34.37
1080
-331
4.3 Ablation Study.
To demonstrate the necessity of each component introduced in our algorithm, we remove the accel- eration terms a in m (am) and v (a,), the bias correction term 1/(1 - -1-+) ) E 2,3 (be), and the layerwise learning rate regularization factor as introduced in Lamb (Irr). We then conduct an ablation study (Table 7). The results show that all these components are indispensable, and that ALTO and Lamb complement each other well.
4.4 The Choice of 81 and
According to Equation (3), 3n measures the persistence of exploration, while |a| determines the scale of local minima that can be escaped during the exploration as discussed in design part. Experimental results reveal the effects of 31 and a on ALTO's performance (Figure 5). Generally, larger batch size requires a larger 31 and smaller a. For large batches, we set a to be negative for larger exploration range, flatter minima (Figure 3), and this leads to a better test performance. However, for small batches we set it positive. Although a negative a suggests flatter local minima, the resulting improvement in generalization is insufficient to offset the usually lower training loss achieved with positive a. If a flat minimum is desired (Figure 3), setting to negative value is also acceptable.
9

---

Related Works
Lion [6] using double momentum method computes the EMAg of the gradient (gt), while ALTO first computes an EMA of acceleration (gx 9k-1), adds the EMA to Jk, and then computes EMA of the addition. Moreover, ALTO uses a negative a, whereas Lion uses a positive effective a. Additionally, ALTO considers the second moment, while Lion does not.
Adan [48] also uses EMA of Ok - 9k-1- However, there are two major differences. First, Adan introduces the FMA of 9t 9h-1 directly to the momentum in update equation, but we introduce it as an acceleration to gradient Jk- This means we use the EMA of the EMA of gA 9k-1 to estimate the momentum in the update equation which is more natural in form. Second, in large-batch training, to accelerate the optimizer for exploration, its hyperparameter a is usually set to a negative value, such -5 in our experiments. In contrast, the equivalent a of Adan is positive.
Indeed, ALTO requires more memory and computation than these two optimizers per iteration (Table 12 in Section D.I), but ALTO can achieve higher accuracy (Tables 10 and I in Section D.1) 1) and require only one third of the number of iterations compared to LION reach a reasonable perplexity (Figure 9 in Section D.I). This implies that the training time required to reach an acceptable pre- training perplexity (PPL=200) using ALTO is 56.2% less than using LION (Section D. However, when compared with matrix
optimizer like Muon [21] and Shampoo [ [14] with computational complexity (NN (N, the number of parameter), the extra computation related to is much less, with a complexity of ((N).
A class of recently popular optimizers, which reduce the variance of the loss function :=
V (0,()) EE( [4), related to our adaptor design idea that reduces the norm of gradient
(0) The relationship between dh (0) and [[)]( is given by [V
ox (O). if we define (0) == (((0)-(((0) [[]
called horizontal amplitude at and assume &| (0) + 0
at 0. The optimization objectives of the two methods are the same up to standard deviation of & From the definition of the horizontal amplitude 6;, we find EE(() (0) + 6g ( x
(0 + (0) [[)]] ). This means if we regard the graph of (() as translation of the graph of
(f,(
l (0), the smallest translation length should be dg. The d₂ measures how much the graph of Cu (0) oscillates around the graph of ())
Apart from first-order optimizers, method "E" can also adapt zero-order optimizers [27]. second- order optimizers [49, 17] and matrix-based optimizers (Muon, Shampoo), by replacing gradient estimator ge with gk oaş (Algorithm 5).
Limitations and Future Work
The introduction of two hyperparameters (3, and a). The optimal values of the two hyperparame- ters vary for different tasks, origin optimizers, and platforms. We use one default setting, which may not be the best but consistently well-behaved under various conditions. In the following process of maintaining the adaptor "E" and optimizer ALTO package, we will further improve the parameter settings.
The improvement is limited in small batch case. Small training batch size usually leads to approximately optimal test performance, which limits the potential for improvement. Meanwhile, the gradient estimator in small batch case noisy, so we should not use a large B1, which limits the effect of adaptor.
Conclusion and Results
We propose gradient- pradeent-based optimizer adaptor, which can make the optimizer continue exploring the parameter space along valleys rather than circling around the minimum that traps the optimizer. We use as suitable tool for latch-batch training tasks. The large-scale exploration capability of the adapted optimizer can mitigate the constraints imposed by unadjustable learning rates. We conduct experiments on typical large-batch training tasks, and the adapted Lamb (ALTO) outperforms current top optimizers, especially in NLP tasks, since their effective batch sizes are extremely large.
10

---

8 Acknowledgements
This work is supported by the following funding: National Science Foundation of China (92270206, 62032023, 62372435, T2125013) and Huawei Technologies Co., Ltd. The model training was performed on the robotic Al-Scientist platform of Chinese Academy Science. We extend our sincere gratitude to Xirui Yang for insightful discussions during the conceptualization phase of this research. Special thanks to Dr. Jianchao Tan at Meituan Co., Ltd. for deploying ALTO on their internal large language models and utilizing it for pre-training, whose expertise in enterprise deployment was crucial for practical and industrial verification. Finally, special thanks to Jiacheng Li for his discussions and core contributions in experimental implementation of computer vision, natural language processing, and reinforcement learning tasks.
References
[1] GPT2 Output Dataset.
2023. 7
[2] IMDB Dataset 2023 Version. http: //wp:///wwwwww// httt:://wwwiimmb.omm. htt.a...oo//ttteefccsss///// 2023.
[3] Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. In International Conference on Machine Learning, pages 404-413. PMLR, 2018.
[4] Vineeth Bhaskara and Sneha Desai. Exploiting uncertainty loss landscape for stochastic optimization, 2019, URL MRL.httss////xiiv.oom hwt.://ggg/bb///e/eeeee abs/1905. 13200. 10
[5] Léon Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning, 2018. URL https: //aps://///xvvvv org/asg////./ooooooo.. 606606601 04838. 4
[6] Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V. Le. Symbolic discovery of optimization algorithms, 2023. 9, 10
[7] Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of class of adam-type algorithms for non-convex optimization. arXiv preprint urXiv:1808.02941 2018.
[8] Yineng Chen, Zuchao Li, Lefei Zhang, Bo Du, and Hai Zhao. Bidirectional looking with novel double exponential moving average to adaptive and non-adaptive momentum optimizers. In International Conference on Machine Learning, pages 4764 4803. PMLR, 2023. 6,
[9] Ashok Cutkosky and Harsh Mehta. Momentum improves normalized sgd. In International conference on machine learning, pages 2260 2268. PMLR, 2020.
[10] Jacob Devlin, Ming-Wor Wiig-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 1810004850 2018.
[11] Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341-2366 2013. 6
[12] Saeed Ghadimi, Guanghui Lan, and Hongchao Zhang. Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming, 155 (1-2):267-305, 2016. 6
[13] Marco Gori, Alberto Tesi, et al. On the problem of local minima in backpropagation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(1):76-86, 1992.
[14] Vincet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization, 2018. URL
haarg//bb/////// 09581/1688... 10
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016. 7
11

---

[16] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98):
1735-1780, 1997. 7
[17] Siyu Hu, Wentao Zhang, Qiuchen Sha, Feng Pan, Lin- Sinn Wang, Weile Jia, Guangmng Tan, and Tong Zhao. Rlekf: An optimizer for deep potential with ab initio accuracy, 2022. URL https: paaps://arxivy
ht//bbb//2222/ 06989. 10
[18] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708. 2017. 7
[19] Stuart Hunter. The exponentially weighted moving average. Journal of quality technology, 18 (4):203-210, 1986. 3
[20] Xianyan Jia, Shutao Song, Wei He, Yangzihao Wang. Haidong Rong, Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang, Liwei Yu, al. Highly scalable deep learning training system with mixed-precision: Training imagenet in four minutes. arXiv preprint 1rXiv:1807.1 2018.
[21] Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng. Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https:/ / gi thub. wo/posts/mutt /mnon/. 10
[22] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
[23] Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks, 2014. URL https://arxiv. org www.oogggg/////eeeeeeeeeeee abaaaaas//// 5997. 4
[24] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[25] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[26] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (((11)2227-2333 2324, 1998. 8
[27] Sijia Liu, Pin- Yu Chen, Bhavya Kailkhura, Gaoyuan Zhang, Alfred Hero, and Pramod K. Varshney. A primer on zeroth-order optimization in signal processing and machine learning, 2020. URL tps:ps//// /arxiv.vvvv org/ brgg/bs/// /2066 20024.6222........ 06224. 10
[28] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 7,8
[29] Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. arXiv preprint arXiv: 19Xvv:9902.09843, 2019.
[30] Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of larce-batch training, 2018. URL https:
www.oorgg/aa////ooee/eeee/ 06162. 4
[31] Katta G Murty and Santosh N Kabadi. Some np-complete problems in quadratic and nonlinear programming. Technical report, 1985.
[32] Myle Ott. Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation, 2018.
[33] Raul Puri, Robert Kirby, Nikolai Yakovenko, and Bryan Catanzaro. Large scale language modeling: Converging on 40gb of text in four hours. In 2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), pages 290-297. IEEE, 2018.
[34] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 188)99 2019. 2, 7
12

---

[35] Sashank J Reddi, Ahmed Hefny, Suvrit Sra, Barnabas Poczos, and Alex Smola. Stochastic variance reduction for nonconvex optimization. In International conference on machine learning.
pages 314-323. PMLR, 2016. 6
[36] Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv preprint arXiv: 1904,09.00 2019, 6
[37] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.
[38] Erik F Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Language- independent named entity recognition. arXiv preprint cs/0306050, 2003. 7
[39] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 063477, 2017. 44
[40] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron- Training multi-billion parameter language models using model parallelism. arXiv preprint 1909. 2019. 8, 39
[41] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409./54 2014.
[42] Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoe V Le. Don't 't decay the learning rate, increase the batch size. arXiv preprint arXiv: 100.0000099,.0.0 2017. 2
[43] Ruoyu Sun, Dawei Li, Shiyu Liang, Tian Ding. and Rayadurgam Srikant. The global landscape of neural networks: An overview. IEEE Signal Processing Magazine, 37 2020. 2
[44] Emanuel Todorov, Tom Erez, and Yuval Tassa Mujoco: A physics engine for model-based control. In 2012 JEEE/RS/ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012. 7
[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL htttttss:/pprroceedigeeo inwp:eeedddinnnnnneiiiieeee neur ips. wtt.cc//pppaoa wwtpppperee.... http:paeeaeleppppppppeeeee wtt.fees/paaeee////////.. Paper .df. 8
U. Von Luxburg,
[46] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 18800.00761,, 2018. 7
[47] Jiyy Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Dubureq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforcement learning library. The Journal of Machine Learning Research, 23(1):
2022. 45
[48] Xingyu Xie, Pan Zhou, Huan Li, Zhouchen Lin, and Shuicheng Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models, 2023. 10
[49] Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael W. Mahoney. Adahessian: An adaptive second order optimizer for machine learning, 2021. URL https://aaa aiv. orwoooog//oo.ooooooooooee /a//abb///00.. 00719. 10
[50] Yang You, Zhao Zhang, Cho-Jui Hsieh, James Demmel, and Kurt Keutzer. Imagenet training in minutes. In Proceedings of the 47th International Conference on Parallel Processing, pages 1-10, 2018.
[51] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.000. 1904.0092. 2019. 2, 4, 6, 7
[52] Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems, 33:18795-18806, 2020.
6, 7.8
13

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: [TODO]
14

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't ' acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. wttpccccppcccccceccc ceaiic/gguiesss.eee CodeSubmi ississss onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
16

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:[Yes Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
17

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ht /ttp://eeerippeeeeeeeee
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper poses such risks.
18

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi code.. assett has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [TODO]
19

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
seteurrpps.cc/onfeeeenee///////e /LLM)
20

---

Appendix
The Challenge in Large-Batch Training
22
Adapted Optimizers
C Convergence Analysis
23
C.I Preliminary
23
C.2 Convergence Analysis of ALTO for Non-Convex Optimization
24
C,3 Layerwise Convergence Analysis of ALTO for Convex Optimization
31
Experimental Details
38
D.I Supplementary Experiments
38
D.2 Achitectures and Hyperparameters
46
D.3 Hyperparameter Tuning...
50
21

---

A The Challenge in Large-Batch Training
For clearly explaining the challenge of large batch training, we conducted preliminary experiments on a simple regression task using basic neural network architecture.
Loss Curves
Loss
Loss Curves
Tor L0G5
Losa Vol Loea
Than Loss
Vol Los6
val Loss
2.-
15-
1.00
0.75
830
0.5
0.29
0.0
2000
2000
500
3000
1500
2000
2500
3000
3938
4000
500
3000 1000
2000
2500
3000
2003
4000
Figure 6: training with batchsize 32768 of Adam
Curves
Loss Curves
0.8
Val Loss
Vall Loss
0.-
Vol Loss
0.4
0.4
0.3
0.2
0.2
0.1
0.3
0.0
0.0
100
500
300
200
300
400
500
Figure 7: training with batchsize 256 of Adam
In this task, constructed three-layer fully connected neural network, with the final layer consisting of a single neuron to fit the regression task: y=sin(x), where the model input is x and the output is y. For this simple univariate function fitting task, our training set was constructed with uniformly sampled points from y=sin(x), where x ranges from -10 to 10, with a total of 32,786 0.8 samples. Noise with a mean of zero was added to the corresponding y values. In the dataset, 80% was used as the training set and 20% as the validation set.
We conducted total of six training runs, dividing them into two sets based on batch sizes: three runs with a batch size of 256, and three runs with batch size of 32,768. The resulting loss-epoch graphs are shown in Figures 6 and 7. It can be observed that in all instances of training with the larger batch size, the loss-epoch graphs exhibited a 'staircase' pattern, characterized by a sudden drop in the loss value a certain point. This phenomenon was not observed in the training runs using the smaller batch size. It is noteworthy that in this study, we did not employ a learning rate scheduler, but rather trained consistently with a fixed learning rate. Therefore, the occurrence of this phenomenon warrants particular attention.
In a common training task, we would stop too early to the 'sudden step' observed in the training with larger batch sizes. Training that number of epochs consumes too much computational resource. Notably, this phenomenon occurred Therefore, in experiments involving more complex tasks and models, it's possible that this 'sudden step' will occur. Please notice that the training finishes using 20 or even less epochs in small batch case. In contrast, the training finishes after around 2000 epochs. This means 32786-batch training requires more than 100 times of epochs that required by 256-batch training, meanwhile 100 32786 In fact, is impossible to train so many epochs (no acceleration in large-batch
in such simple task with basic network architecture.
approximate 256
22

---

training). The only method is to enlarge learning rate with same proportion. However, the batch size can be infinitely enlarged, whereas the learning rate can not.
B Adapted Optimizers
Algorithm 3: ESGD
Algorithm 4: EAdam
Input: initialize 01, learning rate 7k. momentum Input: initialize 01, learning rate 7kk momentum
factor (3,, 32) [0, 1) acceleration factor |a|| < 1/(11 - B1), ag = 0, and m. =0. O.
factor (31, B2, 3g) E [0. 1)9. stable parameter E1 > 0, acceleration factor |a| < <1/(1 31),
Output:
TT
ag 0, mg 0, and Vo 0.
Output: {0x 7
while k < do
k=1 I: while T do
2: gk -2l Ex,e,, VE (0x: S:);
3: a 3/a; 1 (1 B1) (g: gk-1)
2:
gi [2xl 2 Cc,z, Ve(0%: C:):
4: m2 = 82m + (gk oa;):
3:
a₁ = B,ax-1 + (1 - 31) (gk gx-1);
5:
0x+1 0g mm
4:
m& 82m
(1 $2) (gk + oak);
6: end while
Va 8/VA-1 + +(1- (1- 3s) gg + oay 2
6:
m. m2// (1 B+)
Vk Va/ //(1111 5/)
8:
Tk = m (V + E11
9:
O 0k+1 O2 177
10: end while
Algorithm 5: Generie form E-adapted optimizer
Initialize 0, gradient estimation operation <6-). standard origin optimzier updating operation (.),, number of iterations T, and learning rate ты iteration k. for k = 1, 1,2,... I do 1. Gradient estimation:
g+ =[ (,,)),,,)
(6)
where 2g denotes set of Gradient replacement:
stochastic samples used at iteration k;
gk
gi: oak, where a (1aa-1 + (1 B1) (gx gk-1),
3. Standard parameter updating of original optimizer:
0, = 1/ (gk, gt-1,.... .gi. 01, n)).
(8)
end for
C Convergence Analysis
C.1 Preliminary
Before starting the proof, we first provide all notations here for looking up. Let
1. (x, (vVx + +1)) /vv"" = (x, (v + + v).
2. u2 == B2 + (1 -32) gk,
3. ng 82ng-1 + (1 (2) ak:
4. m = + anz
5. P = m VK + E1)
6. Eg & + Ax7
7. 0, = (Vvx + $1) 0:
23

---

C.2 Convergence Analysis of ALTO for Non-Convex Optimization
In this subsection, we give the convergence analysis of ALTO in non-convex case for Algorithm 6.
Algorithm 6: ALTO Vanilla
Input: initialize 0,, learning rate YJ,, momentum factor (81,82, .3) E [0. 1), stable parameter
61,62 > 0, acceleration factor |ax| < 1/81, weight decay A > 0, ag 0, mg 0, and Vo
Output: {0x}-- 1.
I: while k < T do
2:
=,, 2C,E2 (e(0:: Ci);
2x|
1.
3: a
-1-B1) ak-1 + $1 (gx gk-1)
4: my
(1 -B2) m + (g(k + oaa)
5:
(1 [18 v + $g [g + oas) 22
6:
Tk ma/ Vv =1 + 10
mur " [10
7:
0
[1rk 0 0
8:
end while
Lemma 1. If Assumption 3.2 holds, we have
σ²
E
gk
gk
b
Proof.
E
gxll
1x 2
Σ
8k
1x
6,E22
Σ
[1 C.) BE(0,
Z 2
S,EZ2
6²
Lemma (Bound for [0x+1 0z)l and [1 0x)1l (m
T/03 E
17²
SE m + 1x /|2
172
m 11
E
@x+ill < E m 2x
G2 d
E? + A
EGE?
T2 m
E
< <E
0k+1l s SI
11|mm A401 2
17²
SE m:
1r ||
n7
God
(82 + Awn)
39
Proof. According to the update equation
(
( k+1
000
(1)
v/rk
(i)
(0(3))
rx
+ EgΦ
have
(10(
0x + m
rx
+ E2Φ
Ag0 + Pk
(0x. 0x+1).
V + 61
g/o (1
24

---

Then, compute the second moment on both side
no (1e xx
AgO2 m.
E ( 0
0x+1)1
VVA + E1
(() 1k
+ E2 0
< ma dnD 1
1/2
12 1/²
S EE
m+ + 0x
of (62 2w1)
6783
we find the following estimate
[ (10())
no
3
7 &1 (&2 + A
S
G vd
Vk
+ E1
(8) Fk
+ EgΦ (1055))
6162
based on Lemma 10, switching B2 and B3 pair with 1 - B2 and constant omitted, we have
33 pair, then with unrelated
32
d(1 - 33)
vd
(8)
E3
33 (1 - fB) (1 11-33) B32²²)
E3
Thus,
n'e3 E m - 200el²
ms + 1x0x 2
||
n?
0E
0x+1l1 s E my + 11
S E
G d
E? (62 + Awn) )
69
n/
m 2101l'² S 0x+il S m + 2m
172
m + 2v || S
n2
G
&1 (82 + dRT))
2
E
Lemma 3. If Assumptions 3.2 and 3.3 hold. We have: ||ue|lo S Goo, || Goo, Volllo S
G2
Proof. By the definition of ak. Ug n2. Vk. we can have that:
n. 32
(1 B2) k-t a:
k
V4 83
Σ
(1 B3) k- 8a)=- (g: + ara))
1=1
k
Up 32
(1 3z) /2 k-t B:
1=1
a 31
(1 81) k-
) (g. g(-1)
1=1
- Σ
31 (1 1-8 a 1-
gi +
Considering |a,81 < 1, get:
25

---

Go
2
aak
20 -
2
B1GG
| =
2
| = =Ilus + +anill S Goo
|/velloo S G
Lemma 4. If Assumptions 3.2 and 3.3 hold, we have:
(
VV + (v/==++))
+
- 1 Gr
which implies
Ak+1
A++1 |0
110+ta dx 110
VK+T
1
Proof. Give any index i E [d] and the definitions of Vk, we have:
VV v ( -
+
Vk-1 VVk
VV tE
Note that, by the definition of Vk. we have:
M ( Vv
V - VA & +
VVk +
Vk-1 (ge + aaa)" )
G
BB
VR +
8
Lemma 5. Consider a moving average sequence:
uy (1 (/2 + 82gk:
(9)
Then we have:
33)LE (lue-1 11- (1- 22
(1-B2)21²
E | -ga113) Bx.l2)=(( <(1
E (10k-11 10:-1-000 -bal²) + 3302.
B2
Proof. According Equation (9), we have
u -g- (1 32) (u ga-1) + (1 (2x 8x) + + (gr -g)
26

---

Then, take the second moment and then expectation on both sides:
E ||ue Bx|1
(1 82) (1lug-1 BR-11(²) + (1 - +1-3)))²B (IBL-1 - -Ba13) + 3302
b +
2(1- - B2) ((ug-1 - Bk- 8k-1 - (i))
$ ((1 82)² + (1-3)) IE (luB-11 Bk- 112)
[1+ _
3B0²
(1 B |18:-1-
BBll²) --xll
a
(a)
$ (1 32 (l4 - (1 + (1 [1=
B20²
E [g - (Bal') +
32
b
5(1 32) (lus-1 - -la--1- (1 L2
+ 3300
E
- (10k-1-0811)
B2
b
where for (a), we set
08
1-92-
Lemma 6, Consider a moving average sequence:
a (1 - 31)as-11 + B1 (gx- -k--1)
(10)
Then we have:
E ||ax|| S < - 31)E (la--1l²) - 3 (lBx BK-111 11² + 2,3002
Proof.
Take the second moment and then expectation on both sides of Equation (10):
E (lax!l") = (1-31)23 31) ||a--11 I|²) B [|g: Bk-1ll" + 23 (1 - 31) ((ax gi - gk-1))
= (1 - 31)²E (lac- + E ||g: -0x-gg-1ll g+-1l1²) + (1 - B1 )E ((a: Ex gk-1)) (11)
S (1 - 2 (las-1117 11)) + 33 B (IE 11 + 2,3 (1 - ) ((a 1 8. gk- gk-1)) + 2030²
b
(12)
< (1 - 3E (lax- + B (IBi Bx- |1²) 2,3 - B,) ((ak-1. 1. Bx gk- 1)) 2030²
b
(13)
2830²
$ (1 B (|an-111)) + 31 (IEx B
(14)
where for Equation (1I), we utilize the independence between g& and a while for inequality (12):
E (1EA gk 11²) 11) E [Ige Bill²) E (IEAA- B1-11(²) + E (IBi Bx-1l²).
for inequality (13), know:
E ((ax-1, 8h-1 gx-1))
E (((1- a1)ak-2 + 31 (gi-1 gk- 25-22) gk-1))
({(1 - - -188- Bk-1 8x-1)) + 31 FE (BB-1 - 8k-1 + -8 8: 8)-1))
-3,1E (lIBA-1 - 112) and thus
E ((a=-1, gt 1)) = E ((ak-1,8: - -B- - 3 -1B-111---111 11²)
- gx-1
Finally, for inequality (14), we use:
2E (( (a.-1, 1, Sh
B BB-1) 1)) < E (lan- 11²² + ||B. B -1-1)²²)
27

---

Lemma 7. Consider a moving average sequence:
ng. (1 f2])m $2ak:
(15)
we have:
[xll S (1 B2) (m==-11²) + 12 (Ila=l1).
Proof. Take the second moment and then expectation on both sides of Equation (15);
IE [|xx11²) E (I(1 ((20--2)) + 82axxl1)
s B2) (1m + 32B (IEA Bx-211?)
Lemma 8. Assume Assumption 3.1 holds, with 17 S 3L and then we have:
ÉZNE3 2 3dG.
oin
fa+1 (0++1) - fx (0x)
E
my + On
" B
Blga m
8162
6
Proof. Recall that the update of ALTO is the following
(
(K)
k
k++
(1)
[
+ E2Φ
Thus,
|0(00
Ox + me
rx
+ E2Φ
0g + P
(0, Ox+1).
(16)
Vv + E1
no (1 (3)
Using Taylor expansion, we have:
L
Ak+1
f841 (0++1) S E fa+1 (0x) + (Vl++1 (0x), 0x+1 - 0z) +
10
0xll2 +
10
11/ lv
2
2
Lemmna 4 I
L
d
la (0x) + (V (0k) 0x+1 - -0k) +
0 0xl1
+
110+11
2
2
(0x) + ((B... - 22, A6 + VVx + +
gk
+
L/E1 + d
10 - 0x11
(17)
VV=
= Sv
L/e1 dl 11100111
00 VVx +E
0k+1 0x: AnO
Pk + gk m.
2
/V2 + 61
/VL
Equation (16)
Sa (Ox) +
1L/1+t1p
+
E B1 0xl12 VVN + +E
0ktt -B(00t) Ok: g m
"
/V + 61
(0x)
L/51
&2
E100++ - 0x113 vv
E
+
& E 110 6.4
0x112
" 28
E l|g
m
||
"
27
(18)
S (0x)
E2 E 10 k1 0xll/ve +
17 EE 1lg m
(19)
3m
20182
Lemma 2 5
E2713 E
S (0x)
mE + 04
E ||g:
.mal
3dG 0c
28162
< fl (0x)
E
2
EE
+ 0x
2
+
77
E
||gk
ml
77
E
||gx
gvll
odG
8162
E162
Lermnn 1 2
E2
2
n
d'n
fu (0x)
mm + A4
+
gx
m
3dGxx
8162
6
28

---

and inequality (17) is from:
0 00-1 II 1l (10xa11 (100 +2(0++) 0x, 0z)
0x).v))v vv. 10 - 0x 0x(l(va))
and to obtain inequality (18), we utilize:
Ok+1 - Ok. g+ m
10++1 - -0x1 + 17
gg -mmml
VV& t
21
2
To get inequality (19), we use condition 1) S E1
3L
Theorem 1. Suppose Assumption 3.1, 3.2, and 3.3 hold, p VFaGo/ <1,
b o8
a² dG
118
18G
E r S
T 2 max
GdL
nEZEEES
B2E
where Do fo (0o) fō and Algorithm 6 satisfies:
m + 1x00 (
Σ
T
and
T
T
uu - Be
Σ
a2
n
T
4
T+1
4
k=
k=
Hence, we have
1
Σ E (IVFA(0W))*)
S $
T+1
Based on these conditions, if Assumptiin 3.4, and
DX2
T2 2 1²
we have
T
E (1V
< 66
T
k=
Proof. We have:
m Ball S 2 2//uuk gx 1x 2 +22 ||mll
By Lemmas 5 to 8, we have:
fet1 (0x+1) < (0x)
EZME3
my + xxx /|
+ E162
21
l|g. - u 11 +
2n
a²
|n 1 +
n
3dG
E162
6
(20)
(1 - 1-222 I2
330²
E (1uRt1 1 -B++2l²) S 5(1 1a-Ba B2) E (IuI - Ball? +
E
(10 - -2al²) +
B2
b
(21)
E (1n)+il12) ( (1-32) E [|mal13]
+ B21 (laz))*)
(22)
288002
E
(|a0+al?2
< (1 31) E ||a ||?
+ 3 E
[ - gx 12) +
(23)
b
Then by adding Equation (20) with Equation (23), we can get:
Equation (21),
no 325
Equation (22), and
no
X
29

---

E ( S (2x)
mi xrOal 0l12²
+
02n
7
(1-
32)² L2
10 0x11 + 3200
3dG
328182
32
+ 1102
28f02 b
|2LEE 10
0x112+
815182
$21133
my + xx 1|
n1
(11 ((1-
B2)²
(B2
2a2
+
no²
$ E (b))
E
+
+ a²
E 0kx+1
- 0x11 +
3dG
E162
52
b
E162
E
nL
+ Benjo
< <E(#))
E
mk + 1x
110+11
0x1
3dG
B3
e1
11
11
321E3
Beno
< (9)) +
E
m + X1 01
+
B3
3dG
E162
n 2 od
Beno²
E (0x)
EE'
my + 12
+
8162
where we let:
4x == fx (0k) - f%t +
1
Iu: -8x11
+
na?
|naI1 +
na²
||al
325162
825182
B1E1E2
B. 82 + 21 +1
E7E9=333
n²²
2dL
By telescoping sum, we have:
E ( Σ
E (
nE23
E
m
Acbill +T +
(T++1 1)
6dG cc k=
E162
k
km=0
Hence, can get:
Σ E (Ime + Se0x 1) <
T
odG To
GdGooBo02
6dGo + 6dG
6dG
Se,
T+1 kmo
nTEEc
E1E253
nTEE2
33
E1
where
2 25 2 Be S
18G Do
18
T max
18G o2
B2ERE1E2EB
From Equation (21), we can conclude that:
σ²
12
320²
2
Σ u Bx(1")
T+1
BOT
4
From Equation (22), we can conclude that:
T
(Imall3)
S
1
E
(laull²)
(24)
T+1
T +
k=
From Equation (23), we can conclude that:
1221222
2010²
(1aa1
5
+
(25)
EGEB
b
k=
Combining Equations (24) and (25), we have
T T
QZLL2722
2023/02
Σ
02
E
02
2
Σ
E
2
|
11
+
2/4
T+1
E7EJ
K=U
k=0
30

---

where
dG
a
81825503
Finally, for fkk we have:
B
T
A
Onl + EE (Oks <>] L
1
T
2
k=U
E
1x + 8h +m us -ams)
T
k=
1
m A2 1| + l BAll 4a² |m |maal²)) 1))
E
2
T+1
k=0
$462.
is bounded, we have
T
1
EAA 0gx |
T+1
k=0
Lemma 3 X2DG2
T
x Σ (1 11)
T++1 k=
X²DG2
Tp (2 - 14)
2²
For f.f. we have:
T
1
(IV
T+1
k=
(ma u ans) + Ax 1n0x 1²)
gx
T+1
T
E 2 m + 1c 12 +6 llu - Ball + 6a 1 12 + 626 [ox 1
$ ST+
A==
<6E².
C.3 Layerwise Convergence Analysis of ALTO for Convex Optimization In this subsection, we give the convergence analysis ALTO in non-convex case for Algorithm 7. Lemma 9. If Assumption 3.4 holds, we have
Tk (108))
S oT.
+ E2 (110500
31

---

Algorithm 7: ALTO Vanilla
Input: initialize 01. learning rate 7k, momentum factor (81, 32, .s) E. [0. 1). stable parameter
81 > 0, acceleration factor |ol < 1/(1 - ;31), weight decay Ax= > 0, ag = 0, mg = 0, and
Vo = 0. Output: (0x) while &s < do
Compute gi:
[2 Σc,ez, Ci);
ag = 3Tay-1
+ (1 (gk Bk-1)
m. 82mk- + +(1 (2) (gk + oak);
Vk faVk-1 (1 83) gg + aaxl?
Tk ma/ (v te) + 10
(1) (
mer
( 0 [10(0)]
k+1 k
(1r( 522 [10
end while
Proof.
Obviously,
(1 (100 1)
S Dx
and
Fk,i
S
(0(0)
rk
+ 0
rk
Therefore,
h
d2
2
h
rg.
kR
(2)
(3)
2
=
k
j=1 j=li=l
j=1
Finally, we have
(100(00
Fk, ok
$ vhD
[000
E2Φ
Lemma 10. If 1////3 < 1, we have
(5)
((())) 1)
rk +
62Φ
1- - Bg d3g
5
(1)
E3 V
(1 33) (3g -3
Proof.
m; (1- - (2)
gh gk-t
(g. + a,a))
1=1
V& (1 - Ba) Σ
3/ (g. + a,a;)²
Therefore, with unrelated constant omitted, we have
rk
(4)
+ E
m
(105
[
VVB
In fact.
32

---

(1 - B2) Σ/= $/ k- (gt, + agar)))
(1 B2) Σ 83 (git + a
Σ t= 2
gk-t (gt, + a(ar;))
1- 32
vI BB
t= B3-1 (gt,) a
E 3 gi, + aray,))) )²)
EE (33 k-t
1 1- 32
VI- - B3
Σi 3 (gi, + a)a
d
$ (1 - B2)
(1 Bz) (Bg (5))
Lemma 11 (Bound for ΣT (0" 0x. m²)).
(1 82) dD & VTG
d33
0 O2. m²) S
+ +AVTD&G>
ne3
(1 Ba) (Bs 132)
k=
+ TVThdD + An (1 + log DVT.D
Proof. We focus on the hh dimension of the parameter vector Os E Rd. From the update rules presented in Algorithm 2,
(10%)) 11)
0 0++1, = =Ok, 7,Tkii
+ c2 (1055)
rx
(10%%) |) (105501
ma.i
Oz.i
1p
+ AnOu..
Vi.i + 81
(1)
+
E2
Subtract the scalar 0) and square both sides of the above update rule, we have,
(
2
rk
(0.41,- -0,)² = (0z.i 03)² + Th
(1)
(10...1]
rx
+ Eg0
-0 07) 7k
m
+ AcO
/Vk,i +
rk
(6)
E
[0
33

---

Rearrange the above equation, we have
(4)
+ 62Φ ((055))
m2, (0x,s
02))
Vh,i E1
(10us-- 0¹x)²² -0x)²
- - -0:)²)
-(0k+1,
217
0
+ Au (v +1i) (0" On.i) Okii
rk (106/-1)
t
+
2
Fk.s
(4)
+ t20 (1088)
(8)
(100%1))
Vv,i &1
ri
+ $20
((Bws 07h2² (0k+1,
- 07)))
217
-110%%))
+ A Vv + 51) (0z 0k.z) Ox.
TR (10(())
+
I/k
mg.i + Xo xx.
2
1x
(1)
+ E2Φ
|0(0)
Obtain the regret upper bound via summing the above equation across k [7] and all the dimensions for ioric E [d]. and then using Lemma 9, we have the following inequality:
((())
+ 62Φ (100%)))
T
Go
O
(01,, 00,)
21
(1
1=1
(Vii. + t1) (|1=/") (
+ E2 (100%1))
( UK- + t1) (11-2-1)
(10 ())
T
0k.i
07)²
E2Φ
k=
2
na
(1063))
nk 1h-1
k-
20 VhdD
A 00' O2, On (vva + +
2
2
Using telescoping sum, Cauchy-Schwer inequality, with some negative end terms omitted, consider- ing Lemma 10, we have:
T
D2co
dBg
(1- (1-32) dD2 00
VTG
d33
0 Or. m&)
7E3
(1- Bz) (Bg 32)
TE
- Bz) (3g 33)
+ XVTDD + nTT WnDD + Xy (1 + log DV DVh.D
(1 dD2 VT
d8g
V (1- Bz) (3g 522
ne3
>VTD'GG + TVThdD An (1 log T)) DVhD
Lemma 12 (Bound for Σ (0k-1 0g, (m.m)-)).
(0 0g, 1) S ln T.Th
34

---

Proof. Using Lemma 9, we have
10 k-1
T
T rk 11
0
OR. m
mk-
S
mm6-1
(-)
[
k=
k=1
22
0
T
rk - 10
||m=-10
(1022)
1=1
Fk-1
+ E20
S TVThDD
Theorem 2. Suppose Assumptions 3.3 3.6 hold. If
Ba
1, The
Ve
as
v/
A
vk'
2
Algorithm 7 achieves the following guarantee. for all 2 1.
R(T) dD3 VTG
dβg
av TVTTD vd+ AVTD'G
ne3
V (1 B3) (3g (32)
(1 (1-13) -2)
nV To Doo
An (1 log T) DVDDDDD. Do
+
(1 32)
(1 - B2)
Proof. Using Assumption 3.6, have,
lx (0x) - 4x (0*) g. (O2 0²) Σ ga.i (On.i 0%))
We focus the th dimension of the parameter vector 02 E Rdd. Based on the update rules in Algorithm/ 7, we have
(1) ()
0,+1. Ou,i M,ER
(1) rr
+ E2Φ (10%)
(0) k
B2
(1 B2)
Φ
Ok.i
81
m
VRR + E1
(gr,s + agar,) + Xe
r
(4)
+ E2Φ
(1080)
[1(0)
Subtract the scalar 0% and square both sides of the above update rule, we have,
[
(0++1, -0,)² =(0,. -0;)² + +m
rr
(2)
+ 82Φ
(110%²
(9)
- -2 -2(0ks 0%) m1
32
(1 32)
(gk. + a4aa))) + Ax
k
VVk,
+&1
1,5
/Vki + 81
r
+
E2Φ
[0(00
Rearrange the above equation and use Young's inequality, ab - a²//2+ 6² /2. Then
35

---

(10%)) ((0k.: 0²r²² - (0k+1,5 - 07
VKki + E1
F.k
+ E2Φ
gk.i (Ok. -0+)
2T/ (1 52)
0(1%%) (
$
82
m + (0) - 0k.)) auaRR + A VVk, (0" 0k.,) Ou.
(0) - 0k,))
(1 82)
(1 - B2)
(106(2
Tk
Va.i +-
2 2 82)
Fh.
(
(10(
rk
(1)
(0(00) 1)
VR, E1
rk
+ 52Φ
((0 07)) -(0x+1,5 -)
21T (1 - 82)
(8)
82
(0% Ok.)) 11- + (0) - OR.i) Ou.RRR + d VVx.i + E1 (0), Os.s) O
(1 82)
B2)
rk (10%
ma,s + AxBr.s)
(1 - B2)
( rk
+ e2Φ (10()) l)
Obtaining the regret upper bound via summing the above equation across A E 1... T and all the dimensions for i 1,..., d and considering Lemma 9, we have the following inequality:
(1)
+ 5 ((00%))
Go
ri
R(T)= < Σ
(01,: 07)²
2m (1 32)
(1019)
1=1
d
T
1
(0., - -0,)²
211 (3]
1=1 2=
Vv. + +1)
+ E20 (10%)
(vVk-In +ei)
(11222)
[2
+ t20
(102211) ())
n (10%)
(()))
7k 14-1
k-
T
82
(0"
sr, m )) + Or (0) 0x, ax) +
(0* Oz, O2 (VVx + E1))
(1 - 32)
(1 12)
k=1
7o vhdD
AaM DymhG Doc
2 2(1 32) 2 32)
Using telescoping sum, Cauchy-Schwanl inequality, with some negative end terms omitted and considering Lemmas 10 to 12 we have:
36

---

R(T)= - 00
dD vTG
dB3
1E3
(1 - 83) (3g 32)
1e3
(1 B3) (33 - B3)
B2
(0k-1- 6k. mk-1) + +OVTDG XVTD (1 - B2)
00 1 m 1)
(1 32)
nV TVThhooo Doc
An (1 + log T) DV
(1-32)
(1-B)
dD² vTG
d3g
+aVTDG Vd+ AVTD&G>
1JE3
(1- (1- (3g 83)
(1-
7 yVTdlG Dox
An + log T) Do
(1-32)
B2)
37

---

D Experimental Details
D.1
Supplementary Experiments
Table 8: Test top-1 Acc. (%) on ResNet20 with CIFARIO and CIFAR100 and on ResNet34 with
ImageNet (for hyperparameters and architecture details, see Section D.2)
Dataset
CIFAR10
CIFARI00
ImageNet
batch size
128
128
256
SGD
91.85
64.93
70.64
ESGD
91.83
65.01
70.59
Adam
89.88
64.35
65.06
EAdam
90.12
65.24
65.31
Lamb
90.89
61.29
69.17
ALTO
91.24
65.74
69.95
Different Optimization Test Functions
Adam
ALTO
Function Visualization
Optimization Path
Final Solution
Initial Point
6.05
0.3
35
1.0
0.6
0.4
0.0
0.0
-3.00
-0.55
50
-0.25
3.50
0.25
0.50
0.75
1.00
-0.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.25
3.00
Figure 8: Optimization process of Adam (31 = 0.95) and ALTO (8, = 0.99, = 0.95, a 80)
on functions fi (x) (0.05ziii (2x) + bn 1)), 2.3 and Adam (31 0.9999) and ALTO
(81 0.99, B2 0.9, a 900) on f2(2) xx) -00 -00(2+0.6)
25x
Fo = 0.95.
38

---

Performances of Different Optimizers on BERT
base
Because most of the parameters of the model have been pre-trained to relative accurate extent, the advantage of ALTO would limited but still stable. Even so, still demonstrates broad adaptability, at least on the datasets CONLL 2003, IMDB, and MRPC The smaller the dataset is (IMDB (25K), CoNLL2003 (I4K), and MRPC (3668)) or the larger the batch size (32, 1024, 4096), the bigger advantage of ALTO over other optimizers is. This is maily because ALTO is designed for large-batch training.
Table 9: Test result (%) of fine-tuning BERT hase model the CONLL2003, IMDB and MRPC tasks.
BiSz
Optimizer
2003 (<<44K)
IMDB (25K)
MRPC (3668)
Avg
F1
Prec.
Recall
Acc.
FT
Free.
Recall
Acc.
Prec.
Recall
FI and Ace
Adam
86.97
85.70
87.15
91.77
91.52
94.43
88,78
84.46
88.68
85.99
91.34
88.68
AdamW
90.25
92.42
92.43
92.24
92.62
86.75
89,50
AdaBelief
89.98
89.01
90.98
92.93
92.85
93.01
91.86
83.36
87.87
85.24
90.67
89.39
32
Lamb
89.86
88.82
90.93
93.13
93.09
93.56
92.63
84.40
88.59
86.22
91.10
89.81
ALTO
90.29
89.53
91.06
93.24
93.19
93.82
92.57
84,86
89.01
86.07
92.15
90.11
Adam
89.50
88,09
90.96
92.97
92.91
93.79
92.04
79.76
85.05
83.58
86.37
88.03
Adamt
89.46
88.17
90.78
92.98
92.91
93.92
91.92
80.86
86.32
82.24
90.84
88,50
1024
AdaBelief
90.38
89.66
91.12
92.82
92.69
94.46
90.98
79.53
84.61
84.58
84.65
88.00
Lamb
90.05
89.37
90.75
92.07
92.30
89.74
95.01
80.98
86.04
84.03
88.14
88,28
ALTO
89.95
91.24
93.10
93.17
92.19
86.77
82.26
91.80
89.00
Performance Comparison with Related Works.
Table 10: Test top-1 Acc. (%) on ResNe120 with CIFARIO and CIFARI00 and on ResNet34 with ImageNet (for hyperparameters and architecture details, see Section D.2)
Dataset
CIFAR10
CIFAR 100
ImageNet
batch size 128
16384
128
16384
256
4086
Adan 90.62
76.42
62.27
47.96
69.36
67.66
ALTO
91.24
88.83
65.74
57.78
69.95
70.83
Table 11: Comparative performance of LION and ALTO in training the GPT-2 (345M parameters) Model with 4096 batch size on Megatron-LM Framework [40] by the OpenAl's open-sourced GPT-2
Output Dataset (1M).
Optimizer
Train Loss
Test PPL
LION
4.66
110.54
ALTO
4.33
78.37
Table 12: Comparison of average elapsed time per iteration for ALTO and LION. LION computes 31.2% faster than ALTO at each iteration. When the global batch size divided by the micro batch size cquals data parallelism, it indicates that each GPU performs only one forward pass and one backward pass per iteration.
Optimizer LION
Time (ms)
253
ALTO
332
VGG and DenseNet for CIFARs
In addition to ResNet, we expanded our experiments include other network architectures such as VGG-16 16 and DenseNet, particularly focusing on training with large batch sizes CIFAR10 10 and CIFAR 100 datasets. The network structures for VGG-16 and DenseNet are outlined Tables 13 and 14:
39

---

ALTO LION
3000
2500
2000
1500
500
250
500
750
1000
1250
1500
1750
2000
Iteration
Figure 9: PPL Variation over First 2000 Iterations 345M GPT with Det-t--Outut-Dattast. Dataset. In this experiment, the number iterations required for train _PL to converge to 200 was 66% fewer for ALTO compared to LION.
VGG-16 comprises approximately 33.73 million parameters
DenseNet-166 comprises approximately 7.27 million parameters.
Table 13: Architecture of VGG-16
Layer Type
Configuration
Convolutional
2x[96], 2x[128], 3x[256), 3x[512], 3x[512] MaxPool (2x2, stride 2)
Pooling
Fully Connected
3 FC (4096, 4096, num classes)
Batch Normalization
After each Convolution
Activation
ReLU
Dropout
0.3 after first two FC layers
Table 14: Architecture of 1enseNet-169
Layer Type
Configuration
Basic Convolution
Initial Conv Layer
Dense Blocks
6. 12, 32, 32 Bottlenecks per block
Transition Layers
Between Dense Blocks 256, 128, num cam_cllsses)
Fully Connected
FC
Growth Rate
32
Reduction
0.5 after each Dense Block
Batch Normalization
After each Convolution in Bottleneck
Activation
ReLU
Dropout
0.4 after FC layers
In our experiments, we employed various neural network architectures distinct from those described the main text, utilizing prominent optimizers such as SGD, Adam, AdamW, AdaBelief, and Lamb for comparison against our optimization algorithm, ALTO2. These experiments were conducted
40

---

on CIFARIO and CIFARI00 datasets. The results obtained using VGG-16 and DenseNet- 169 are presented in Table 15.
Table 15: Ace. (%) of VGG and NenseNet CIFARIO and CIFARI00 for batch size 16384
Dataset
CIFARIO
CIFAR100
Architecture
VGG-16
DenseNet-166 169
VGG-16
DenseNet- 169
SGD
86.56
65,44
55.46
41.75
Adam
85.01
81.89
40.44
48.41
AdamW
83.33
81.55
37.62
48.58
Lamb
90.63
85.77
60.44
53.75
AdaBelicf
87.87
76.88
59.44
43.41
ALTO
90.92
86.19
63.15
54.24
During the training process, the acc-epoch and loss-acc relationships depicted in Figures 10 and 11, respectively. Our optimization algorithm, in these extended experiments, was employed with training parameter of batch size 16,384. This approach was taken to explore its optimization effectiveness across various network architectures and to ensure that our optimization algorithm possesses robust model generalizability.
VGG
DenseNet
A
B
80
CIFAR-10 val-ace
70
60
SGD
SGD
50
Adam
Adam
40
AdamW
40
AdamW
30
AdaBelief
30
AdaBelief
20
LAMB
20.
LAMB
ALTO
ALTO
10
10
100
200
300
400
100
200
300
400
Epoch
Epoch
VGG
DenseNet
60
C
D
50
CIFAR-100 val-acc
40
SGD
30
SGD
30
Adam
Adam
AdamW
20
AdamW
AdaBelief
AdaBelief
10
LAMB
10
LAMB
ALTO
ALTO
0
100
200
300
400
0
100
200
300
400
Epoch
Epoch
Figure 10: test accuracy with batch size 16384
Comparative Illustration of Training with Batches from Small Large
conducted a comparison of training the VGG-16 16 convolutional neural network using the ALTO and Lamb optimizers on the CIFAR-100 dataset. Our comparison involved batch sizes ranging from
41

---

VGG
DenseNet
A
SGD
B
SGD
5
Adam
2.0
Adam
CIFAR-10 train-loss
AdamW
AdamW
AdaBelief
AdaBelief
1.5
LAMB
LAMB
3
ALTO
ALTO
1.0
2
1
0.5
100
300
400
100
300
400
Epoch
Epoch
VGG
DenseNet
5
SGD
SGD
Adam
4
Adam
CIFAR-100 train-loss
4
AdamW
AdamW
AdaBelief
AdaBelief
3
3
LAMB
LAMB
ALTO
ALTO
2
2
1
0
100
300
400
100
300
400
Epoch
Epoch
Figure 11: training loss with batch size 16384
200 to 50,000 (the total number of training samples in the CIFAR-100 dataset). The results of the experiment are presented in the following Table and Figures 12 and 13.
In this experiment, to more objectively demonstrate the effectiveness of the ALTO optimizer, both and ( were held constant across all batch sizes during training, and all other hyperparameters were set to their default values. This experiment provides a clear illustration of the performance of the ALTO optimizer.
The Relationship between Epoch and Training Speed, and Comparative Experiments
To explore the relationship between batch size and training speed, we conducted experiments with the ALTO training setup for VGG-16 16 on CTFAR-100. The batch size ranged from 200 to the total number of training samples (50K). We calculated the time required for each epoch during stable training. For details, see Table 17.
"Computation Time' represents the total time spent on computational tasks during the training of a sin- gle epoch. It encompasses the duration of all the calculations performed by the algorithm. In contrast, 'Total Time' not only accounts for the CComputation Time' but also includes the time overhead caused by the data transfer between the CPU and GPU, well as the time required for memory allocation (malloc) operations. The latter is subject to the influence of the machine's s capabilities and can vary significantly with different hardware configurations. With ongoing advancements in the field of high-performance computing, the time costs associated with these hardware- aedeeeee-dependent processes are expected to diminish. Therefore, the CComputation Time' is of greater interest from an algorithmic
42

---

Table 16: Comparison of ALTO and Lamb Optimizers Across Different Batches with Corresponding Learning Rates, Beta Values, Accuracy, and Loss
Batch Size
Learning Rate
P1
ALTO
Lamb
Accuracy
Loss
Accuracy
Loss
200
0.001
0.999
66.9
0.0034
67.3
0.0025
500
0.001
0.999
63.35
0.0029
63.3
0.0034
1K
0.001
0.999
59.79
0,0046
59.82
0.0057
2K
0.01
0.999
70.43
0.0021
70.24
0.002
5K
0.01
0.999
67.55
0,0008
67.05
0.0009
10K
0.01
0.999
64.51
0.0023
62.92
0.0046
15K
0.01
0.999
64.38
0.0065
62.64
0.0132
25K
0.01
0.999
59.19
0.2003
57.79
0.4173
30K
0.01
0.999
58.69
0.2859
58.16
0.3718
40K
0.01
0.999
58.72
0.3535
57.11
0.5281
50K
0.01
0.999
49.69
1.4763
39.91
2.0295
train_loss
2.5
2
1.5
0.5
200
500
1000
2000
5000
10000
15000
25000
30000
40000
50000
ALTO Lamb
Figure 12: loss of training with batch size 200 to 50K of ALTO and Lamb
Table 17: The relationship between batch size and training speed, showing total time(s) and pure computation time for epoch.
Batch Size
Total Time
Computation Time
200
14.264
12.327
500
6.533
5.330
1000
4.147
3.034
2000
3.290
1.917
5000
3.488
1.233
10000
4.563
1.026
25000
7.542
0.891
50000
13.100
0.847
perspective, as more accurately reflects the efficiency of the algorithm itself, independent of the underlying hardware performance
43

---

val acc
80
50
20
10
ALTO Lamb
Figure 13: Acc(%) of training with batch size 200 50K of ALTO and Lamb
We conducted comparative experiments on VGG-16 16 using the CIFAR-100 dataset to compare the training time required for ALTO and Lamb reach the same level of accuracy. In these experiments of batch size 16384, both algorithms maintained a consistent learning rate throughout the training process. For ALTO, the hyperparameters were with fixed 31 of 0.999 and of -5. For details, see Table 18.
Table 18: Comparison of the time it takes for ALTO and Lamb to achieve the same accuracy
ACC(%)
ALTO Total Time (s) ALTO Computation Time (s)
Lamb Total Time (s)
Lamb Computation Time (s)
20
137.103
5.348
195.992
7.34
30
202.674
7.905
276.694
10.364
40
333.817
13.020
409.277
15,330
50
482,842
18.833
582.210
21.807
60
608.023
23.715
864.669
32.387
70
Reinforcement Learning
, - of
Figure 14: Swimmer
Figure 15: Ant
Figure 16: Humanoid
Specifically, we tested three sets of reinforcement learning games in three environments simulated by the MuloCo engine: Swimmer-v3, Ant-v3, and Humanoid-v3. We conducted our experiments using the Proximal Policy Optimization (PPO) algorithm [39], which is widely used in the field of
44

---

reinforcement learning. The experiments were based on the widely-used open-souree codebase Tian- shou [47]. In our comparative experiments, we replaced the default optimizer of the framework with the ALTO optimizer, while keeping other hyperparameters as the default settings of the framework, such as 1r=3e-4, epoch=200), and each set of experiments was conducted using ten random seeds.
In the Swimmer-v3 environment, there is a simple organism with two to three segments, moving in a twoddimensional space. Here, the challenge to learn coordinate its body segments to propel forward in water. In the Ant-v3 environment, the agent is four-legged creature similar to an ant. The objective to control this ant to move quickly as possible in a three- piree-immensionl space. The Humanoid-y v3 environment is one of the most complex settings within the MuJoCo suite. It involves bipedal humanoid robot that needs to learn walking, running, and maintaining balance. The challenge here lies in the high dimensionality of the action space and the requirement for intricate balance and coordination.
The training tasks in our three reinforcement learning experiments vary in complexity. The results show that our optimization algorithm is highly versatile. The results of three comparative experiments are shown in Figures 14 to 16.
Full-Batch Training for LSTM
We also investigated the impact of large-scale training of ALTO traditional network architectures. Specifically, for the Named Entity Recognition (NER) task on the CONLL2003 dataset, we conducted full batch training on two-layer BiLSTM network (with batch size equal to the size of the training dataset, which is 14,987 samples). A BiLSTM network is an extension of the traditional LSTM model, which processes data in both forward and backward directions, providing a richer context for each sequence element. This bi-directional approach allows the model to capture dependencies from both past and future states, enhancing In this task, we continued to identify the optimal learning rate setting for each optimization algorithm. Additionally, uniform learning rate adjustment strategy was employed, where the learning rate was reduced to one-tenth of its original value after every 100 training epochs, with total of 400 training epochs. Our BiLSTM network structure is shown in Table 19.
ability to recognize and classify entities text sequences.
Table 19: Architecture of BiLSTM Network for NER
Layer Type
Configuration
Embedding Layer
Dimension: 100
BiLSTM Layer
Hidden Dimension: 128.
Num Layers: 2,
Bidirectional: True,
Dropout: 0.5
Dropout Fully Connected Layer
Dropout: 0.5
Output Dimension: 9
Table 20: Comparison of Different Optimizers for Full Batch Training on BiLSTM
Optimizer
Train Loss
Test Acc
Test F1
Adam
0.0663
15.00
9.66
Adam
0.0374
27.39
15.89
AdaBelief
0.0476
25.68
13.98
Lamb
0.0164
61.19
52.20
ALTO
0.0139
62.36
57.27
The performance of all optimizers this task is shown in Table 20. All optimizers were tested across learning rates [0.005, 0.01, 0.05], with each optimization algorithm selecting the best performing learning rate. The F1 scores corresponding to each learning rate for all optimizers are shown in Table 21.
In this full-batch experiment targeting the CONI.L2003 dataset, the hyperparameter 31 for ALTO set to 0.7, with all other parameters remaining at their default values.
45

---

Table 21: F1 Scores for Different Optimizers Various Learning Rates
Optimizer LR 0.005
LR 0.01
LR 0.05
Adam Adam W 8.24
9.66
0.12
15,89
4.69
AdaBelief
0.24
13.98
Lamb
10.35
20.27
52.20
ALTO
24.43
57.27
D.2 Achitectures and Hyperparameters
CV Projects
In our CIFAR training tasks, we employed the ResNer20 architecture as part of our experimental setup. This section provides detailed overview of the network structure and the number of parameters involved the model.
The ResNet20 model used in our experiments is variant of the ResNet architecture, specifically designed for the CIFARIO and CIFAR 100 dataset. The model structure is as follows:
Table 22: Architecture of ResNet20
Layer Type
Output Size
Details
Convolution + BN
32x32
3x3 conv, 16, stride 1, padding 1
BasicBlock x3
32x32
[3x3 conv, 16] x 2 each
BasicBlock x3
16x16
[3x3 conv, 32, stride 2] + [3x3 conv, 32] x 2 each
BasicBlock x3
8x8
[3x3 conv, 64, stride 2] + [3x3 conv, 64] x 2 each
Global Avg Pooling
1x1
Avg pool
Fully Connected
number of classes
number of classes-way softmax
The ResNet20 model for CIFARIO and CIFAR100, as implemented our experiments, comprises approximately 0.27 million parameters. This count includes parameters from all convolutional layers, batch normalization layers, and the final fully connected layer, which shown
Table 22.
In the experiments conducted on the CIFAR1O and CIFARION 100 datasets, the default hyperparameters of the optimizer were used. When the batch size was to 16.384, B1 was fixed 0.99, whereas for a batch size of 256, 81 was set to 0.1. For all compared optimizers at a batch size of 256, the learning rate adjusted within the [0.01, 0.05, 0.1) } select the best model, with total of 200 training epochs. Conversely. at a batch size of 16,384, the learning rate was fine-tuned between 0.01 and 0. determine the optimal model, over a course 400 training epochs. The optimal learning rates for each optimizer on CIFAR- 10 and CIFAR- are shown in Tables 23 and 24, respectively.In all experiments of CIFAR10 10 and CIFAR 100, the learning rate was reduced by a factor of ten after every quarter epoch of training: n 7) 0.1 when T%( epoch 0.
Table 23:
Learning Rates for CIFAR10 training
Batch Size
128
16384
SGD
0.05
0.1
others
0.01
0.05
In the ablation study section, which focuses on the CIFAR100 dataset, experimented with adjusting optimizer parameters for different batch sizes using the ResNet18 network architecture. The details of this network architecture presented in Table 25.
In our experiments targeting ImageNet, we employed the ResNet34 as the fitting model. The specific network structure is detailed in Tables 26 and 27.
46

---

Table 24: Learning Rates for CIFARI00 training
Batch Size
128
16384
SGD
0,05
0.1
others
0.01
0.01
Table 23:
Architecture of ResNet18 for CIFARIO0 in Ablation Study
Layer Type
Output Size
Details
Convolution + BN
32x32
3x3 conv, 64, stride 1, padding
BasicBlock x2
32x32
[3x3 cony, 64] 2 each
BasicBlock x2
16x16
[3x3 conv, 128, stride 2] + [3x3 conv, 128] x 2 each
BasicBlock x2
8x8
[3x3 cony, 256, stride 2] + [3x3 conv, 256] 2 each
BasicBlock x2
4x4
[3x3 conv, 512, stride 2] + [3x3 conv, 512] x 2 each
Global Avg Pooling
1x1
Avg pool
Fully Connected
100
100-way softmax
Table 26; Architecture ResNet34 for ImageNet
Layer Type
Output Size
Details
Convolution + BN
112x112
7x7 conv, 64, stride 2, padding 3
Max Pooling
56x56
3x3 max pool, stride padding
BasicBlock x3
56x56
[3x3 conv, 64] x each
BasicBlock x4
28x28
[3x3 conv, 128, stride 2] + [3x3 conv, 128] 2 each
BasicBlock x6
14x14
[3x3 conv, 256, stride 2] + [3x3 conv, 256) 2 each
BasicBlock x3
7x7
[3x3 conv, 512, stride 2] + [3x3 conv, 512) 2 each
Global Avg Pooling
1xl
Avg pool
Fully Connected
number of classes
Linear layer with 1000 outputs (for ImageNet)
Table 27: Architecture of ResNet-50 for ImageNet
Layer Type
Output Size
Details
Convolution + BN
112x 12
7x7 conv, 64, stride 2, padding 3
Max Pooling
56x56
3x3 max pool, stride 2, padding
Bottleneck x3
56x56
[1x1 conv, 64] + [3x3 conv, 64] + [1xl conv, 256] 3
Bottleneck x4
28x28
[1x1 conv, 128] + [3x3 conv, 128] [Ix] conv, 512] x 4
Bottleneck x6
14x14
[1x1 conv, 256] + [3x3 conv, 256] [1x1 conv, 1024] 6
Bottleneck x3
7x7
[1xl conv, 512] + [3x3 conv, 512] + [Ix1 conv, 2048) x
Global Avg Pooling
1x1
Avg pool
Fully Connected
number of classes
Linear layer with 1000 outputs (for ImageNet)
In these experiments, we trained models using two different batch sizes, with corresponding adjust- ments in learning rate and optimizer settings. In all our training sessions for ImageNet, the range of learning rates was set to [1e-3, 5e-3, le-2, le-1) ).
For batch size of 256, selected the best model based performance over different learning rates. The learning rate scheduler is T=90 and 7) * 0.1 when epoch% For the ALTO optimizer specifically, we set 31 to 0.01, keeping other hyperparameters at their default values.
For larger batch size of 4096, we also selected the optimal model over different learning rates. The number of training epochs and the learning rate reduction schedule remained the same as for the batch size 256 training works. In the case of the ALTO optimizer for this larger batch size, 31 was set to 0.99, while other parameters were maintained at their default settings. For each optimizer, the learning rate and accuracy corresponding to the best-performing model we ultimately selected are shown in Table 28. This approach ensured consistency in training duration and adjustment of the
47

---

learning rate across different scales of batch sizes while tailoring the optimizer settings to specific batch size requirements
Table 28: Learning Rates (LR) and Top-1 and Top-5 Accuracy (%) of ResNet34 on ImageNet- 1k
Batch Size
256
4096
LR (256)
LR (4096)
Top-1 Acc.
Top-5 Acc.
Top-1 I Acc.
Top-5 Acc.
SGD
70.64
89.68
49.35
74.41
le-2
le-2
Adam
65.06
86.47
54.96
79.07
le-3
le-2
Adam W
69.64
88.90
68.40
88.07
le-3
le-2
AdaBelief
70.12
89.24
70.18
89.26
le-3
le-2
Lamb
69.17
88.81
70.34
89.55
5e-3
le-2
ALTO
69.95
88.94
70.83
89.64
1e-3
le-2
In the experiment with ResNet-50 on ImageNet- 100, where batch sizes ranged from 1K to 32K, the learning rate scheduling method incorporated a combination of warmup and polynomial decay to optimize the training process. The random seed was set to 42, ensuring reproducibility across different runs. The specific learning rates and warmup epochs for each batch size are detailed in Table 29. This approach aids in stabilizing the training in its early phases by gradually increasing the learning rate from lower initial value during the warmup period, followed by a polynomial decay to finely tune the model as it converges to optimal solutions.
Table 29: Learning Rate and Warmup Epochs for Different Batch Sizes
Batch Size
IK
2K
4K
8K
16K
32K
Learning Rate
22.x
22 100
21. x
100
21. 100
20.0x
20.0 x 100
Warmup Epochs
0.625
1.25
2.5
10
20
NLP Projects
In the experiments focusing on natural language processing, we utilized the BERT-bas model for fine-tuning on downstream tasks. This approach allowed to evaluate the performance of various optimizers under different batch size settings. The BERT-base model, known for its efficacy in a range of NI.P tasks, comprises specific network architecture. The detailed structure of BERT-base, including its layers and configurations, presented Table 30. At the same time, the network structure for the pre-training of the GPT architecture is shown in Table 31.
Table 30: BERT-base Network Structure
Layer Type
Deseription
Token, Segment, Positional Embeddings Size: 768
Embedding Layers
12 Layers
Hidden size: 768
Transformer Blocks
Feed- feeel-fowwardfflterr size: 3072
Attention heads: 12
Output Layer
Linear Layer with Softmax Function
In our experiments on the CONLL2003 Named Entity Recognition (NER) task, each of exper- iments was fine-tuned over epochs. For training with batch size of 1024, the learning rate of AdaBelief, ALTO and Lamb were set 1e-3; for Adam and AdamW, the learning rate was set to 5e-5 because for Adam, Adam W, and AdaBelief, increasing the learning rate similarly results in poorer convergence performance in this task. We meticulously recorded the FI-score on the validation set after each epoch. The model yielding the highest F1-score on the validation set was subsequently used for calculating metrics on the test set. Table 32 shows the performance of each optimizer with different learning rate in experiments with batch size of 1024.
48

---

Table 31: GPT Model Architecture
Component
Specification
Layers
24 Transformer blocks
Hidden Size
1024
Attention Heads
16
Sequence Length
1024
Max Position Embeddings
1024
Table 32: Fl-score of Experiments Using Various Optimizers with different learning rate for CONLL2003
Optimizer
le-3
5e-4
5e-5
Adam
86.62
89.45
89.50
Adam W
83.51
89.29
89.46
AdaBelief
90.38
90.14
59.20
Lamb
90.05
88.74
ALTO
90.59
88.77
Accuracy was not utilized as a metric this task due to its limited effectiveness in scenarios with imbalanced data. Accuracy may yield misleading results by overemphasizing the majority class while neglecting the model's performance on the minority class. Hence, more informative metrics like Fl-score, precision, and recall were employed to provide balanced evaluation of the model's performance across different classes.
For the IMDB task with batch size of 1024, the learning rate was selected in (5e-4, 5e-5, le-5), Each set of experiments was fine-tuned over epochs. The accuracy with different learning rate was shown in Table 33. In these experiments, also shown the training loss and training accuracy at the end of each epoch are presented in the table below.
Table 33: Accuracy of Experiments Using Various Optimizers with different learning rate for IMDB
Optimizer 5e-4
5e-5
le-5
Adam
50.00
92.97
92.43
AdamW
50.00
92.98
91.79
AdaBelief
91.75
92.82
91.75
Lamb
92.19
92.07
ALTO
93.10
92.19
In the natural language processing experiments, the batch size had the most significant impact on the MRPC task, primarily because of its small training dataset size, consisting of only 3,668 samples. For this task, the hyperparameters were set follows: a batch size of 1024 and a 81 value of 0.999. The learning rate was selected in (5e-4, 5e-5, le-5). The performance effects of different learning rates corresponding various optimizers are presented in Table 34, Each experiment was fine-tuned over the course of 10 epochs.
Table 34: Accuracy Experiments Using Various Optimizers with different learning rate for MRPC
Optimizer 5e-4
5e-5
lc-5
Adam
68.75
79.76
72.69
AdamW
66.49
80.86
66.49
AdaBelief
79.53
70.08
66.49
Lamb
80.98
70.60
ALTO 81.39
69.56
In our pre-training setup for the GPT model, we employed a deep network architecture consisting
of 24 Transformer layers, with each layer configured to have a hidden size of 1024 and 16 attention
49

---

heads. This configuration supports the processing of sequences up to 1024 tokens in length, allowing the model capture long-range dependencies within the data. The training utilized a micro-batch size of 4, with an effective global batch size of 4096. This large-scale training was facilitated by MPI and NCCL to optimize multi-GPU communication. Furthermore, gradient clipping was applied at threshold of 1.0 to prevent exploding gradients, common issue in training such deep networks. Additionally, mixed precision training was leveraged to enhance training speed and reduce memory consumption without compromising model accuracy. Our pre-training experiments utilized the dataset from OpenAl's open-source dataset: the spt-2-outtut-dataset, which includes total of million data entries.
All the optimizers compared in our study were experimented with different learning rates to determine the optimal results. The training used a random seed of 1234. The best learning rates for each optimization algorithm are shown in Table 35. See Figure for the PPL--iter graph of the first 1.5k iterations in our 5k iterations of training.
Adam
2500
AdamW
Adabelief
Lamb
2000
1500
PRL
1000
01
200
400
600
800
1000
1200
1400
Iteration
Figure 17: PPL Variation over First 1500 Iterations on 345M GPT with GPT:2-Output-Dataee
Table 35: Best Learning Rates for Various Optimizers on Best Learning Rate
Optimizer
Adam
6c
AdamW
Se
AdaBelief
5e
LAMB
le
ALTO
le -
D.3 Hyperparameter Tuning
In our extended experiments, we present box plots of the training results for ResNet-18 on CIFAR- 100 under various hyperparameters, with the training graphs displayed here. In experiments focusing on Bi, we observe that the value of B1 is generally positively correlated with the size of the batch size. the comparative experiments for different a, due the closeness of the curves, we extract the graphs of the last 20 epochs. The specific display diagrams are as shown in Figures 18 to 20.
50

---

BATCHSIZE 16384
BATCHSIZE 128
topl accuracy
5.4
B1 = 0.01
B1 0.1
B1 0.5
B1 0.9
B1 0.99
Epoch
Epoch
Figure 18: acc-epoch for different on batch size 16384 and 128.
Batch Size 16384
Batch Size 128
03375
63952
topl accuracy
10.0
03325
50.0
a 5.0
40.0
a= 30.0
== 0.1
20.0
0.2279
== 10.0
0.500
a= == 0.9
5,0
5.0
== 0.5
0.5
0= 10.0
20.0
3825
105.0
387.5
230.0
1325
125.0
337.3
200
1825
1853
1873
330.
1925
355.7
3375
2530
Epoch
Epoch
Figure 19: acc-epoch for different on batch size 16384 and 128.
51

---

Batch Size 16384
Batch Size 128
50.0
10.00
1,005
a
40.0
30.0
0.5
20.0
00034
0.1
10.0
a= 0.5
train train_loss loss
5.0
a=0.9
0.5
5.0
003335
0.5
10.0
0,0010
20.0
0.334
0,033
Epoch
Epoch
Figure 20: training loss-epoch for different a on batch size 16384 and 128.
52


Reviews:
[{"review_id": "0myp9aJy0s", "reviewer": "Reviewer_9xUK", "summary": "The paper proposes an adaptor to optimization algorithms by introducing a term that minimizes the norm of the gradient, thereby, leading the parameters to a flatter minima. The authors show across multiple experiments that their method improves performance, especially for large mini-batch training.", "questions": "Please see Weaknesses for actionable items that will improve the paper. I am willing to update the score based on authors addressing some or all of these.", "limitations": "There is no limitations section and I highly suggest one especially contrasting large vs small batch training & failure modes where alpha is highly sensitive (e.g. cases where every minima is quite sharp etc).", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) The paper is well written and mathematically rigourous \n2) The authors motivate the intuition well and describe the design choices systematically, to motivate their contribution\n3) The results are very convincing for large batch training and show good improvement\n\nWeaknesses/Questions:\n1) Negative Examples: Can the authors find any negative examples/tasks for training with their new optimizer? For instance, take the case of a novel task where inherently the loss function turns out to have minimas that are only sharp and not much of flatter minimas. At the very least, can a toy loss landscape be designed that specifically has only sharp minimas with a large value of Hessian at the minimas? How does the norm of Hessian correlate with the optimal alpha parameter in such cases for convergence?\n2) The method introduces a new alpha parameter that is akin to a learn rate but for the grad(||grad f||^2) term. How hard or easy is this to tune? How does this correlate with the learning rate for the term grad(f)? It is not clear if the new parameter is quite sensitive from Figure 4 where, especially for small-batch training, the top 1 accuracy seems to be varying significantly with the alpha parameter value. \n3) The results are very convincing for large batch training but not so significant for small-batch training. Do the authors suggest users resorting to their method only for large batch training?\n4) An arxiv preprint (https://arxiv.org/abs/1905.13200) looks into similar issues and uses a biased estimator to reduce the variance in the loss landscape, which effectively leads it to a flatter minima. (See AdamUCB). What is the relationship of such methods where the variance of the loss landscape is reduced via a separate loss term Versus reducing the norm of the gradient with a Hessian term as in the current paper?", "was_revised": true, "final_justification": "The authors have addressed most of my concerns relating to the new hyperparameters introduced and the weak small-batch training results. In the light of the detailed rebuttal and suggestions to include a brief version of them in the final copy of the paper, I'm increasing my final rating to Accept.", "rebuttal": "Thanks for your carefully reading and expertise. I will summarize your questions and answer them one by one. \n\n**Q1: How does the norm of Hessian correlate with $\\alpha$?**\n\nAlthough I am not very sure about the target of the question, I try my best to give a tentative answer. If we want to find the correlation between $\\alpha$ and sharpness of a minimum, let us set the toy landscape $f(\\theta)=\\frac{\\sigma\\theta^2}{2}$. The sharper the landscape is, the greater the $\\sigma$ is. As we discussed in Remark 2.2, the convergence process is governed by the following equation:\n$$\n\\eta^2\\dddot{\\theta}+\\eta\\ddot{\\theta}(2-\\beta_1-\\beta_2)+(1-\\beta_1)(1-\\beta_2)\\dot{\\theta}+(1-\\beta_1)(1-\\beta_2) g+(1-\\beta_2)\\eta\\dot{g}(1+(1-\\beta_1)\\alpha)=0,\n$$\nwhere $g=\\nabla f$. If $f(\\theta)=\\frac{\\sigma\\theta^2}{2}$, we have\n$$\n\\eta^2\\dddot{\\theta}+\\eta(2-\\beta_1-\\beta_2)\\ddot{\\theta}+(1-\\beta_2)(1-\\beta_1+\\eta\\sigma(1+(1-\\beta_1)\\alpha))\\dot{\\theta}+(1-\\beta_1)(1-\\beta_2)\\sigma\\theta=0.\n$$\nFor fast convergence, large $\\sigma$ (stiffness) usually requires large $\\alpha$ (damping). If the damping is too large, the optimizer converges very slowly. If the damping is too small, the optimizer will oscillate. For the optimal $\\alpha$, we should use Routh-Hurwitz Criterion. The calculation is very long, we only give a detail idea here. Considering equation\n$$\nr(x)=\\eta^2x^3+\\eta(2-\\beta_1-\\beta_2)x^2+(1-\\beta_2)(1-\\beta_1+\\eta\\sigma(1+(1-\\beta_1)\\alpha))x+(1-\\beta_1)(1-\\beta_2)\\sigma=0,\n$$\nwe believe the optimal $\\alpha$ should be $$\\alpha^{\\star}=argmin_{\\alpha}(\\max\\\\{Re(x_1),Re(x_2),Re(x_3)\\\\}).$$ \nConsidering \n$$\ng(x)=r(x-h)=\\eta^2(x-h)^3+\\eta(2-\\beta_1-\\beta_2)(x-h)^2+(1-\\beta_2)(1-\\beta_1+\\eta\\sigma(1+(1-\\beta_1)\\alpha))(x-h)+(1-\\beta_1)(1-\\beta_2)\\sigma=a_3(h)x^3+a_2(h)x^2+a_1(h,\\alpha)x+a_0(h,\\alpha)=0,\n$$ where $h>0$, we let the real part of all roots of $g$ be negative. Using Routh-Hurwitz Criterion, problem convert to an optimization problem $$\\max_{\\alpha} h$$ with constraints $a_3(h)>0, a_2(h)>0, a_1(h,\\alpha)>0, a_0(h,\\alpha)>0, a_2(h)a_1(h,\\alpha)>a_3(h)a_0(h,\\alpha)$. Solving the optimization problem, we obtain the optimal $\\alpha$.\n\nAs we said in Remark 2.1, $\\alpha$ could be negative and positive. In detail, if we want the optimizer to explore more possible minima (to escape local minima), $\\alpha$ should be negative. The sharper the local minima are (greater the norm of Hessian), the smaller the $\\alpha$ should be (greater $|\\alpha|$). In contrast, if we do not want the optimizer to explore more possible minima but want it to exploit the possible global minimum found at first (to converge to the possible global minimum), $\\alpha$ should be positive. The faster the convergence we want, the greater the $\\alpha$ should be. Carefully, as we said in Remark 2.2, $|\\alpha|$ is bounded by $\\frac{1}{1-\\beta_1}$ for convergence stability. When $\\alpha$ is negetive and very smalll (near $-\\frac{1}{1-\\beta_1}$), the optimizer can escape nearly every minimum (no matter how sharp it is), but it need very long time to converge. \n\n**Q2: How to tune  $\\alpha$ and $\\beta_1$?** \n\nIn fact, it is easy to tune. Usually, the larger the batch size is , the larger the $-\\alpha$, $\\beta_1$, and learning rate $\\eta$ are. For large batch training, the landscape is almost unchanged from batch to batch, especially full-batch (totally unchanged). Meanwhile, there is a efficiency problem in large batch training. We hope the optimizer converges to the same accuracy level with the same the number of epochs as it does in small batch training. This means large batch training needs large learning rate $\\eta$ (for fewer iterations), but the learning rate is bounded by geometry of the landscape. However, the landscape of large batch is more stable, which is friendly to convergence. Therefore, we should let the optimizer to explore the landscape in large batch case (negative $\\alpha$) and to exploit the landscape in small batch case (positive $\\alpha$). As we said in Line 216, we default to $\\alpha=0.5, \\beta_1=0.01$ for small batch and $\\alpha=-5, \\beta_1=0.99$ for large batch in all our experiments (of course, it can be better).  As we show in Figure 4, there are more better results can be achieved with different $\\alpha$ and $\\beta_1$. We just want to show the method works. Only choosing the best result in experiment may be thought to be cherry-picky.\n\n**Q3: The optimizer is for only large batch training or both large and small batch training?**\n\nSince large batch training problem is very difficult in industry field, we just thoroughly focus on large batch training problem to show the ability of our optimizer for large batch training. Of course, the adaptor also theoretically works for small batch, since it degenerates to an optimizer itself when $\\alpha=0$. For small batch case, we should set $\\alpha$ positive and $\\beta_1$ small (defalut to $\\alpha=0.5$ and $\\beta_1=0.01$). The not so significant result for small batch case just be due to the unified default hyperparameter setting is not very suitable for certain small batch task. We do not spend much time on choose a suitable hyperparameter. Slightly tuning the $\\alpha$ and $\\beta_1$ will improve the result. We will systematically add more experiments of small batch training in our paper.\n\n**Q4: What is the relationship between reducing the variance of the loss landscape via a separate loss term and reducing the norm of the gradient with a Hessian term as in the current paper?**\n\nAdamUCB is an adapted Adam optimizer by adding $\\sigma_k:=\\sqrt{\\mathbb{E}\\left\\|\\ell_k(\\boldsymbol{\\theta}, \\zeta)-\\mathbb{E}\\ell(\\boldsymbol{\\theta},\\zeta)\\right\\|^2}$ to loss function $\\ell_k$ and tries to reduce both $\\sigma_k$ and $\\ell_k$. Our method adds $\\\\|\\nabla\\ell_k(\\boldsymbol{\\theta})\\\\|^2$ to loss function $\\ell_k$ and tries to reduce both $\\\\|\\nabla\\ell_k(\\boldsymbol{\\theta})\\\\|$ and $\\ell_k$. The relationship between $\\sigma_k(\\boldsymbol{\\theta})$ and $\\\\|\\nabla\\ell_k(\\boldsymbol{\\theta})\\\\|$ is \n$$\n\\\\|\\nabla\\ell_k(\\boldsymbol{\\theta})\\\\|\\sqrt{\\mathbb{E}\\delta_k^2(\\boldsymbol{\\theta})}=\\sigma_k(\\boldsymbol{\\theta}), \\tag{1} \n$$\nif we name $\\delta_k(\\boldsymbol{\\theta}):=\\frac{\\mathbb{E}\\ell(\\boldsymbol{\\theta})-\\ell_k(\\boldsymbol{\\theta})}{\\\\|\\nabla\\ell_k(\\boldsymbol{\\theta})\\\\|}$ horizontal amplitude at $\\boldsymbol{\\theta}$ and assume $\\\\|\\nabla\\ell_k(\\boldsymbol{\\theta})\\\\|\\neq 0$ at $\\boldsymbol{\\theta}$. Substituting the definition of $\\delta_k$ into the definition of $\\sigma_k$ yeild the equation \\eqref{eq:gauss}. We find the difference is $\\sqrt{\\mathbb{E}\\delta_k^2(\\boldsymbol{\\theta})}$. From the definition of the horizontal amplitude $\\delta_k$, we find $\\mathbb{E}\\ell(\\boldsymbol{\\theta})=\\ell_k(\\boldsymbol{\\theta})+\\delta_k(\\boldsymbol{\\theta})\\\\|\\nabla\\ell_k(\\boldsymbol{\\theta})\\\\|\\approx\\ell_k(\\boldsymbol{\\theta}+\\delta_k(\\boldsymbol{\\theta})\\frac{\\nabla\\ell_k(\\boldsymbol{\\theta})}{\\\\|\\nabla\\ell_k(\\boldsymbol{\\theta})\\\\|})$. This means if we regard the graph (landscape) of $\\mathbb{E}\\ell(\\boldsymbol{\\theta})$ as a translation of the graph of $\\ell_k(\\boldsymbol{\\theta})$, the smallest translation length should be $\\delta_k$. The $\\delta_k$ measures how much the the graph of $\\ell_k(\\boldsymbol{\\theta})$ shakes around the graph of  $\\mathbb{E}\\ell(\\boldsymbol{\\theta})$, and the difference between the two methods is the standard deviation of $\\delta_k$.\n\n**Thank you again for finding paper formatting problem, we will revise our paper soon.**"}, {"review_id": "Z2pkRxbBig", "reviewer": "Reviewer_vVfW", "summary": "This paper introduces a new optimizer method, ALTO which not only can have valley-following behavior, but also be able to escape from small-scale local minima. The author further provides the convergence analysis of algorithm and demonstrate its effectiveness across different experiments.", "questions": "Please check the weakness part. My main concern is about format and presentation issue, and limited experimental improvements. \nIf the author can: \n1. largely improve the format and presentation of paper; \n2. provide the standard deviation of the experimental results. \n3. explain clearly why ALTO mainly works in large batch size while the improvement is still limited. Meanwhile, give some practical suggestions when implementing ALTO.", "limitations": "Not applicable.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The intuition behind algorithm is well explained.\n2. The experimental settings are quite comprehensive.\n\nWeakness:\n1. The paper format is quite bad, especially mixing one-column and two-column formats. For example, line 76-88, line 121, line 150-151, etc. These are quite unfriendly for reader to read.\n2. For the experimental results. The performance improvement is limited, and only observed in large batch size. However, large batch size is not widely used in practice and from Figure 3, we can see that large batch size suffer worse performance.\n3. It will be helpful to report the standard deviation for the experimental results.", "was_revised": true, "final_justification": "The method is great and the explanations help to address my concerns.", "rebuttal": "Thanks for your comments and questions. We will improve the paper format and report the standard deviation for the experimental results. \n\n**Q: Why ALTO mainly works in large batch size while the improvement is still limited?**\n\nIn fact, the optimizer adaptor also works for small batch training since it is just an optimizer itself when $\\alpha$ degenerates to $0$. Our experiment mainly focus on large batch training and we want to use it to show the ability of our optimizer adaptor, since large batch training is very important and difficult in industry field and our improvement (2.5% averagely) is not very limited compared with similar works in this field. If we pretraing a large model with large dataset, using large batch training can much increase the training efficiengy. Only using small batch to training can not fully utilize GPUs to accelerate the training process (data parallelism) . Large batch training is usually required to use the same number of epoches to training as small batch training, which means the number of parameter iteration in large batch training is much less than that in small batch training. Traditional method is to enlarge the learning rate directly, but the learning rate can not be too large (bounded by the geometry of the landscape of certain task). Otherwise, the training will explode. In order to show the abilibty of the adaptor, we do not spend much time on choosing a hyperparameter. This is a possible reason for worse result than SOTA in small bach training.  We will systematically add more experiments of small batch training in our paper.\n\n**Some suggestions for using our optimizer adaptor (for large batch training, ALTO).**\n\nUsually, the larger the batch size is , the larger the $-\\alpha$, $\\beta_1$, and learning rate $\\eta$ are. $\\alpha$ should be postive for small batch training and negative for large batch training. As we said in Line 216, we default to $\\alpha=0.5, \\beta_1=0.01$ for small batch and $\\alpha=-5, \\beta_1=0.99$ for large batch in all our experiments (of course, it can be better).  As we show in Figure 4, there are more better results can be achieved with different $\\alpha$ and $\\beta_1$. We just want to show the method works. Only choosing the best result in experiment may be thought to be cherry-picky."}, {"review_id": "gFUxYwPCQe", "reviewer": "Reviewer_LVE5", "summary": "This paper introduces ALTO, an adaptation of gradient-based optimizers, aimed at improving convergence to flatter and better-generalizing minima by continuing exploration along low-loss valleys even after encountering a local minimum. The key idea is to combine gradient momentum with an estimated acceleration term (based on gradient differences) to help escape sharp minima and encourage further traversal through flat regions of the loss landscape. The authors provide convergence proofs for both convex and non-convex cases, and extensive experiments show that ALTO improves performance in large-batch training tasks.", "questions": "- How does ALTO compare to recent optimizers such as Muon, or second-order optimizers such as Shampoo? \n- Given a model and a dataset, is there a principled way to select $\\alpha$ and $\\beta_1$?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The ALTO method elegantly integrates an EMA-based acceleration term into existing first-order optimizers and is backed by rigorous convergence guarantees in both convex and non-convex settings.\n- The proposed optimizer performs well across various tasks in computer vision, natural language processing, and reinforcement learning, which suggests its versatility. \n- The authors conduct thorough ablation studies that validate the necessity of each design component, including acceleration in the first- and second-moment estimates, bias correction, and layer-wise scaling.\n\n**Weaknesses**\n- While the method excels in large-batch settings, its performance in small-batch training is either comparable or worse than baseline optimizers (e.g., Table 4, where SGD outperforms ALTO on CIFAR-10 with batch size 128). This suggests ALTO may not guarantee good convergence rate unless the batch size is very large.\n- There could be more discussion on how the convergence results compared to other optimizers.", "was_revised": true, "final_justification": "The author's response has addressed my concerns regarding comparison to other optimizers.", "rebuttal": "Thanks for your carefully reading. I will summarize your questions and answer them one by one. \n\n**Q1: How does ALTO compare to recent optimizers such as Muon, or second-order optimizers such as Shampoo?**\n\nWe have compared our method with some recent similar optimizers in section 5. Due to the length limitations of Neurips paper, we may not compared our method with some recent works. I think the biggest different between our method and Muon or Shampoo is the computational complexity. Large computational complexity means long training time of an epoch. If the number of parameter is $N$, the computational complexity of our method is $\\mathcal{O}(N)$. If the weight matrix is square matrix, the computational complexity of Muon and Shampoo is $\\mathcal{O}(N^{\\frac{3}{2}})$ at least. As for performance comparison, we will conduct a experiment on this. In fact, we design a optimizer adaptor, not an optimizer, so we can apply this adaptor to Muon and Shampoo.\n\n**Q2: Given a model and a dataset, is there a principled way to select $\\alpha$ and $\\beta_1$?**\n\nA general but simple principled way is given in section 4.5. Indeed, the $\\alpha$ and $\\beta_1$ are determinded by the geometry of the landscape of a certain task, with is determinded by models and datasets by part. If the model without skip connection, usually the landscape is relatively rough, we should choose large $\\alpha$ and small $\\beta_1$. If the dataset is large, usually the landscape is relatively smooth, we should choose small $\\alpha$ and large $\\beta_1$."}, {"review_id": "VMxtsG0njP", "reviewer": "Reviewer_uGBV", "summary": "The paper aims to develop an algorithm that converges to local minima that have better generalization performance, compared to traditional gradient-based algorithms such as Adam and SGD. Starting from the intuition that flatter minima may have better generalization, the authors propose ALTO – an algorithm that has the usual (stochastic) gradient component in its update for being captured by wide local minima but also a second component that is for escaping narrow local minima. Theoretical analysis of ALTO’s convergence is provided under certain conditions, but there is no proof for the claimed better generalization or flatter minima.", "questions": "N/A", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\nThe paper conceptually utilizes second order information of the loss landscape, namely the Hessian matrix, for escaping sharp minima and finding flat ones. Generally, it is computationally expensive to directly use Hessian, this paper smartly avoided such computation by using gradient difference $g_k-g_{k-1}$ as an approximation. Such a design intuitively makes sense.\n\nThe proposed method achieves good test performance in various tasks, better than existing algorithms in many such tasks. \n\n**Weaknesses**:\n\nThe paper provides no evidence, neither theoretically nor numerically, showing the proposed methods indeed find flat minima, which is the major claim of the paper though. Theoretical analysis of the paper only shows convergence of the algorithm, it would be better if the paper analyzes/discusses where it converges to or whether it converges to flatter minima. It would also enhance the paper if there is a comparison of the sharpness/flatness of the convergence point of different algorithms.\n\nThe proposed algorithm introduces two hyper-parameters. On one hand, this makes the algorithm less usable in practice; on the other hand, it is unclear whether the improved test accuracy reported in the paper is because of the flexibility that comes with the additional hyper-parameters or it is from the claimed flatter minima.\n\nThe paper claims “accelerated convergence” and “preferential convergence” (Line 44), however, there is no evidence of the claims.\n\nMinor:\n\nLine 116: missing reference", "was_revised": true, "final_justification": "[+] The algorithm design is theoretical motivated, and intuively makes sense. \n\n[+] Not much additional computation is needed, although implicitly using second order information.\n\n[+] Good performance in experiments.\n\n[-] No theoretical or experimental evidence of the major claim.\n\n[-] Introduces two hyer-parameters, which introduces additional effort to tune. Rebuttal partially addressed this concern.", "rebuttal": "Thanks for your carefully reading. I will summarize your questions and answer them one by one. \n\n**Q1: The paper provides no evidence, neither theoretically nor numerically, showing the proposed methods indeed find flat minima, which is the major claim of the paper though.**\n\n\"It would also enhance the paper if there is a comparison of the sharpness/flatness of the convergence point of different algorithms\" is a good advice. We will add a  such comparison. In fact, our optimizer try to reduce $\\\\|\\nabla f\\\\|$, which is directly related to the sharpness of a local minimum and Hessian matrix.  SInce the optimization problem is a stochastic optimzation problem, we can not even garantee the final minimum is a global minimum, let alone the flatest one. We only need to let the optimizer incline to flatter minima, so we require the optimizer to reduce $\\\\|\\nabla f\\\\|$. This is what \"preferential convergence\" refers to. As for “accelerated convergence”, many expriments show this (Table 17, Figure 14, and etc.). \n\n**Q2: The proposed algorithm introduces two hyper-parameters. On one hand, this makes the algorithm less usable in practice; on the other hand, it is unclear whether the improved test accuracy reported in the paper is because of the flexibility that comes with the additional hyper-parameters or it is from the claimed flatter minima.**\n\nIndeed, we did not fine-tuned the hyperparameter. In small batch training case, the performance is even little worse than baseline. And we said in Line 216, we set $\\alpha$ and $\\beta_1$ as fixed default value in all our experiments, we do not introduce the flexilibty.\n\n**Q3:The paper claims “accelerated convergence” and “preferential convergence” (Line 44), however, there is no evidence of the claims.**\n\nPlease see answer of **Q1**."}]

Author rebuttal:

Meta-review: This paper introduces ALTO, an adaptor for gradient-based optimizers designed to find better minima by continuing to explore the loss landscape along low-loss "valleys" rather than halting at the first local minimum it encounters. The core mechanism involves augmenting the standard gradient update with an acceleration term, approximated by the difference between consecutive gradients (g(k) - g(k-1)), which helps the optimizer escape sharp minima and traverse flat regions more effectively. The reviewers (9xUK, LVE5) found this approach to be well-motivated, elegant, and supported by a rigorous theoretical analysis that provides convergence guarantees in both convex and non-convex settings. The paper's primary strength, noted by all reviewers, lies in its extensive and convincing empirical results, which demonstrate significant performance improvements over state-of-the-art optimizers, particularly in the challenging and practically relevant domain of large-batch training.

The initial reviews raised several important concerns. A key point of discussion, highlighted by reviewers 9xUK, vVfW, and LVE5, was that the method's strong performance was primarily demonstrated in large-batch settings, with less significant or even inferior results in small-batch scenarios. Reviewers 9xUK and uGBV also questioned the introduction of two new hyperparameters (alpha, beta_1) and their tuning sensitivity. Finally, reviewer uGBV noted a lack of direct experimental evidence confirming that ALTO indeed finds flatter minima. The authors provided a comprehensive rebuttal that effectively addressed most of these points. They clarified that their focus was intentionally on the difficult large-batch training problem and provided clear, practical intuition for setting the new hyperparameters based on batch size—linking them to a trade-off between exploration (large-batch) and exploitation (small-batch). This explanation satisfied reviewers 9xUK, vVfW, and LVE5. The authors also agreed to add further discussion, improve formatting, and include numerical comparisons of minima sharpness in the final version to address reviewer uGBV's valid point.

This paper presents a novel and well-supported contribution to the field of optimization. The proposed method of using an acceleration term to facilitate valley exploration is a valuable idea that demonstrably improves upon strong baselines in large-batch training across various domains, including CV, NLP, and RL. The discussion period was constructive, and the authors showed a clear commitment to improving the paper based on reviewer feedback. Although the final scores are borderline, I believe the paper has significant enough contributions to be of interest to the NeurIPS community. I therefore recommend that this paper be accepted. I strongly encourage the authors to incorporate the detailed feedback from all reviewers and the clarifications from the rebuttal period into the final version of their paper.
Decision: Accept (poster)