Title: Mechanic: A Learning Rate Tuner
Venue: NeurIPS 2023
Accepted: yes
Keywords: optimization

deep learning

online convex optimization
Abstract: We introduce a technique for tuning the learning rate scale factor of any base optimization algorithm and schedule automatically, which we call Mechanic. Our method provides a practical realization of recent theoretical reductions for accomplishing a similar goal in online convex optimization. We rigorously evaluate Mechanic on a range of large scale deep learning tasks with varying batch sizes, schedules, and base optimization algorithms. These experiments demonstrate that depending on the problem, Mechanic either comes very close to, matches or even improves upon manual tuning of learning rates.

Mechanic: A Learning Rate Tuner

Abstract

We introduce a technique for tuning the learning rate scale factor of any base optimization algorithm and schedule automatically, which we call MECHANIC. Our method provides a practical realization of recent theoretical reductions for accomplishing a similar goal in online convex optimization. We rigorously evaluate MECHANIC on a range of large scale deep learning tasks with varying batch sizes, schedules, and base optimization algorithms. These experiments demonstrate that depending on the problem, MECHANIC either comes very close to, matches or even improves upon manual tuning of learning rates.

1 Introduction

Modern deep learning is driven by first-order stochastic optimization algorithms. These are algorithms that are designed to solve the classical stochastic optimization problem:

where z is a minibatch of examples, x ∈ R d is the model parameters, and f is the loss incurred by using weights x on the minibatch z . A first-order algorithm follows the protocol:

Output a t th iterate x t .

Sample a random minibatch z t .

Compute g t = ∇ f ( x t , z t ) (the gradient is taken with respect to x t only).

Possibly update some internal algorithm state based upon g t in preparation for computing the next iterate x t +1 .

The prototypical such optimization algorithm is stochastic gradient descent (SGD), which exemplifies the attractive features of this approach: it is computationally cheap (running in O ( d ) time per update), and with proper tuning obtains minimax optimal convergence guarantees [1, 2]. Modern practice makes use of a wide range of variants of SGD, such SGD with momentum, AdaGrad [3], Adam [4], AdamW [5] or Lion [6]. Interest in such improvements to SGD is driven by the increasing computational demands of training large neural networks: better optimization means cheaper training, which translates to significant savings in terms of time, cost, and environmental impact.

Most modern algorithms for training neural networks are equipped with a scalar 'scale factor' or 'learning rate' hyperparameter s ∈ R . Roughly speaking, these algorithms produce iterates of the form x t +1 = x t + s · u t where u t is some update vector produced as a function of the observed gradients g 1 , . . . , g t (we will use bold font for vectors in R d like u and normal font for all other quantities like s ). As an example, the classical SGD algorithm sets u t = -η t g t for some sequence of scalars { η t } typically called the schedule . The formula for the SGD update is:

The process of selecting the optimal s is called 'tuning', and is a key resource sink in machine learning. The typical approach is simply to try many possibilities to find the empirically optimal s , which requires multiple expensive training runs. This paper introduces a technique for choosing s automatically on-the-fly in order to avoid this expense.

Our procedure, which we call MECHANIC, is a generic wrapper around any base optimization algorithm (BASE) that produces a new optimizer which does not require tuning of the scalar s . The base optimization algorithm is allowed to make any kind of update (for example, it may use any kind of schedule, preconditioner or weight decay). If x BASE t ∈ R d is the t th iterate of BASE, then the wrapper will produce a scalar s t ∈ R and set the t th iterate of the wrapped algorithm to be x t = x BASE 1 + s t ( x BASE t -x BASE 1 ) . As an example, suppose that BASE is the classical SGD algorithm with update equation (1). Then, given s t , we would set x t = x BASE 1 -s t ∑ t -1 i =1 η i g i . Disregarding for now the fact that the gradients g i actually depend on the iterates x i 1 , we see that x t is what the t th iterate of SGD would have been if the schedule were scaled by s t .

Removing tuning of learning rate scalars is already a well-studied problem. One of the main attractions of early work in 'adaptive' optimization such as AdaGrad and Adam [3, 7, 4] is that these algorithms require less tuning than ordinary SGD. Over the last decade, a number of works have aimed to tackle this problem from both an empirical and theoretical perspective [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]. An intuitive approach might take the route of 'hypergradient descent': that is, differentiating the update step of the optimization algorithm itself (e.g. [8, 9]). Strangely, it appears to be difficult to prove that such schemes behave well: theory-based approaches often adopt rather different strategies. Instead, we start from known theory and propose a few important modifications to produce a simple and effective practical implementation. We then rigorously evaluate our algorithm on a variety of datasets. We emphasize that our primary contribution is not new theoretical development, but instead the translation between theory and practice, which involves fusing several known analytical techniques as well as subtle departures from theory.

Previous works that investigate deep learning performance of 'learning-rate free' optimization inspired by theory (e.g. [20, 21, 16, 15]) have already demonstrated impressive results. However, these works typically build 'hand-crafted' algorithms that often blend theoretical analysis with specific empirically successful algorithms such as Adam. In contrast, our wrapper works well with any base algorithm and so can seamlessly integrate new empirical advances in optimization: one does not need intimate familiarity with the analysis of our approach to apply it to a new algorithm.

2 Background: Online Convex Optimization

We develop our formalism via online convex optimization (OCO) [22, 23, 24]. OCO is a popular framework for design and analysis of stochastic optimization algorithms. In brief, for each of T rounds (corresponding to T iterations of optimization), the OCO algorithm must first output a t th iterate x t , after which the algorithm is presented with a t th loss function ℓ t . Typically, one envisions the case ℓ t ( x ) = ℓ ( x , z t ) for some fixed loss ℓ and new data point z t . The goal of an algorithm ALG is to minimize the regret R ALG (˚ x ) :

Many references focus primarily on the case ˚ x = argmin ∑ T t =1 ℓ t ( x ) in order to consider the single scalar value sup ˚ x R T (˚ x ) [25, 26], but we will employ the formulation of regret as a function above instead as it is strictly more general. When ℓ t is convex, then with g t ≜ ∇ ℓ t ( x t ) (or, more generally when g t is a subgradient of ℓ t at x t ), we have:

As a result, the vast majority of OCO algorithms provide analysis that bounds only the linearized regret R ALG linear (˚ x ) . Such algorithms do not need to observe the entire function ℓ t : instead, they only make use of the gradients g t . That is, the t th output of ALG (i.e. x t ) is purely a function of the previous sequence of gradients g 1 , . . . , g t -1 so that ALG is a first-order algorithm.

2.1 Learning the Scale in OCO

Just like stochastic optimization algorithms, most OCO algorithms also require a scale factor s . In fact, many stochastic optimization algorithms (such as SGD and AdaGrad) are also OCOalgorithms. Setting η t = η for all t , SGD ensures the regret bound:

From this equation, one can deduce in hindsight that for any given ˚ x , the optimal value for s is ∥ ˚ x -x 1 ∥ η √ ∑ T t =1 ∥ g t ∥ 2 , which would provide the bound:

This result is minimax optimal [27], but requires knowledge of the unknown optimal s . Very recently, [14, 16, 15] have produced algorithms that estimate the value of ∥ x BASE 1 -˚ x ∥ on-the-fly and use this estimate to quickly identify the optimal scaling value s . These algorithms achieve impressive practical performance, but they require an understanding of the closed-form solution for the optimal s value above. Our goal is to learn the correct scaling regardless of the base algorithm.

To this end, we will leverage a scheme recently developed by [28] that allows one to automatically tune the scale of a base OCO algorithm using another 'meta' OCO algorithm. We reproduce their result below (with notation altered to suit our application) along with the short proof:

Theorem 1 ([28]) . Suppose BASE and TUNER are both OCO algorithms. Let { x BASE t } ⊂ R d indicate the iterates of BASE in response to an arbitrary sequence of gradients { g t } , and let { s t } ⊂ R indicate the iterates of TUNER in response to the sequence of scalars { h t = ⟨ g t , x BASE t -x 1 ⟩} . Define a new online algorithm MECHANIC via:

Then x MECHANIC t ensures regret:

Proof. By definition, for any ˚ s , we have:

With this result, the job of finding the optimal s can usually be completely relegated to TUNER. Although the value ˚ s appears in both terms of the sum R TUNER linear (˚ s ) +˚ sR BASE linear ( x BASE 1 +(˚ x -x BASE 1 ) / ˚ s ) , it turns out that for essentially all plausible BASE algorithms, there is a particular value ˚ s that causes ˚ sR BASE linear ( x BASE 1 + (˚ x -x BASE 1 ) / ˚ s ) to obtain the optimal regret bound (3). Thus, by setting ˚ s to be this value, which is unknown a priori , we need only ensure that R TUNER linear (˚ s ) is small enough to not significantly affect the overall regret bound. Note that this setting of ˚ s is done entirely in the analysis. For example, if BASE is actually SGD with a learning rate η and s = 1 as in (2), we have

Thus, if TUNER obtains low regret, then we will obtain the same regret bound as if we had optimally tuned the scaling factor for SGD. Intuitively, the gradient h t provided to TUNER approximates the gradient over the entire course of the base optimizer rather than just at the most recent iterate. That is, for SGD, h t ≈ df ( x t , z t ) ds where x t = x 1 -s ∑ t -1 k =1 η k g k .

2.2 Parameter-Free Online Optimization

The problem with the above result is that we seem to have simply pushed the problem off to TUNER: what if TUNER itself requires us to set a scale factor? Solving this problem has been the focus of a substantial effort in the online optimization community [29, 30, 10, 11, 28, 12]. The most advanced such algorithms are able to ensure for all ˚ s simultaneously:

Thus, if we set h t = ⟨ g t , x BASE t -x BASE 1 ⟩ , we obtain:

In a theoretical development of this technique, it is necessary to prevent the terms ⟨ g t , x BASE t -x BASE 1 ⟩ 2 from becoming too large (as otherwise R TUNER linear is too large). Typically, this is accomplished by constraining the base algorithm to satisfy ∥ x BASE t -x BASE 1 ∥ ≤ ρ for some user-specified arbitrary ρ . Enforcing such a constraint means that the regret bound (2) would only apply to ∥ ˚ x ∥ ≤ ρ , but ensures that ⟨ g t , x BASE t -x BASE 1 ⟩ 2 ≤ ρ 2 ∥ g t ∥ 2 . Thus, by setting ˚ s = ∥ ˚ x -x BASE 1 ∥ /ρ , the combined algorithm obtains the optimal regret bound of O ( ∥ ˚ x -x BASE 1 ∥ √ ∑ T t =1 ∥ g t ∥ 2 ) (amazingly, the value of ρ is irrelevant!). In practice however, we do not attempt to explicitly enforce any such constraints and simply rely on the intuition that any non-diverging algorithm is unlikely to produce excessively large iterates.

At no point in this process do we need access to the internal state of the base algorithm BASE. This means that improvements to BASE will automatically be reflected in improvements to the overall algorithm. In this paper, we investigate the performance of MECHANIC on deep learning tasks. We consider a variety of settings for the base algorithm BASE (i.e. AdamW, Lion, SGD, with various batch sizes and learning rate schedules of various shapes), and employ a parameter-free algorithm as the TUNER to automatically find the best scale factor for the base algorithm.

3 The MECHANIC algorithm

Our MECHANIC algorithm is specified in Algorithm 1. The algorithm is built by applying Theorem 1 to a parameter-free TUNER algorithm presented in Algorithm 2, which is described along with theoretical analysis in Appendix D. However, when building MECHANIC, we modify the 'pure' theoretically tractable Algorithm 2 to simplify the implementation while still capturing the essential intuition and maintaining the same performance. In the remainder of this section we will provide some intuition behind the TUNER update as used in MECHANIC as well as describing some potentially unfamiliar subtleties relating to our use of exponentially weighted moving averages.

MECHANIC takes as input a base algorithm that generates update vectors u t as described in the previous sections. We then set ∆ t +1 = ∑ t k =1 u k = x BASE t +1 -x BASE 1 . The majority of the algorithm contains our TUNER method, which is a variant of the analytically tractable Algorithm 2, with a

Algorithm 1 MECHANIC

1: Input: Base algorithm BASE, β ∈ [0 , 1] n (default n = 6 , β = (0 . 9 , 0 . 99 , 0 . 999 , 0 . 9999 , 0 . 99999 , 0 . 999999) , λ ∈ R (default λ = 0 . 01 ). s init ∈ R : first non-zero s value (default s init = 10 -8 ). ϵ = 10 -8 : small value for numerical stability.

2: v 0 ← 0 ∈ R n , r 0 ← 0 ∈ R n , m 0 ← 0 ∈ R n , x ref ← x BASE 1 .

3: ∆ 1 ← 0 ∈ R d

4: s 1 ← 0 ∈ R n . // We will use s t,i to indicate the i th coordinate of s t .

5: for t = 1 . . . T do

6: g t ←∇ f ( x t , z t ) . // x t is the t th set of model parameters and z t is the t th minibatch.

7: Send g t to BASE, receive update u k . // On its own, BASE would update x t +1 ← x t + u k .

8: [Optional] Set ∆ t = x t -x ref ( ∑ n s t,n ) + ϵ to save memory instead of storing ∆ t from last round.

9: ∆ t +1 ← ∆ t + u t .

10: h t ← 〈 ∆ t , g t + λ ( ∑ n i =1 s t,n ) ∥ g t ∥ x t ∥ x t ∥ 〉 // Note use of ∆ t rather than ∆ t +1 .

11: m t ← max( βm t -1 , h t ) (multiplications by β and maximum are taken coordinate-wise)

12: v t ← β 2 v t -1 + h 2 t

13: r t ← βr t -1 -s t -1 h t

14: r t ← max(0 , r t ) // This step is used instead of more complicated procedures in Algorithm 2

n 16: s t +1 ← W t √ v t + ϵ

15: W t ← s init · m t + r t

17: x t +1 ← x BASE 1 +( ∑ n i =1 s t +1 ,i ) · ∆ t +1

18: end for

few modifications. Note that the indexing on ∆ is very important and may be counterintuitive: the definition of h t does not include ∆ t +1 , but rather ∆ t . h t is the 'gradient' that is supplied to TUNER, as described by Theorem 1.

To gain some intuition behind the update, let us consider the case that n = 1 and β = 1 . 0 (that is, without employing any exponentially-weighted moving averages). We keep track of the quantity W t = s init · m t -∑ t k =1 h k s k , which is usually called the 'wealth' of the algorithm (the quantity r t = -∑ t k =1 h k s k is sometimes called the 'reward'). s init specifies the starting value for s t and should be an under-estimate of the true optimal scaling. We then set s t +1 = W t √ v t (neglecting the ϵ included for numerical stability). To understand this update strategy, we can re-write the update as:

Thus, the update looks like a combination of an AdaGrad-esque gradient descent step with learning rate scaled by s t and a kind of 'adaptive decay' (multiplication by 1 -h 2 t 2 v t ). The adaptive decay is very important for stabilizing the algorithm: without it the values for s t are prone to unchecked exponential growth due to scaling by s t in s t h t √ v t . Intuitively, this decay is the minimum amount required to prevent instabilities.

In Appendix D, we provide a formal Theorem bounding the regret of a variant of the procedure described above. Roughly speaking, for β = 1 this result suggests:

In fact, the dependence of O (log( T )) in equation (5) can be improved to O ( √ log( T )) via more complicated algorithms (e.g. [28, 12, 31, 32]). However, we favor the simpler update and pleasing resemblance to familiar algorithms like AdaGrad via the Taylor expansion analysis above. Of note, the dependence on s init is very mild: this suggests that we should be able to set s init to a very small

i

=1

value without damaging performance. In practice, we choose s init = 10 -8 , which we expect to dramatically underestimate the optimal value in all scenarios.

Wehypothesize that the simplified TUNER we use in MECHANIC in fact possesses a rigorous theoretical analysis (although perhaps only with respect to simpler non-fully-worst-case adversaries), but demonstrating such a bound appears to involve difficult technical hurdles. In particular, our implemention is designed to be 'scale-free': rescaling the values of g t by any constant scalar will have no effect on s t . This property was first achieved only recently in theoretical analysis of parameter-free algorithms [12], and as-yet requires significantly more involved algorithms [12, 33].

3.1 The use of β

We include β to introduce some recency bias in the statistics recorded by MECHANIC, a common feature of practical optimizers. Mathematically, we accomplish this by up-weighting the t th feedback to TUNER by β -t : h t → h t β -t . Thus, for example, we have v t = ∑ t k =1 h 2 k β -2 kt and r t = -∑ t k =1 h k s k -1 β -k s . Using these weights directly results in numerical stability issues as the weights become exponentially large. Instead, since we only need to maintain the correct ratio W t √ v t , we can cancel a factor of β -t s from both sides, giving the update equations in Algorithm 2.

We found that tuning the value of β can significantly improve performance on different tasks. Thus, we incorporated multiple β values simultaneously in a way that obviates the need for such tuning.

Our approach is inspired by work on 'combining' parameter free algorithms [34]. The idea is simple: parameter-free algorithms typically ensure R linear (0) ≤ ϵ for some constant ϵ set by the user. So, if s t, 1 , . . . , s t,n are the outputs of n parameter-free algorithms with regret bounds R 1 linear (˚ s ) , . . . , R n linear (˚ s ) , we have for any j :

̸

̸

So, with small constant additive overhead in the regret, the sum of all the outputs s t, 1 + · · · + s t,n achieves the same regret as the best of all the outputs. Motivated by this observation, we instantiate n = 6 copies of TUNER with different β values and add their iterates to produce a final scaling.

3.2 Weight decay

Finally, we found that an addition of a peculiar weight-decay-esque term helped significantly on certain tasks, including vision tasks with smaller datasets, multi-objective NLP tasks and especially with reducing the variance in final results for all tasks. Specifically, rather than providing h t = ⟨ g t , ∆ t ⟩ as the input to the TUNER algorithm, we instead provide h t = ⟨ g t + λ ∥ g t ∥ ( ∑ n i =1 s t,i ) x t ∥ x t ∥ , ∆ t ⟩ . We conjucture that this term is helpful in the common case that the base algorithm itself is incorporating regularization or weight-decay.

This extra term is the derivative of the regularizer x ↦→ λ ∥ g t ∥ ( ∑ n i =1 s t,i ) ∥ x ∥ . From a standard theoretical perspective, this regularization may seem overly large. However, it may not have as big an impact as one might imagine because the base algorithm does not see this regularization. Instead, the base algorithm may (or may not) perform weight decay using another method that MECHANIC has no insight into. That said, we do not propose an analytical explanation for this modification. We simply observed that in practice it performed well with a fixed λ = 0 . 01 .

3.3 Runtime and Memory Cost

MECHANIC incurs little additional cost over that of BASE. In Algorithm 1, we denote d -dimensional vectors with bold font, and n -dimensional vectors and scalars with normal font (note that typically n = 6 ). We use 1 additional O ( d ) memory slot, and four O ( d ) -time steps in lines 8, 9, 10 and 17. All other steps are O (1) or O ( n ) time and so have negligible cost.

4 Experiments

In this section we describe our experiments using MECHANIC to tune various base optimizers on different tasks. Note that almost all base optimizer implementations require a user-specified scale factor which is is not directly visible to MECHANIC. We set this value to 1.0 before applying MECHANIC. Since MECHANIC multiplies the base update by s t , setting the base scale factor to 1 . 0 allows us to interpret s t as the 'correct' value for the base scale.

4.1 Masked Language Modeling

We perform BERT pre-training on the Wikibooks dataset following the procedure from [35] with a few minor changes, most notably, we omit the Next Sentence Prediction (NSP) loss following [36]. Masked language modeling (MLM) requires reconstructing randomly masked tokens given an input sequence of tokens. As shown in Table 1, using MECHANIC leads to a noticeable improvement in MLMaccuracy.

Varying batch size and model size: Past works observe that the scale factor s should decrease as either batch size is decreased or model size is increased [37, 38]. To inspect the scale factor that MECHANIC learns, we vary the batch size and model size while pre-training BERT using MECHANIC. As shown in Figure 1, in both cases, MECHANIC learns to decrease the scale factor s when decreasing the batch size and when increasing the model size.

Addition Ablations: Ablation studies on the effects of n , λ , s init can be found in Appendix B.

Finetuning pre-trained models: In addition to pre-training, we evaluate our models on the 5 largest datasets from the GLUE suite [39]. One possible failure mode of MECHANIC tuned pre-trained models could have been that, even though they lead to high accuracy at pre-training time, transfer learning may fail at finetuning time.

To ensure that standard transfer learning pipelines still work with MECHANIC pre-trained models, we finetune them without a learning rate tuner using the AdamW optimizer and find that MECHANIC pre-trained models lead to higher accuracy at pre-training time, and they also outperform in finetuning more often than not. We finetune BERT-B (110M) and BERT-L (340M) models for at most 10 epochs on each of the GLUE datasets and report results on the GLUE dev set in Table 1.

Using MECHANIC for finetuning: Wealso investigated using MECHANIC for finetuning. Typically, to not erase the progress already made, a much lower base learning rate is employed at finetuning

(a) Varying batch size

65

Adamw

batch size=512

64

M-AdamW

10-2

batch size=1024

batch size=4096

AcC

batch size=8192

63

10-3

MLM

S

62

10-4

61

10-5

60

256

512

1024

2048

4096

8192

10-6

0.0

0.5

1.0

# of examples

1.5

2.0

2.5

3.0

1e8

75

10-1

Adamw

BERT-Ti

70

M-AdamW

BERT-S

10-2

BERT-M

65

BERT-B

AcC

60

10-3

M

S

MLI

55

10-4

50

10-5

45

40

BERT-Ti

BERT-S

BERT-M

BERT-B

10-6

0.0

0.5

1.0

1.5

2.0

2.5

3.0

# of examples

1e8

(b) Varying model size

time. This could easily have been a potential failure mode of any kind of automatic learning rate tuner as such strategies might 'explore' a high learning rate at the beginning of the optimization procedure. Fortunately, we observed that this inductive bias typically baked at finetuning time is still maintained when using MECHANIC.

4.2 Image Classification

In this Section, we present results on popular Image Classification tasks. Apart from training from scratch, we also perform transfer learning experiments where we pre-train on the JFT-300M [40] dataset and finetune on ImageNet, Cifar-10 and Cifar-100 datasets. We follow the exact setting employed in [41] for both pre-training and finetuning.

As shown in Table 2, MECHANIC is quite competitive across the board and produces results either very close to the baseline or better. Since MECHANIC optimizes for the train loss, in general, we observe that it results in better test performance on tasks with large amounts of data where the model is unable to overfit to the train set. For instance, we see that MECHANIC beats the baseline substantially when pre-training ViT models on JFT-300M, whereas it lags slightly behind on smaller datasets like ImageNet-1k or CIFAR-10/100. Even though we fix λ to 0.01 as default for all our reported experiments, we find that for small datasets like CIFAR-10, increasing it led to better test performance.

4.3 Comparison with D-adaptation

Recently, [16] introduced the D-adaptation algorithm, with the same goal of learning the correct scale s for SGD and Adam base optimizers. D-adaptation showed impressive empirical results on a range of popular deep learning tasks, so we compare MECHANIC with D-adaptation on a selection of tasks that D-adaptation worked well on, using code provided by the authors. Hyper-parameter settings were kept the same to ensure a fair comparison. In contrast to D-adaptation, MECHANIC does not require modification for different base optimizers and, as shown in Figure 2, it remains quite

competitive on small datasets like CIFAR-10/100 while outperforming both a manually tuned baseline and D-adaptation on bigger tasks like IWSLT14 and language modeling on BookWiki dataset. We present additional results in Appendix C.3, including a comparison on a suite of 12 logistic regression problems.

5 Conclusion

MECHANIC is a new technique for scaling the updates of any base optimization algorithm. Our approach provides a practical implementation of recent developments in optimization theory, and is able to match the performance of tuned baselines on large-scale machine learning tasks. This work suggests several natural future directions. First, is there a theoretical motivation for our weight-decay term? Next, is it possible to leverage similar techniques to learn a per-layer scale factor? Such a capacity would not significantly increase computation cost, but by allowing more degrees of freedom may yield a method that significantly outperforms baselines since it is infeasible to manually tune a scale factor for every layer.

Acknowledgments

Ashok Cutkosky acknowledges funding support from NSF grant CCF-2211718, an Amazon research award, and Google.

References

Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

0

50

100

150

200

250

300

Epoch

75

80

85

90

95

Test Accuracy (%)

CIFAR-10 (WRN-16-8)

D-Adapt SGD (95.56% SE 0.04)

SGD (95.59% SE 0.03)

Mechanic (95.46% SE 0.03)

0

50

100

150

200

250

300

Epoch

55

60

65

70

75

80

Test Accuracy (%)

CIFAR-100 (DenseNet)

D-Adapt SGD (76.90% SE 0.06)

SGD (76.41% SE 0.14)

Mechanic (75.44% SE 0.08)

0

10000

20000

30000

40000

50000

60000

Step

5

6

7

8

Test Loss

IWSLT14 (LSTM)

Adam (4.31 SE 0.003)

D-Adapt Adam (4.33 SE 0.003)

0

10000

20000

30000

40000

50000

60000

Step

15

20

25

30

35

40

Test Perplexity

BookWiki (GPT Transformer)

Adam (19.49 SE 0.012)

D-Adapt Adam (19.46 SE 0.019)

Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. arXiv preprint arXiv:1912.02365 , 2019.

J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. In Conference on Learning Theory (COLT) , pages 257-269, 2010.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2018.

Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, et al. Symbolic discovery of optimization algorithms. arXiv preprint arXiv:2302.06675 , 2023.

H. Brendan McMahan and Matthew Streeter. Adaptive bound optimization for online convex optimization. In Proceedings of the 23rd Annual Conference on Learning Theory (COLT) , pages 244-256, 2010.

Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. Journal of machine learning research , 18, 2018.

Kartik Chandra, Audrey Xie, Jonathan Ragan-Kelley, and Erik Meijer. Gradient descent: The ultimate optimizer. Advances in Neural Information Processing Systems , 35:8214-8225, 2022.

Francesco Orabona and D´ avid P´ al. Coin betting and parameter-free online learning. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29 , pages 577-585. Curran Associates, Inc., 2016.

Ashok Cutkosky and Kwabena Boahen. Online learning without prior information. In Conference on Learning Theory , pages 643-677, 2017.

Zakaria Mhammedi and Wouter M Koolen. Lipschitz and comparator-norm adaptivity in online learning. Conference on Learning Theory , pages 2858-2887, 2020.

Kfir Levy, Ali Kavis, and Volkan Cevher. Storm+: Fully adaptive sgd with recursive momentum for nonconvex optimization. Advances in Neural Information Processing Systems , 34: 20571-20582, 2021.

Yair Carmon and Oliver Hinder. Making sgd parameter-free. Conference on Learning Theory , 2022.

Maor Ivgi, Oliver Hinder, and Yair Carmon. Dog is sgd's best friend: A parameter-free dynamic step size schedule. arXiv preprint arXiv:2302.12022 , 2023.

Aaron Defazio and Konstantin Mishchenko. Learning-rate-free learning by d-adaptation. arXiv preprint arXiv:2301.07733 , 2023.

Naman Agarwal, Rohan Anil, Elad Hazan, Tomer Koren, and Cyril Zhang. Disentangling adaptive gradient methods from learning rates. arXiv preprint arXiv:2002.11803 , 2020.

Zhou Lu, Wenhan Xia, Sanjeev Arora, and Elad Hazan. Adaptive gradient methods with local guarantees. arXiv preprint arXiv:2203.01400 , 2022.

Xinyi Chen and Elad Hazan. A nonstochastic control approach to optimization. arXiv preprint arXiv:2301.07902 , 2023.

Francesco Orabona and Tatiana Tommasi. Training deep networks without learning rates through coin betting. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA , pages 2157-2167, 2017. URL http://papers.nips.cc/paper/ 6811-training-deep-networks-without-learning-rates-through-coin-betting .

Ashok Cutkosky and Kwabena A Boahen. Online convex optimization with unconstrained domains and losses. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29 , pages 748-756. Curran Associates, Inc., 2016.

Shai Shalev-Shwartz. Online learning and online convex optimization. Foundations and Trends in Machine Learning , 4(2):107-194, 2011.

Elad Hazan. Introduction to online convex optimization. arXiv preprint arXiv:1909.05207 , 2019.

Francesco Orabona. A modern introduction to online learning. arXiv preprint arXiv:1912.13213 , 2019.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning (ICML-03) , pages 928-936, 2003.

S. Shalev-Shwartz. Online Learning: Theory, Algorithms, and Applications . PhD thesis, The Hebrew University of Jerusalem, 2007.

Jacob Abernethy, Peter L Bartlett, Alexander Rakhlin, and Ambuj Tewari. Optimal strategies and minimax lower bounds for online convex games. In Proceedings of the nineteenth annual conference on computational learning theory , pages 415-424, 2008.

Ashok Cutkosky and Francesco Orabona. Black-box reductions for parameter-free online learning in Banach spaces. In Conference On Learning Theory , pages 1493-1529, 2018.

Brendan Mcmahan and Matthew Streeter. No-regret algorithms for unconstrained online convex optimization. In Advances in neural information processing systems , pages 2402-2410, 2012.

Brendan McMahan and Jacob Abernethy. Minimax optimal algorithms for unconstrained linear optimization. In Advances in Neural Information Processing Systems , pages 2724-2732, 2013.

Liyu Chen, Haipeng Luo, and Chen-Yu Wei. Impossible tuning made possible: A new expert algorithm and its applications. In Conference on Learning Theory , pages 1216-1259. PMLR, 2021.

Francesco Orabona and D´ avid P´ al. Parameter-free stochastic optimization of variationally coherent functions. arXiv preprint arXiv:2102.00236 , 2021.

Andrew Jacobsen and Ashok Cutkosky. Parameter-free mirror descent. In Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 4160-4211. PMLR, 2022.

Ashok Cutkosky. Combining online learning guarantees. In Proceedings of the Thirty-Second Conference on Learning Theory , pages 895-913, 2019.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423 .

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint , abs/1907.11692, 2019. URL https://arxiv.org/ abs/1907.11692 .

L. Bottou and O. Bousquet. The tradeoffs of large scale learning. In Advances in Neural Information Processing Systems (NIPS) , 2008.

Shixiang Shane Gu, Sergey Levine, Ilya Sutskever, and Andriy Mnih. Muprop: Unbiased backpropagation for stochastic neural networks. CoRR , abs/1511.05176, 2015.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. ArXiv , abs/1804.07461, 2018.

Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. 2017 IEEE International Conference on Computer Vision (ICCV) , Oct 2017. doi: 10.1109/iccv.2017.97. URL http://dx.doi.org/10.1109/ iccv.2017.97 .

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=YicbFdNTTy .

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research , 15(56):1929-1958, 2014. URL http://jmlr.org/papers/v15/ srivastava14a.html .

Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V. Le. Symbolic discovery of optimization algorithms, 2023.

Ashok Cutkosky. Artificial constraints and hints for unbounded online learning. In Proceedings of the Thirty-Second Conference on Learning Theory , pages 874-894, 2019.

Zakaria Mhammedi, Wouter M Koolen, and Tim Van Erven. Lipschitz adaptivity with multiple learning rates in online learning. In Conference on Learning Theory , pages 2490-2511. PMLR, 2019.

Michal Kempka, Wojciech Kotlowski, and Manfred K Warmuth. Adaptive scale-invariant online algorithms for learning linear models. In International Conference on Machine Learning , pages 3321-3330, 2019.

A Limitations

In this paper, we introduce a technique for automatically learning the right scale of the learning rate called MECHANIC and evaluate it on a broad range of practical deep learning problems and settings. We find that, depending on the problem, MECHANIC can be quite effective and even surpass performance of manual tuning of learning rates at a fraction of the cost. We also find that in addition to training from scratch, MECHANIC also works for finetuning.

While the initial set of results are encouraging, many challenges remain. Firstly, we found that MECHANIC does not seem to work well with dropout [42]. While MECHANIC is effective against noise from sampling, we believe there may be a more fundamental reason why dropout does not work well with MECHANIC. Second, MECHANIC re-purposes the gradient coming from the train set for learning the learning rate, which means it optimizes for train loss. This is different from manual tuning of learning rates where researchers tune it based on performance on the validation set. A principled way to handle this discrepancy is also in interesting avenue of future research.

B Additional Ablations

B.1 Setting BASE learning rate with Mechanic

Even though we recommend setting peak learning rate of Base optimizer to 1.0, to make sure that Mechanic truly is insenstive to the peak learning rate and robust to the choice of this hyperparameter, we conduct a study where we vary peak learning rate of AdamW on BERT-B masked language modeling task. As shown in Table 3, Mechanic is largely robust against choice of peak LR set for AdamW on BERT-B MLM task.

Table 3: MECHANIC is largely robust against choice of peak LR set for AdamW on BERT-B MLM task.

B.2 Robustness to s init

B.3 Ablation of λ

B.4 Number of β values

C Additional Experimental Details

C.1 Hyperparams for BERT

C.2 Hyperparams for Image Classification

ResNet18/CIFAR10

WideResNet-40-10/CIFAR100

0.07

0.020

0.06

0.015

0.05

0.04

0.010

0.03

0.02

0.005

0.01

0.000

0.00

0

25

50

75

100

125

150

175

200

0

25

50

75

100

125

150

175

200

C.3 Hyperparams and additional results on comparisons with D-adaptation

D Theoretical Analysis

Here we provide the theoretically tractable version of TUNER as well as its analysis.

D.1 Algorithmic Simplifications

To simplify the implementation of MECHANIC, we replaced all of the red text in Algorithm 2 with a single line r t ← max(0 , r t ) right after the definition of r t . This is motivated by two ideas.

0

20

40

60

80

100

Epoch

60

70

80

90

Accuracy (%)

Sensorless

Adam (89.5 SE 0.01)

D-Adapt Adam (90.0 SE 0.07)

0

20

40

60

80

100

Epoch

70

75

80

85

90

95

Accuracy (%)

Aloi

Adam (94.1 SE 0.01)

D-Adapt Adam (97.1 SE 0.01)

0

20

40

60

80

100

Epoch

70

80

90

100

Accuracy (%)

DNA

Adam (100.0 SE 0.00)

D-Adapt Adam (100.0 SE 0.00)

0

20

40

60

80

100

Epoch

20

40

60

Accuracy (%)

Adam (72.0 SE 0.16)

D-Adapt Adam (72.3 SE 0.19)

0

20

40

60

80

100

Epoch

20

40

60

80

100

Accuracy (%)

Adam (98.3 SE 0.11)

D-Adapt Adam (98.6 SE 0.00)

0

20

40

60

80

100

Epoch

70

72

74

76

78

Accuracy (%)

Letter

Adam (77.8 SE 0.01)

D-Adapt Adam (77.7 SE 0.01)

0

20

40

60

80

100

Epoch

70

80

90

Accuracy (%)

Pendigits

Adam (96.2 SE 0.02)

D-Adapt Adam (95.8 SE 0.02)

0

20

40

60

80

100

Epoch

20

40

60

80

Accuracy (%)

smallNORB

Adam (92.9 SE 0.03)

D-Adapt Adam (95.8 SE 0.04)

0

20

40

60

80

100

Epoch

90

92

94

96

98

Accuracy (%)

USPS

Adam (98.6 SE 0.01)

D-Adapt Adam (98.6 SE 0.01)

0

20

40

60

80

100

Epoch

30

40

50

60

70

80

Accuracy (%)

Vehicle

Adam (82.1 SE 0.08)

D-Adapt Adam (81.9 SE 0.16)

0

20

40

60

80

100

Epoch

20

40

60

80

Accuracy (%)

Vowel

Adam (77.4 SE 0.11)

D-Adapt Adam (77.4 SE 0.15)

0

20

40

60

80

100

Epoch

20

40

60

80

100

Accuracy (%)

Wine

Adam (100.0 SE 0.00)

D-Adapt Adam (100.0 SE 0.00)

0

50

100

150

200

250

300

Epoch

10

2

10 0

10 2

Train Loss

CIFAR-10 (WRN-16-8)

D-Adapt SGD (1.3e-03 SE 4.9e-05)

SGD (1.6e-03 SE 4.6e-05)

Mechanic (2.9e-03 SE 9.1e-05)

0

50

100

150

200

250

300

Epoch

10

1

10 1

Train Loss

CIFAR-100 (DenseNet)

D-Adapt SGD (1.3e-02 SE 5.0e-04)

SGD (1.3e-02 SE 5.2e-04)

Mechanic (9.9e-03 SE 5.4e-04)

0

10000

20000

30000

40000

50000

60000

Step

20

40

60

80

100

Training Perplexity

BookWiki (GPT Transformer)

Adam (20.19 SE 0.084)

D-Adapt Adam (20.16 SE 0.093)

0

10

20

30

40

50

Epoch

0.4

0.5

0.6

0.7

0.8

Train loss

Adam (0.4281 SE 0.00021)

D-Adapt Adam (0.4276 SE 0.00028)

0

10000

20000

30000

40000

50000

60000

Step

5.0

7.5

10.0

12.5

Training Loss

IWSLT14 (LSTM)

Adam (4.27 SE 0.003)

D-Adapt Adam (4.03 SE 0.002)

Algorithm 2 TUNER(theoretically tractable version

First, we conjecture that the clip operation using q t / √ v t may even be unnecessary in theory 2 : we observed no change from removing this operation in practice, and observe that the update has an intuitive interpretation via the Taylor expansion discussed in Section 3.

Second, the clip operation on h t using m t -1 is essentially designed to prevent the wealth W t from becoming negative or zero using the gradient truncation technique employed by [44, 45, 12]. While less consistent with known theory, we found it simpler to ensure the wealth W t does become negative simply by clipping r t directly (we did not clip W t to be nonnegative as W t = 0 would cause the algorithm to output s t = 0 for all future iterations). We found these changes simplified the algorithm while having no noticeable effect on the performance. Although these deviations technically do not come with guarantees, the accomplish similar intuitive goals and so we expected (and observed) that they simplified the implementation while not damaging performance.

D.2 Eliminating W 0 in favor of s init

While TUNER makes use of the 'initial wealth' value W 0 , MECHANIC instead adopts a varying value for W 0 proportional to s init · m t . This makes the first s value proposed by MECHANIC equal to s init , which is more intuitive to set than W 0 . The exponential growth in s allows us to set s init to a very small value of 10 -8 . It also makes the values for s 'scale-free' in the sense that rescaling the updates u t by any constant will have no effect on the resulting s t .

D.3 Regret Bound

Theorem 2. With β = 1 , Algorithm 2 guarantees for all ˚ s ≥ 0 :

Proof. First, we employ an argument developed by [44]:

So, in the following, it suffices to bound ∑ T t =1 ˆ h t ( s t -˚ s ) . This is helpful because we will be able to use the bound | ˆ h t | ≤ m t -1 , and m t -1 is known before ˆ h t is revealed.

As is typical in the analysis of parameter-free algorithms, the proof proceeds by lower-bounding the wealth. Define a function a ( x ) piecewise by:

Notice that a ( x ) is differentiable, monotonically increasing and 1-Lipschitz. We are going to roughly show that W t ≥ Ω(exp( a ( -∑ t k =1 ˆ h k / √ v t ))) , after which the regret bound will follow from the wealth-regret duality [10].

The key technical inequality in the proof is the following: for any A , B , m with B ≥ 4 m 2 , and any | x | ≤ m , we have:

Once (6) is established, we proceed as follows: defining c t = clip ( ∑ t -1 k =1 ˆ h k / √ 4 m 2 t -1 + v t -1 , 0 , 1) √ v t , we have:

where we have used c t ≤ 1 / 2 and the identity log(1 -x ) ≥ -x -x 2 for x ≤ 1 / 2 (which applies since ˆ h t ≤ m t -1 by definition). Now, set A = ∑ t -1 k =1 ˆ h k and B = 4 m 2 t -1 + v t -1 and x = ˆ h t in (6), we see that:

Thus by telescoping the sum, we have:

Now, observe that 2 ˆ h 2 t 4 m 2 t -1 + v t -1 ≤ 2 ˆ h 2 t v t ≤ 2(log( v t ) -log( v t -1 )) , so we have ∑ T t =1 ˆ h 2 t v t -1 ≤ 2 log( Tm T /m 1 ) so that overall:

Thus, if we define p ( H ) = W 0 m 1 T 2 m T exp [ a ( H √ v T )] , we have W T ≥ p ( -∑ T k =1 ˆ h k ) . Now, we employ the reward-regret duality:

Where p ⋆ is the Fenchel conjugate of p and the evaluation of the conjugate follows from direct calculation (see, e.g. [10, 28, 46]).

Thus, to prove the theorem we need only show (6). This is established via casework in a manner similar to [46].

Case 1. -A √ B ≤ 0 : In this case, the statement is equivalent to: x 2 B ≥ a ( -A -x √ B + x 2 ) . Note that since -A √ B ≤ 0 , we have A ≥ 0 . Therefore:

Further, we clearly have -x √ B + x 2 ≤ 1 so that:

So, in the following we assume -A √ B ≥ 0 .

Case 2. -A -x √ B + x 2 ≤ 0 : In this case, it suffices to show -x √ B clip ( -A √ B , 0 , 1 ) ≥ -x 2 B . The case assumption implies m 2 ≥ x ≥ -A ≥ 0 . Therefore, since B ≥ 4 m 2 , clip ( -A √ B , 0 , 1 ) = -A √ B so that -x √ B clip ( -A √ B , 0 , 1 ) = xA B ≥ -x 2 B as desired.

So, in the following we now further assume -A -x √ B + x 2 ≥ 0 .

Case 3. -A √ B ∈ [0 , 1] : We have a ( -A √ B ) = A 2 2 B , and also since a ( z ) ≤ 1 2 z 2 for all z , a ( -A -x √ B + x 2 ) ≤ ( A + x ) 2 2( B + x 2 ) . Thus, it suffices to show that A 2 2 B + xA B ≥ ( A + x ) 2 2( B + x 2 ) -x 2 B , but this is equivalent to ( A + x ) 2 2 B ≥ ( A + x ) 2 2( B + x 2 ) -x 2 2 B , which clearly holds.

Case 4: -A √ B ≥ 1 and -A -x √ B + x 2 ≥ 1 : In this case it suffices to show -A √ B -x √ B ≥ -A -x √ B + x 2 -x 2 B . To see this, we have:

where in the second-to-last line we have used the fact that h ↦→ 1 √ B + h is a convex in h , and in the last line we have used √ B ≥ m ≥ x .

Case 5: -A √ B ≥ 1 and -A -x √ B + x 2 ∈ [0 , 1) : In this case we need to show -A √ B -1 2 ≥ ( A + x ) 2 2( B + x 2 ) -x 2 B + x √ B . To see this, we first observe that since -A -x √ 2 ∈ [0 , 1) , we have

B + x

Thus, by quadratic formula, A ≥ -x -√ x 2 + B , so that we have A ∈ [ -x -√ x 2 + B, -√ B ] . Next, our target identity can be rearranged into an equivalent form as follows:

so that it suffices to show the second line above. Notice that the RHS of this expression is convex in A and so is maximized at the boundary of the range [ -x -√ x 2 + B, -√ B ] . When A = -√ B we have:

Alternatively, when A = -x -√ x 2 + B , we have

This establishes the claimed inequality (6) and completes the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on a method (MECHANIC) for tuning the learning rate of any given base optimizer. This is compatible with any given base optimizer along with a learning rate schedule*. Let u_1, u_2,..u_t be the steps of the base optimizer up until now, then MECHANIC chooses a s_t such that the neural network weights are set to w_{init} + s_t (\\sum_i u_i). They do this by leveraging a similar scheme developed for online convex optimization. They implement this method with some minor changes to simplify the implementations and improve the convergence speed (by using momentum and a variant of weight decay).\n\nThe authors show that in experiments on masked language modeling and image classification, using MECHANIC leads to faster optimization (although it does sometimes lead to worse generalization).\n\n*One thing to note is that this is not a replacement for learning rate schedulers. In the CIFAR10 plots (Fig 2), there is still a need for learning rate decay. Rather, it seems to help over the tuned baselines for any given learning rate schedule.\n\nTypos:\nLine 32: The end quote is flipped.\nLine 17 in algorithm: s_{t+1,n} should be s_{t+1,i}.\nLine 113: Text is missing.\nTable 3, last line: MAdamW should be M-LION.\n\nSTRENGTHS:\nImproving over tuned baselines is an important contribution. The authors do so for large scale deep learning datasets such as BERT and JFT-300m. It is a bonus that the authors are able to do so by using a theoretically motivated algorithm.\n\nWEAKNESSES:\nThe results would have been more impressive if the authors had used more recent baselines such as Mosaic ML’s BERT codebase.\n\nQUESTIONS:\n1. Could the authors provide some intuition about the fact that s_t is multiplied to the sum of the steps taken by the network till now rather than the next step (as is done in learning rate schedulers). \n2. It is surprising that the learned value of s (Figure 1 and 3) is usually  << .1? For comparison against tuned learning rates I would have expected s to be not significantly smaller/larger than 1. Does this suggest that the learning rates being compared to could have been reduced by a factor of 10 without significant effects? Could the authors provide some intuition about the learnt values of s? \n3. The main body says that lamba is always set to .01 while Table 2 says that lambda=.1 in one of the cases. Was lambda tuned for MECHANIC? If so, was weight decay tuned for the base optimizers? \n4. I may have missed the justification referred to in “This seems like a significant issue to disregard, but we will provide mathematical justification presently.”. Could the authors point where this is? Also, I think the first line of the proof of 3.1 is assuming this i.e. that g_t is independent of s_t.\n5. Line 100: What is “s”? The previous lines only had s^{o}.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a parameter-free technique to tune the learning-rate scale factor automatically, which can be applied to any given base optimization algorithm to match its performance given carefully tuned hyper parameters. This approach is mainly empirical in nature, though grounded in reduction from recent advancements in theoretical understanding of online convex optimization. The implementation is also \"scale-free\", i.e. invariant to scaling the stochastic gradients $g_t$ by a constant. The authors provide experimental evidence to support their claims, including both training from scratch as well as transfer learning, and across domains and model architectures.\n\nSTRENGTHS:\nWith the rapid rise in both demand and costs to train large models, the compounding effects of hyper-parameter tuning transitioned from being a technical inconvenience to a true bottleneck. While there have been some advancements in parameter-free optimization, any practical contribution to this field is highly valuable. A general-use, easy to implement extension to existing optimizers that act as a force multiplier and improves upon their results falls comfortably into this category.\nIn addition to providing some theoretical justification to their method, the authors also provide a detailed, easy-to-follow background to the field and its main results thus far. The paper also include some interesting intuitions by the authors, instead of post-hoc explanations of empirical findings. \nThe experiments performed by the authors include various domains and model architecture, and notably include both pre-training from scratch as well as fine-tuning which are known to behave differently.\n\nWEAKNESSES:\nAs stated by the authors themselves: their \"primary contribution is not new theoretical development, but instead the translation between theory and practice\". Since this paper is focused mainly on empirical evidence rather than new theoretical results, the burden of proof grows larger. While the authors do perform various experiments, there is key information that is missing to fully convince in the efficacy of the approach.\n1. The experimental setting is lacking some details. Namely, how were the hyper-parameters for the base algorithm chosen? Following existing procedures while altering some hyper-parameters (e.g. dropping the NSP loss) may affect the results. Additionally, the given hyper-parameters are not guaranteed to be optimal in the first place. To make the claim that MECHANIC helps in achieving competitive results to carefully tuned optimizers, the base optimizers should indeed be carefully tuned.\n2. Moreover, statistical significance of the results is not assured, as no measure for variance in the results is given. Very often, the advantage of the MECHANIC based training is small enough to potentially be attributed to the inherent noise in the evaluations.\n3. It is not clear that all models converged in the presented results as there are missing details. For example, Table 1 provides fine-tuning results of BERT like models on GLUE, which underperform the results reported by Devlin et al. 2018, and Liu et al. 2019. This may be due to the difference in pre-training corpus, or due to undercutting of the models. In the latter case, additional training may have changed the results.\n4. There are missing experiments to show how well does MECHANIC operate when the hyper-parameters of the base algorithm are not well tuned. If it fails in such cases, then one still has to tune the parameters of the base algorithm, and gain nothing by using this approach. \n5. there are missing ablation tests - what is the effect of $\\lambda$ and the effect of $\\beta$? Also, what is the effect of learning rate schedulers on the performance of MECHANIC?\n6. While the paper focuses on empirical findings, the authors suggest the method should be robust, but with no guarantees. For example, line 194-196 the authors claim that weight decay should not affect the operation of MECHANIC, but provide no proof (lie 195-196) and no empirical evidence. \n7. While the method is presented as parameter-free, there is evidence that tuning its hyper-parameters may indeed be required. For example, lines 175-176 discuss the importance of setting $\\beta$. While the authors try to obviate the need to tune it, the choice of defaults may not generalize well to other settings. Also, lines 239-240 mention that tuning $\\lambda$ may be beneficial. \n\n\nSome additional comments regarding the presentation of the paper:\n* Algorithm 2 is referred to quite a lot and is being analyzed in the paper. As such, I think it will be beneficial for the reader if it is included in the body of the paper.\n* In line 49, you state the \"Empirical studies often take the route of \"hyper-gradient descent\". However, this approach is rarely used in practice, and even in the work you cite, most empirical approaches do not follow it, nor compare to it.\n* The \"scale\" buy which the paper measure gains is not consistent. When MECHANIC is outperformed by base algorithms it is said to be \"quite competitive\", while when it has the advantage by smaller absolute gains, it is said to outperform. For example, see lines 247-248 which discuss the results of Figure 2. \n\n\nThere are a few typos and styling issues that should be fixed prior to final submission: \n1. line 38, there is a missing \"to\" between \"allowed\" and \"make\"\n2. line 93, the words \"let and\" were swapped\n3. line 113, **missing key information** - the sentence of how you set the scaling factor is missing \"and so by setting <MISSING> To show...\" \n4. In Table 2, there are at least 5 cases where the base algorithm achieved **identical** results to those of the MECHANIC tuning, while the table gives only the results of the MECHANIC tuning in boldface which is confusing and misleading.\n5. line 218, \"works\" $\\rightarrow$ \"work\"\n6. line 219, \"not that\" is swapped\n7. The bibliography style is inconsistent in its presentation. For example, [7,11,14,33] are all cited from CoLT, each with different styling. [20,21,36] are both from NeurIPS but with very different styles as well. Also 35 cites from arXiv differently from e.g. 38.\n8. All references in the paper are not \"clickable\". Namely, figures, table and citation references are not clickable in the PDF\n\nQUESTIONS:\n1. How were the hyper-parameters of the base algorithms set? How are the results change if the hyper-parameter of the base algorithm are under-optimal? Was there a thorough tuning phase to verify the chosen hyper-parameters (including the total number of update steps) of the baseline algorithm were indeed correctly tuned?\n2. What is the standard deviation in the given results?\n3. What is the effect of $\\lambda$ and the effect of $\\beta$ ? Also, what is the effect of learning rate schedulers on the performance of MECHANIC?\n\nLIMITATIONS:\nThe authors have addressed the limitation of their work adequately. Namely, the authors mention the lack of theoretical guarantees or intuition into some of the results, as well as potential future work. There is no ethical considerations relevant in this case.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a scheme to automatically tune the learning rate scale factor for any gradient-based optimization algorithm.  The method can be viewed as a practical realization of recent theorertical results in online convex optimization (OCO), which reduces the problem to minimizing the regret of a one-dimensional OCO algorithm. The authors test their proposal on a range of large-scale deep learning benchmarks, showing  competitive performance compared to strong manually tuned baselines. It is also shown that the method can outperform a recently proposed learning rate tuner, D-adaptation, without requiring the modifications accross base optimizers that D-adaptation does.\n\nSTRENGTHS:\n* Strong empirical evaluation, which shows solid results against strong baselines in large-scale contexts where learning rate tuning is known to play a criticial role. \n\n* Great potential  for practicaly applicability, as it provides an off-the-shelf  wrapper that can be used on top of any gradient-based optimization algorithm.\n\n* (At least part of) the approach is theoretically motivated -- and subject to theoretical analysis. \n\nGiven the above, I believe the project can be turned into a very strong submission -- provided the weaknesses below are addressed.\n\nWEAKNESSES:\nDespite the paper's great potential, a major reservation I have, which justifies my rating, is about the clarity of the presentation -- at least for those without strong expertise of the relevant literature. \n\n* The background section, which motivates the approach, is based  on prior work -- specifically, recent progress in parameter-free online convex optimization (such as ref [28]). I think the authors can do a much better job at reproducing and explaining these known results (the fact that presentation is plagued with misprints does not help).  \n\nFor example, while Theorem 1, which bounds the regret of Mechanic in terms of the regrets of Base and Tuner, is relatively clear, the exposition of how it is exploited (end of Section 2.1 and Section 2.2 ) is very unclear to me. To be specific, the right-hand-side in the last equality on line 97, s dot shows  up in both regrets, which suggests some sort of tradeoff between both regrets --  I do not follow the reasoning  that the problem is completely relegated to Tuner (for example, the optimal s dot  on line 101 from the BASE regret term looks distinct from the optimal s dot on line 119 from the tuner regret).  Furthermore,  the paragraph between l112-l122 completely lost me.\n\n I think this would be highly beneficial to rewamp entirely this exposition  for the sake of readability,  especially for non OCO experts like myself.(I formulate more specific comments / questions in Questions section.  \n\n* In Section 3 the specific form of Tuner (even  in its simplified form after line 148) seems to come out of the blue. \nI am not sure whether or not it was supposed to follow from the background analysis; if it does, I think being  explicit  about  the transition would be helpful (of course the empiricial justifications of some specific ingredients, as in 3.2 for weight decay, is completely fine).\n\nQUESTIONS:\nI am confused by the notation (line 41 or line 95) suggesting that the MECHANIC iterates are mere interpolations of the initial point and the BASE iterates.  It makes it look like one needs to run the base algorithm  once completely to be able to define the MECHANIC iterates, which is clearly not the case from Algorithm 1. I see how  MECHANIC depends on the BASE *updates* -- but these are built from  gradients that depend on previous MECHANIC iterates, not on the BASE iterates.  (The authors seem to acknlowedge this abuse of notation in Footnote 1, but I cannot see the mathematical justification that the footnote is referring to). \n\n\nI am not sure about the editorial choice that  consists of motivating the algorithm with the OCO analysis. \nI feel a better choice could be to first introduce the wrapper as an additional  optimization step over the scaling factor $s$ with loss $\\ell_t(x_1 - s \\sum_t \\mbox{base updates}_t)$, which I find intuitive and elegant in and of itself  -- as it optimizes in the direction of what would have been a good scale in hindsight  ; then to introduce OCO subsequently for the sake of the analysis.  \n\nMiscalleneous: \n\n* l38 allowed make <-- allowed to make \n\n* Footnote 1: presently <-- below or shortly\n\n* Inconsistent index for s_t between l41 and l42. \n\n* From l45, I suggest to just create a new \"related work\" section. An explicit list of contributions in the introduction would also be helpful. \n\n* l93 let and <-- and let \n\n* l103 approximates of the gradient <-- approximates the gradient\n\n* l110: isn't an example of such an advanced algorithm SGD initialized a s=0? (from l83). \n\n* l110: A TUNER subscript is missing in the regret formula.  \n\n\n* l113 missing words: \"so by setting To show\" ...\n\n\n* On Algorithm 1 : l6 gk <-- gt. I suggest adding What is zt? x_ref not updated? \n\n* Table 1: it would be nice to see standard deviations reported as well \n\n* The last mentioned limitation in section A looks like an advantage to me (no validation set required).\n\nLIMITATIONS:\nLimitations adequately acknowledged in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops a method to automatically determine the learning rate of an optimization algorithm. Their approach, referred to as Mechanic and motivated by theoretical advances in online convex optimization, determines a new learning rate for iteration t using an update that resembles an adagrad update plus an additional decay factor. Their method is tested in deep learning settings ranging from language to large scale vision, where it approaches and sometimes outperforms a \"tuned baseline\".\n\nSTRENGTHS:\nThis paper has many strengths:\n- The problem that the authors explore is very important to the community. Even if their method is not a universal solution, any sound progress towards this target is potentially valuable. If the community succeeds in removing the need for tuning learning rates, significant resources use will be eliminated.\n- Mechanic can be applied to any base algorithm, not only in theory but also in practice: the authors show results for Adam, Lion, and SGD.\n- Mechanic is tested in various different domains from masked language modeling to large scale JFT pre-training.\n- Mechanic has only two tunable hyper-paramters, s_init and the vector of beta.\n\nWEAKNESSES:\n- The central weakness in this paper is the lack of clarity surrounding the \"tuned baseline\". How was that baseline tuned? What scheduler is used? Is it the best learning rate and if so out of how many searched? It would be great to see a line plot of LR vs. perf for the baseline and then a horizontal line for mechanic. This weakness is why I've listed the soundness as only fair.\n- There are no ablation studies for the relevant hyperparameters s_init and the betas.\n- Minor: There are some claims in the paper which I do not believe to be true. For instance, in the conclusion the authors claim that it is infeasible to manually tune a scale factor for each layer. However, there is work, e.g.,  mu-transfer which aims to specify a \"corrected\" learning rate for each layer.\n\nQUESTIONS:\n- Different base algorithms and problem settings result in different performance for Mechanic vs. the baseline. Do the authors have any ideas why this may be?\n- How is performance affected by s_init and the betas.\n- I was slightly lost in the explanation for how multiple betas were used in the section where the adagrad like update transforms into more of an adam like update. Could the authors extend this trick so that multiple betas could be used simultaneously in other contexts, e.g., for the adam estimators for the params?\n- Can you explain the tuning that went into the baseline. Ideally a plot of LR vs. perf for the baseline could be contrasted with mechanic.\n- Why do the authors compare only to D-adapt and not, e.g., DoG https://arxiv.org/abs/2302.12022.\n\nLIMITATIONS:\nYes, the limitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-22"}