SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We include  $\beta$  to introduce some recency bias in the statistics recorded by MECHANIC, a common feature of practical optimizers. Mathematically, we accomplish this by up-weighting the  $t$ th feedback to TUNER by  $\beta^{-t}$ :  $h_t \to h_t\beta^{-t}$ . Thus, for example, we have  $\mathbf{v}_t = \sum_{k=1}^t h_k^2\beta^{-2kt}$  and  $r_t = -\sum_{k=1}^t h_k s_{k-1}\beta_s^{-k}$ . Using these weights directly results in numerical stability issues as the weights become exponentially large. Instead, since we only need to maintain the correct ratio  $\frac{W_t}{\sqrt{w_t}}$ , we can cancel a factor of  $\beta_s^{-t}$  from both sides, giving the update equations in Algorithm 2.
We found that tuning the value of  $\beta$  can significantly improve performance on different tasks. Thus, we incorporated multiple  $\beta$  values simultaneously in a way that obviates the need for such tuning.
Our approach is inspired by work on "combining" parameter free algorithms [34]. The idea is simple: parameter-free algorithms typically ensure  $R_{\mathrm{linear}}(0) \leq \epsilon$  for some constant  $\epsilon$  set by the user. So, if  $s_{t,1}, \ldots, s_{t,n}$  are the outputs of  $n$  parameter-free algorithms with regret bounds  $R_{\mathrm{linear}}^{1}(\mathring{s}), \ldots, R_{\mathrm{linear}}^{n}(\mathring{s})$ , we have for any  $j$ :
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} h _ {t} \left(\sum_ {i = 1} ^ {n} s _ {t, i} - \mathring {s}\right) = \sum_ {t = 1} ^ {T} h _ {t} \left(s _ {t, j} - \mathring {s}\right) + \sum_ {i \neq j} \sum_ {t = 1} ^ {T} h _ {t} \left(s _ {t, i} - 0\right) \\ = R _ {\text {l i n e a r}} ^ {j} (\mathring {s}) + \sum_ {i \neq j} R _ {\text {l i n e a r}} ^ {i} (0) \leq R _ {\text {l i n e a r}} ^ {j} (\mathring {s}) + (n - 1) \epsilon . \\ \end{array}
$$
So, with small constant additive overhead in the regret, the sum of all the outputs  $s_{t,1} + \dots + s_{t,n}$  achieves the same regret as the best of all the outputs. Motivated by this observation, we instantiate  $n = 6$  copies of TUNER with different  $\beta$  values and add their iterates to produce a final scaling.
3.2 Weight decay
Finally, we found that an addition of a peculiar weight-decay-esque term helped significantly on certain tasks, including vision tasks with smaller datasets, multi-objective NLP tasks and especially with reducing the variance in final results for all tasks. Specifically, rather than providing  $h_t = \langle \mathbf{g}_t, \boldsymbol{\Delta}_t \rangle$  as the input to the TUNER algorithm, we instead provide  $h_t = \langle \mathbf{g}_t + \frac{\lambda \| \mathbf{g}_t \| (\sum_{i=1}^n s_{t,i}) \mathbf{x}_t}{\|\mathbf{x}_t\|}, \boldsymbol{\Delta}_t \rangle$ . We conjecture that this term is helpful in the common case that the base algorithm itself is incorporating regularization or weight-decay.
This extra term is the derivative of the regularizer  $\mathbf{x} \mapsto \lambda \| \mathbf{g}_t\| \left(\sum_{i=1}^n s_{t,i}\right)\| \mathbf{x}\|$ . From a standard theoretical perspective, this regularization may seem overly large. However, it may not have as big an impact as one might imagine because the base algorithm does not see this regularization. Instead, the base algorithm may (or may not) perform weight decay using another method that MECHANIC has no insight into. That said, we do not propose an analytical explanation for this modification. We simply observed that in practice it performed well with a fixed  $\lambda = 0.01$ .
3.3 Runtime and Memory Cost
MECHANIC incurs little additional cost over that of BASE. In Algorithm 1, we denote  $d$ -dimensional vectors with bold font, and  $n$ -dimensional vectors and scalars with normal font (note that typically  $n = 6$ ). We use 1 additional  $O(d)$  memory slot, and four  $O(d)$ -time steps in lines 8, 9, 10 and 17. All other steps are  $O(1)$  or  $O(n)$  time and so have negligible cost.
Model	Size	Pre Opt	MLM	Optimizer	MNLI-m/mm	QNLI	SST-2	QQP
BERT-B	110M	AdamW	71.5	AdamW	84.3/84.8	91.0	92.4	90.1
M-AdamW	83.7/83.5	90.6	91.9	90.5
M-AdamW	71.7	AdamW	84.7/85.1	91.2	93.3	90.7
M-AdamW	84.5/84.4	91.3	92.5	91.1
BERT-B	110M	Lion	71.8	Lion	83.4/83.5	86.8	89.7	89.4
M-Lion	83.1/83.8	89.9	91.0	90.2
M-Lion	72.0	Lion	84.5/84.2	89.0	91.2	90.8
M-Lion	84.2/84.2	88.6	91.1	90.2
BERT-L	340M	AdamW	75.4	AdamW	86.2/86.4	92.2	93.9	91.3
M-AdamW	86.1/86.4	92.5	93.7	91.5
M-AdamW	75.3	AdamW	86.3/86.5	92.7	94.4	91.4
M-AdamW	86.1/86.3	91.7	93.5	91.5
BERT-L	340M	Lion	75.7	Lion	86.7/86.6	90.7	92.9	91.1
M-Lion	86.0/86.2	90.3	93.4	91.2
M-Lion	75.5	Lion	87.4/87.4	92.9	93.3	91.7
M-Lion	87.2/87.1	91.5	92.3	91.6
Table 1: Comparing MECHANIC on BERT. 5 largest datasets from GLUE. Results reported are peak validation scores averaged over 3 runs, both for the baseline and MECHANIC tuned models.
In this section we describe our experiments using MECHANIC to tune various base optimizers on different tasks. Note that almost all base optimizer implementations require a user-specified scale factor which is not directly visible to MECHANIC. We set this value to 1.0 before applying MECHANIC. Since MECHANIC multiplies the base update by  $s_t$ , setting the base scale factor to 1.0 allows us to interpret  $s_t$  as the "correct" value for the base scale.
4.1 Masked Language Modeling
We perform BERT pre-training on the Wikibooks dataset following the procedure from [35] with a few minor changes, most notably, we omit the Next Sentence Prediction (NSP) loss following [36]. Masked language modeling (MLM) requires reconstructing randomly masked tokens given an input sequence of tokens. As shown in Table 1, using MECHANIC leads to a noticeable improvement in MLM accuracy.
Varying batch size and model size: Past works observe that the scale factor  $s$  should decrease as either batch size is decreased or model size is increased [37, 38]. To inspect the scale factor that MECHANIC learns, we vary the batch size and model size while pre-training BERT using MECHANIC. As shown in Figure 1, in both cases, MECHANIC learns to decrease the scale factor  $s$  when decreasing the batch size and when increasing the model size.
Addition Ablations: Ablation studies on the effects of  $n$ ,  $\lambda$ ,  $s_{init}$  can be found in Appendix B.
Finetuning pre-trained models: In addition to pre-training, we evaluate our models on the 5 largest datasets from the GLUE suite [39]. One possible failure mode of MECHANIC tuned pre-trained models could have been that, even though they lead to high accuracy at pre-training time, transfer learning may fail at finetuning time.
To ensure that standard transfer learning pipelines still work with MECHANIC pre-trained models, we finetune them without a learning rate tuned using the AdamW optimizer and find that MECHANIC pre-trained models lead to higher accuracy at pre-training time, and they also outperform in finetuning more often than not. We finetune BERT-B (110M) and BERT-L (340M) models for at most 10 epochs on each of the GLUE datasets and report results on the GLUE dev set in Table 1.
Using MECHANIC for finetuning: We also investigated using MECHANIC for finetuning. Typically, to not erase the progress already made, a much lower base learning rate is employed at finetuning
(a) Varying batch size
(b) Varying model size
Figure 1: Scaling values  $s$  learned by MECHANIC while varying batch size and model size.
time. This could easily have been a potential failure mode of any kind of automatic learning rate tuner as such strategies might "explore" a high learning rate at the beginning of the optimization procedure. Fortunately, we observed that this inductive bias typically baked at finetuning time is still maintained when using MECHANIC.
4.2 Image Classification
In this Section, we present results on popular Image Classification tasks. Apart from training from scratch, we also perform transfer learning experiments where we pre-train on the JFT-300M [40] dataset and finetune on ImageNet, Cifar-10 and Cifar-100 datasets. We follow the exact setting employed in [41] for both pre-training and finetuning.
As shown in Table 2, MECHANIC is quite competitive across the board and produces results either very close to the baseline or better. Since MECHANIC optimizes for the train loss, in general, we observe that it results in better test performance on tasks with large amounts of data where the model is unable to overfit to the train set. For instance, we see that MECHANIC beats the baseline substantially when pre-training ViT models on JFT-300M, whereas it lags slightly behind on smaller datasets like ImageNet-1k or CIFAR-10/100. Even though we fix  $\lambda$  to 0.01 as default for all our reported experiments, we find that for small datasets like CIFAR-10, increasing it led to better test performance.
4.3 Comparison with D-adaptation
Recently, [16] introduced the D-adaptation algorithm, with the same goal of learning the correct scale  $s$  for SGD and Adam base optimizers. D-adaptation showed impressive empirical results on a range of popular deep learning tasks, so we compare MECHANIC with D-adaptation on a selection of tasks that D-adaptation worked well on, using code provided by the authors. Hyper-parameter settings were kept the same to ensure a fair comparison. In contrast to D-adaptation, MECHANIC does not require modification for different base optimizers and, as shown in Figure 2, it remains quite
Model	Size	Pre Opt	Pre Acc	Optimizer	I1K	Cifar100	Cifar10
CNN from scratch on CIFAR datasets
ResNet-18	11M	-	-	Mom	-	77.6	95.4
-	-	M-Mom	-	75.3	94.1
-	-	M-Mom (λ = 0.1)	-	76.6	95.3
WRN-40-10	56M	-	-	Mom	-	79.9	-
-	-	M-Mom	-	79.6	-
Pre-train on JFT-300M
ViT-B/16	86M	AdamW	48.5	Mom	84.7	91.9	99.1
M-Mom	84.7	90.7	99.1
M-AdamW	49.9	Mom	84.2	91.5	99.1
M-Mom	84.1	90.3	99.1
ViT-B/16	86M	Lion	47.0	Mom	85.3	92.1	99.2
M-Mom	85.2	91.0	99.2
M-Lion	49.6	Mom	84.7	92.3	99.2
M-Mom	84.6	90.9	99.1
ViT-L/16	307M	AdamW	54.4	Mom	86.7	93.9	99.5
M-Mom	86.6	92.7	99.5
M-AdamW	54.4	Mom	86.3	93.4	99.4
M-Mom	86.0	92.0	99.3
ViT-L/16	307M	Lion	52.0	Mom	86.7	93.8	99.4
M-Mom	86.7	93.0	99.4
M-Lion	55.4	Mom	87.2	94.0	99.4
M-Mom	87.2	93.4	99.4
Table 2: Comparing MECHANIC on vision models. All fine-tuning results are averaged over 3 independent runs with different seeds.
competitive on small datasets like CIFAR-10/100 while outperforming both a manually tuned baseline and D-adaptation on bigger tasks like IWSLT14 and language modeling on BookWiki dataset. We present additional results in Appendix C.3, including a comparison on a suite of 12 logistic regression problems.
MECHANIC is a new technique for scaling the updates of any base optimization algorithm. Our approach provides a practical implementation of recent developments in optimization theory, and is able to match the performance of tuned baselines on large-scale machine learning tasks. This work suggests several natural future directions. First, is there a theoretical motivation for our weight-decay term? Next, is it possible to leverage similar techniques to learn a per-layer scale factor? Such a capacity would not significantly increase computation cost, but by allowing more degrees of freedom may yield a method that significantly outperforms baselines since it is infeasible to manually tune a scale factor for every layer.
Ashok Cutkosky acknowledges funding support from NSF grant CCF-2211718, an Amazon research award, and Google.
[1] Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341-2368, 2013.
Figure 2: Comparing MECHANIC with D-adaptation and Adam or SGD with manually tuned learning rates on vision and language tasks.
[2] Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. arXiv preprint arXiv:1912.02365, 2019.
[3] J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. In Conference on Learning Theory (COLT), pages 257-269, 2010.
[4] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.
[5] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
[6] Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, et al. Symbolic discovery of optimization algorithms. arXiv preprint arXiv:2302.06675, 2023.
[7] H. Brendan McMahan and Matthew Streeter. Adaptive bound optimization for online convex optimization. In Proceedings of the 23rd Annual Conference on Learning Theory (COLT), pages 244-256, 2010.
[8] Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. Journal of machine learning research, 18, 2018.
[9] Kartik Chandra, Audrey Xie, Jonathan Ragan-Kelley, and Erik Meijer. Gradient descent: The ultimate optimizer. Advances in Neural Information Processing Systems, 35:8214-8225, 2022.
[10] Francesco Orabona and David Pál. Coin betting and parameter-free online learning. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 577-585. Curran Associates, Inc., 2016.
[11] Ashok Cutkosky and Kwabena Boahen. Online learning without prior information. In Conference on Learning Theory, pages 643-677, 2017.
[12] Zakaria Mhammedi and Wouter M Koolen. Lipschitz and comparator-norm adaptivity in online learning. Conference on Learning Theory, pages 2858-2887, 2020.
[13] Kfir Levy, Ali Kavis, and Volkan Cevher. Storm+: Fully adaptive sgd with recursive momentum for nonconvex optimization. Advances in Neural Information Processing Systems, 34: 20571-20582, 2021.
[14] Yair Carmon and Oliver Hinder. Making sgd parameter-free. Conference on Learning Theory, 2022.
[15] Maor Ivgi, Oliver Hinder, and Yair Carmon. Dog is sgd's best friend: A parameter-free dynamic step size schedule. arXiv preprint arXiv:2302.12022, 2023.
[16] Aaron Defazio and Konstantin Mishchenko. Learning-rate-free learning by d-adaptation. arXiv preprint arXiv:2301.07733, 2023.
[17] Naman Agarwal, Rohan Anil, Elad Hazan, Tomer Koren, and Cyril Zhang. Disentangling adaptive gradient methods from learning rates. arXiv preprint arXiv:2002.11803, 2020.
[18] Zhou Lu, Wenhan Xia, Sanjeev Arora, and Elad Hazan. Adaptive gradient methods with local guarantees. arXiv preprint arXiv:2203.01400, 2022.
[19] Xinyi Chen and Elad Hazan. A nonstochastic control approach to optimization. arXiv preprint arXiv:2301.07902, 2023.
[20] Francesco Orabona and Tatiana Tommasi. Training deep networks without learning rates through coin betting. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 2157-2167, 2017. URL http://papers.nips.cc/paper/6811-training-deep-networks-without-learning-rates-through-coin-betting.
[21] Ashok Cutkosky and Kwabena A Boahen. Online convex optimization with unconstrained domains and losses. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 748-756. Curran Associates, Inc., 2016.
[22] Shai Shalev-Shwartz. Online learning and online convex optimization. Foundations and Trends in Machine Learning, 4(2):107-194, 2011.
[23] Elad Hazan. Introduction to online convex optimization. arXiv preprint arXiv:1909.05207, 2019.
[24] Francesco Orabona. A modern introduction to online learning. arXiv preprint arXiv:1912.13213, 2019.
[25] Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pages 928–936, 2003.
[26] S. Shalev-Shwartz. Online Learning: Theory, Algorithms, and Applications. PhD thesis, The Hebrew University of Jerusalem, 2007.
[27] Jacob Abernethy, Peter L Bartlett, Alexander Rakhlin, and Ambuj Tewari. Optimal strategies and minimax lower bounds for online convex games. In Proceedings of the nineteenth annual conference on computational learning theory, pages 415-424, 2008.
[28] Ashok Cutkosky and Francesco Orabona. Black-box reductions for parameter-free online learning in Banach spaces. In Conference On Learning Theory, pages 1493-1529, 2018.
[29] Brendan Mcmahan and Matthew Streeter. No-regret algorithms for unconstrained online convex optimization. In Advances in neural information processing systems, pages 2402-2410, 2012.
[30] Brendan McMahan and Jacob Abernethy. Minimax optimal algorithms for unconstrained linear optimization. In Advances in Neural Information Processing Systems, pages 2724-2732, 2013.
In this paper, we introduce a technique for automatically learning the right scale of the learning rate called MECHANIC and evaluate it on a broad range of practical deep learning problems and settings. We find that, depending on the problem, MECHANIC can be quite effective and even surpass performance of manual tuning of learning rates at a fraction of the cost. We also find that in addition to training from scratch, MECHANIC also works for finetuning.
While the initial set of results are encouraging, many challenges remain. Firstly, we found that MECHANIC does not seem to work well with dropout [42]. While MECHANIC is effective against noise from sampling, we believe there may be a more fundamental reason why dropout does not work well with MECHANIC. Second, MECHANIC re-purposes the gradient coming from the train set for learning the learning rate, which means it optimizes for train loss. This is different from manual tuning of learning rates where researchers tune it based on performance on the validation set. A principled way to handle this discrepancy is also in interesting avenue of future research.
B Additional Ablations
B.1 Setting BASE learning rate with Mechanic
Even though we recommend setting peak learning rate of Base optimizer to 1.0, to make sure that Mechanic truly is insensitive to the peak learning rate and robust to the choice of this hyperparameter, we conduct a study where we vary peak learning rate of AdamW on BERT-B masked language modeling task. As shown in Table 3, Mechanic is largely robust against choice of peak LR set for AdamW on BERT-B MLM task.
Peak LR of BASE with Mechanic	MLM Acc
1e-2	71.7
1e-1	71.6
1e0	71.6
1e1	71.4
1e2	71.6
B.2 Robustness to  $s_{init}$
Table 3: MECHANIC is largely robust against choice of peak LR set for AdamW on BERT-B MLM task.
sinit	1e-8	1e-7	1e-6	1e-5	1e-4
Accuracy	49.8	49.8	49.9	49.7	49.6
B.3 Ablation of  $\lambda$
Table 4: Accuracy on JFT-300M using model ViT-B/16 and optimizer  $\mathcal{M}$  -AdamW as a function  $s_{init}$ . MECHANIC is robust to varying this parameter.
λ	0	1e-3	1e-2	1e-1	1e0
Accuracy	49.7	49.8	49.9	49.7	Diverged
Table 5: Accuracy on JFT-300M using model ViT-B/16 and optimizer  $\mathcal{M}$  -AdamW as a function  $\lambda$  We have observed that while  $\lambda$  is helpful in stabilizing MECHANIC on some problems, as long as it is set to a reasonable small value it does not affect performance by a lot.
B.4 Number of  $\beta$  values
Number of β values (n)	2	4	6	8
Accuracy	48.9	49.5	49.9	49.6
C.1 Hyperparams for BERT
Table 6: Accuracy on JFT-300M using model ViT-B/16 and optimizer  $\mathcal{M}$ -AdamW as a function  $n$ , the number of  $\beta$  values. The  $\beta$  values are set as  $1 - 0.1^i$  for  $i$  from 1 to  $n$ . This is the most sensitive parameter in MECHANIC. For smaller  $n$  we see some significant performance degradation, while for larger  $n$  we see milder degradation. Theory suggests that larger  $n$  should result in a degradation that is roughly logarithmic in  $n$ .
Model	Aug	Optimizer	β1	β2	lr sweep	best lr	Weight decay
BERT-B	AdamW	0.9	0.999	[5e-4, 1e-3, 2e-3, 5e-3, 1e-2]	5e-3
BERT-L	AdamW	0.9	0.999	[5e-4, 1e-3, 2e-3, 5e-3, 1e-2]	1e-3
BERT-B/L	M-AdamW	0.9	0.999	0.01
BERT-B	Lion	0.9	0.99	[5e-5, 1e-4, 2e-4, 5e-4, 1e-3]	5e-4
BERT-L	Lion	0.9	0.99	[5e-5, 1e-4, 2e-4, 5e-4, 1e-3]	2e-4
BERT-B/L	M-Lion	0.9	0.99	0.1
Table 7: Critical hyperparameters we used for BERT pre-training. For the baselines we grid searched the learning rate as shown in the table. Batch size  $2\mathrm{k}$ , trained for  $150\mathrm{k}$  steps on original Wikibooks dataset w/o NSP loss (similar to Roberta). We found that a small amount of weight decay makes MECHANIC slightly more effective.
Model	AdamW LR	MECHANIC-AdamW
BERT-B	71.1	71.5	71.5	71.5	71.3	71.7
BERT-L	75.0	75.4	75.4	74.6	74.4	75.3
Lion LR	MECHANIC-Lion
BERT-B	70.8	70.8	71.1	71.8	71.4	72.0
BERT-L	75.1	75.6	75.7	74.7	Diverged	75.5
Table 8: As detailed in Table 7, we performed a grid search of learning rate values for each base optimizer. Here, we present the resulting accuracy values.
Hyperparam	Optimizer	Without MECHANIC pre-training	With MECHANIC pre-training
Learning Rate	Adam	[5e-5, 1e-4, 2e-4, 3e-4, 5e-4]	[1e-5, 2e-5, 3e-5, 5e-5, 1e-4]
Learning Rate	Lion	[5e-6, 1e-5, 2e-5, 3e-5, 5e-5]	[1e-6, 2e-6, 3e-6, 5e-6, 1e-5]
Batch Size	[16, 32]	[16, 32]
Weight Decay	0	0
Max Epochs	10	10
Learning Rate Decay	Linear	Linear
Warmup Ratio	0.06	0.06
Dropout	0.1	0.1
Attention Dropout	0.1	0.1
Table 9: BERT GLUE finetuning hparams with AdamW.
Hyperparam	Value
Batch Size	[16, 32]
Weight Decay	0
Max Epochs	10
Learning Rate Decay	Linear
Warmup Ratio	0.06
Dropout	0.0
Attention Dropout	0.1
C.2 Hyperparams for Image Classification
Table 10: BERT GLUE finetuning hparams when using mechanic at finetuning time. We found a limitations of MECHANIC that it does not perform well in combination to dropout, so we set dropout rate to 0 for these experiments.
Model	Aug	Optimizer	β1	β2	lr	Weight decay	Num epochs
ViT-B/16	AdamW	0.9	0.999	8e-4	0.1	7
ViT-B/16	M-AdamW	0.9	0.999	0.1	7
ViT-L/16	AdamW	0.9	0.999	4e-4	0.1	7
ViT-L/16	M-AdamW	0.9	0.999	0.1	7
ViT-B/16	Lion	0.9	0.99	1e-4	0.3	7
ViT-B/16	M-AdamW	0.9	0.99	0.3	7
ViT-L/16	Lion	0.9	0.99	1e-4	0.3	7
ViT-L/16	M-AdamW	0.9	0.99	0.3	7
Table 11: Critical hyperparameters we used for all the experiments, most of them directly repurposed from [41]. For each baseline we repurposed a well-tuned base learning from previous work [41, 43]. Trained on JFT-300M with batch size 4k with LR cosine decay schedule.
Figure 3: Scaling values  $s$  learned by MECHANIC during ResNet18 training on CIFAR10 and WideResNet training on CIFAR100. Shaded area represents max/min value over 3 runs. Dark line is average.
Hyperparam	ImageNet	Cifar100	Cifar10
Learning rate sweep	{0.003, 0.01, 00.03, 0.06}	{0.001, 0.003, 0.01, 00.03}	{0.001, 0.003, 0.01, 00.03}
Batch size	512	512	512
Weight decay	0	0	0
Num steps	20k	10k	10k
Warmup steps	500	500	500
Learning rate decay	Cosine	Cosine	Cosine
Dropout	0.0	0.0	0.0
Clipping norm	1.0	1.0	1.0
Table 12: We directly use ViT finetuning hyperparams recommended by [41]. For MECHANIC we also use same hyperparameters, omitting just the learning rate sweep, since we don't need it now. We use finetuning resolution of 384.
Hyperparam	Value
weight decay	[0.001, 0.0005, 0.0001]
lr	[0.3, 0.1, 0.03]
SGD Momentum β	0.9
batch size	128
num epochs	200
schedule	Step decay by 0.2 at 60, 120, 160 epochs
augmentations	Random Crop, Random Horizontal Flip
Gradient clip by global norm	1.0
λ	Kept at default 0.01
Table 13: Hyperparameters for tuning ResNet18 on CIFAR10 and WideResNet on CIFAR100
C.3 Hyperparams and additional results on comparisons with D-adaptation
Here we provide the theoretically tractable version of TUNER as well as its analysis.
D.1 Algorithmic Simplifications
To simplify the implementation of MECHANIC, we replaced all of the red text in Algorithm 2 with a single line  $r_t \gets \max(0, r_t)$  right after the definition of  $r_t$ . This is motivated by two ideas.
Figure 4: Comparing MECHANIC with D-adaptation and manually tuned learning rates on a suite of convex tasks.
Figure 5: Complementary train set results of MECHANIC with D-adaptation and manually tuned learning rates on vision and language tasks.