SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Momentum tracking The default EMA factor used to track the momentum in Lion is  $0.99(\beta_{2})$ , compared to the commonly used 0.9 in AdamW and momentum SGD. The current gradient and momentum are interpolated with a factor of  $0.9(\beta_{1})$  before the sign operation is applied. This choice of EMA factor and interpolation allows Lion to balance between remembering a  $\sim 10x$  longer history of the gradient in momentum and putting more weight on the current gradient in the update. The necessity of both  $\beta_{1}$  and  $\beta_{2}$  is further discussed in Appendix L.
Hyperparameter and batch size choices Lion is simpler and has fewer hyperparameters compared to AdamW and Adafactor as it does not require  $\epsilon$  and factorization-related ones. The update is an element-wise binary  $\pm 1$  if we omit the weight decay term, with larger norm than those produced by other optimizers like SGD and adaptive algorithms. As a result, Lion needs a smaller learning rate and in turn a larger decoupled weight decay to achieve a similar effective weight decay strength ( $1r * \lambda$ ). Detailed information on tuning Lion can be found in Appendix M. Additionally, the advantage of Lion over AdamW enlarges as the batch size increases, which fits the common practice of scaling up model training through data parallelism (Appendix L).
Memory and runtime benefits Lion only saves the momentum thus has smaller memory footprint than popular adaptive optimizers like AdamW, which is beneficial when training large models and / or using a large batch size. As an example, AdamW needs at least 16 TPU V4 chips to train a ViT-B/16 with image resolution 224 and batch size 4,096, while Lion only needs 8 (both with bfloat16 momentum). Another practical benefit is that Lion has faster runtime (steps / sec) in our experiments
due to its simplicity, usually  $2 - 15\%$  speedup compared to AdamW and Adafactor depending on the task, codebase, and hardware.
Relation to existing optimizers The sign operation has been explored in previous optimizers [7, 83]. The closest to ours is the handcrafted optimizer signSGD [7] (and its momentum variant) that also utilizes the sign operation to calculate the update but has a different momentum update rule from Lion. Their focus is to mitigate communication costs between agents in distributed training, and they observe inferior performance when training ConvNets on image classification tasks. On the other hand, NAdam [27] combines the updated first moment and the gradient to compute the update, but Lion decouples the momentum tracking and how it is applied to the update through  $\beta_{2}$ . A comparison of Lion with related optimizers can be found in Appendix K.
4 Evaluation of Lion
In this section, we present evaluations of Lion, on various benchmarks. We mainly compare it to AdamW (or Adafactor when memory is a bottleneck) as it is exceedingly popular and the de facto standard optimizer on a majority of learning tasks. The result of momentum SGD is only included for ResNet since it performs worse than AdamW elsewhere. We also benchmark other popular optimizers in Appendix K, including handcrafted and automatically discovered ones. We make sure that every optimizer is well-tuned for each task (see Appendix M for tuning details). By default, the learning rate schedule is cosine decay with 10K steps warmup, and the momentum is saved as bf10at16 to reduce the memory footprint. Ablations studies are performed in Appendix L.
We perform experiments including various datasets and architectures on the image classification task (see Appendix B for dataset details). Apart from training from scratch on ImageNet, we also pre-train on two larger well-established datasets, ImageNet-21K and JFT [89]. The image size is  $224^2$  by default otherwise specified by the subscript.
Train from scratch on ImageNet Following previous works [26, 37], we train ResNet-50 for 90 epochs with a batch size of 1,024, and other models for 300 epochs with a batch size of 4,096. As shown in Table 8 (in the Appendix), Lion significantly outperforms AdamW on various architectures. Empirically, the improvement is more substantial on models with larger capacity, with accuracy increases of  $1.96\%$  and  $0.58\%$  for ViT-B/16 and ViT-S/16, respectively. The performance gaps also tend to enlarge with fewer inductive biases. When strong augmentations are applied, the gain of Lion over AdamW shrinks, but it still outperforms AdamW by  $0.42\%$  on CoAtNet-3, despite the strong regularization during training [21].
Pre-train on ImageNet-21K We pre-train ViT-B/16 and ViT-L/16 on ImageNet-21K for 90 epochs with a batch size of 4,096. Table 8 shows that Lion surpasses AdamW when the training set is enlarged for  $10\mathrm{x}$ . The gaps on larger models are bigger, with  $+0.52\%$  vs.  $+0.33\%$  (ImageNet),  $+0.57\%$  vs.  $+0.23\%$  (RealL), and  $+0.74\%$  vs.  $+0.25\%$  (V2) for ViT-L/16 and ViT-B/16, respectively.
Pre-train on JFT To push the limit, we conduct extensive experiments on JFT. We follow the settings of Dosovitskiy et al. [26] and Zhai et al. [102] for both pre-training and fine-tuning. Figure 1 (Left) and 3 present the accuracy of three ViT models (ViT-B/16, ViT-L/16, and ViT-H/14) under different pre-training budgets on JFT-300M. Lion enables the ViT-L/16 to match the performance of ViT-H/14 trained by AdamW on ImageNet and ImageNet V2 but with 3x less pre-training cost. On ImageNet ReaL, the compute saving further becomes 5x. Another evidence is that even when a ViT-L/16 is trained by AdamW for 4M steps by Zhai et al. [102], its performance still lags behind the same model trained by Lion for 1M steps.
Table 5 in the Appendix shows the fine-tuning results, with higher resolution and EMA. Our ViT-L/16 matches the previous ViT-H/14 results trained by AdamW, while being  $2\mathrm{x}$  smaller. The advantage is larger on more challenging benchmarks, such as  $+1.33\%$  (V2),  $+6.08\%$  (A),  $+5.54\%$  (R) for ViT-L/16. When pretrained on JFT-3B, the ViT-g/14 trained by Lion outperforms the previous ViT-G/14 results [102], with  $1.8\mathrm{x}$  fewer parameters. Our ViT-G/14 achieves a  $90.71\%$  accuracy on ImageNet.
Figure 3: ImageNet Real (Left) and ImageNet V2 (Right) accuracy after we pre-train ViT models on JFT-300M then fine-tune on ImageNet. See Table 4 (in the Appendix) for the detailed numbers.
Table 2: Zero-shot accuracy of LiTs on ImageNet, CIFAR-100, and Oxford-IIIT Pet. As a reference, the zero-shot accuracy of CLIP [75] on ImageNet is  $76.2\%$ .
Model	Optimizer	ImageNet	C100	Pet
LiT-B/32-B	AdamW	68.78	71.41	86.62
Lion	69.88	71.78	87.36
LiT-B/16-B	AdamW	74.26	72.25	89.83
Lion	75.39	72.49	91.20
LiT-g/14288-L	AdamW	83.43	80.93	94.88
Lion	84.09	81.43	95.86
Table 3: One-shot evaluation averaged over three NLG and 21 NLU tasks. The results of GPT-3 [12] and PaLM [17] are included for reference. The LLMs trained by Lion have better in-context learning ability. See Table 11 (in the Appendix) for detailed results on all tasks.
Task	1.1B
Adafactor	Lion	2.1B
Adafactor	Lion	7.5B
Adafactor	Lion	6.7B
GPT-3	8B
PaLM
#Tokens	300B	300B	780B
Avg NLG	11.1	12.1	15.6	16.5	24.1	24.7	23.1	23.9
Avg NLU	53.2	53.9	56.8	57.4	61.3	61.7	58.5	59.4
4.2 Vision-Language Contrastive Learning
This section focuses on the vision-language contrastive training [75]. We compare Lion with AdamW (Adafactor) on zero-shot image classification and image-text retrieval benchmarks. We initialize the image encoder with a strong pre-trained model as it is suggested to be more efficient [103].
Locked-image text Tuning (LiT) We perform a comparison between Lion and AdamW on LiT [103] by training the text encoder [103] in a contrastive manner using the same frozen pre-trained ViT. All models are trained for 1B image-text pairs with a batch size of 16,384. Table 2 shows the zero-shot image classification results on three model scales, with the name specifies the size, e.g., LiT-B/16-B denotes a ViT-B/16 and a base size Transformer as the text encoder. Our method, Lion, demonstrates consistent improvement over AdamW with gains of  $+1.10\%$ ,  $+1.13\%$ , and  $+0.66\%$  on zero-shot ImageNet accuracy for LiT-B/32-B, LiT-B/16-B, and LiT-g/14 $_{288}$ -L, respectively. Figure 7 (Left) in the Appendix depicts an example zero-shot learning curve of LiT-B/16-B. Similar results are obtained on the other two datasets. The zero-shot image-text retrieval results on MSCOCO [56] and Flickr30K [74] can be found in Figure 6 (in the Appendix). The evaluation metric is Recall@K, calculated based on if the ground truth label of the query appears in the top-K retrieved examples. Lion outperforms AdamW on both datasets, with a larger gain in Recall@1 than Recall@10 on Flicker30K, implying more accurate retrieval results:  $+1.70\%$  vs.  $+0.60\%$  for image  $\rightarrow$  text and  $+2.14\%$  vs.  $+0.20\%$  for text  $\rightarrow$  image.
BASIC Pham et al. [72] propose to scale up batch size, dataset, and model size simultaneously, achieving drastic improvements over CLIP. It uses a sophisticated CoAtNet [21] pre-trained on JFT-5B as the image encoder. Furthermore, the contrastive training is performed on 6.6B image-text pairs with a larger 65,536 batch size. To push the limit, we only experiment on the largest BASIC-L, and use Lion on both image encoder pre-training and contrastive learning stages. As illustrated in Table 1, we achieve a significant  $2.6\%$  gain over the baseline, striking a  $88.3\%$  accuracy on zero-shot ImageNet classification. Note that this result is  $2.0\%$  higher than the previous best result [100]. The performance gain is consistent on five other robustness benchmarks. After fine-tuning the image encoder (CoAtNet-7) in BASIC-L obtained by Lion, we further achieve a  $91.1\%$  top-1 accuracy on ImageNet, which is  $0.1\%$  better than the previous SOTA.
4.3 Diffusion Model
Recently, diffusion models achieve a huge success on image generation [24, 40, 41, 84, 88], so we test the performance of Lion on unconditional image synthesis and multimodal text-to-image generation.
Figure 4: Imagen text-to-image  $64^{2}$  (Left) and the  $64^{2} \rightarrow 256^{2}$  diffusion models (Right).
Figure 5: Log perplexity on Wiki-40B (Left) and PG-19 (Right). The speedup brought by Lion tends to increase with the model scale. The largest model on Wiki-40B is omitted as we observe severe overfitting.
Image synthesis on ImageNet We utilize the improved U-Net architecture [24] and perform  $64 \times 64$ ,  $128 \times 128$ , and  $256 \times 256$  image generation on ImageNet. The batch size is 2,048 and the learning rate remains constant throughout training. For decoding, we apply DDPM [41] for 1K sampling steps without classifier-free guidance. The evaluation metric is the standard FID score. Illustrated by Figure 1 (Right) and Figure 7 (Middle and Right) in the Appendix, Lion enables both better quality and faster convergence on the FID score. The gap between Lion and AdamW increases with the image resolution, where the generation becomes more challenging. When generating  $256 \times 256$  images, Lion achieves the final performance of AdamW with  $2.3x$  fewer steps. The final FID scores are 4.1 (Lion) vs. 4.7 (AdamW). For reference, the FID of ADM [24] is 10.94.
Text-to-image generation We follow the Imagen [84] setup to train a base  $64 \times 64$  text-to-image model and a  $64 \times 64 \rightarrow 256 \times 256$  super-resolution model. All models are trained on a high-quality internal image-text dataset with a batch size of 2,048 and a constant learning rate. Due to computational constraints, our base U-Net has a width of 192 compared to 512 in the original 2B model, while the 600M super-resolution model is identical to the original Imagen setup. Along with the training, 2K images are sampled from the MSCOCO [56] validation set for real-time evaluation. We use the CLIP score to measure image-text alignment and the zero-shot FID-30K to measure image fidelity. Classifier-free guidance [40] with a weight of 5.0 is applied as it has been shown to improve image-text alignment. Figure 4 depicts the learning curve. While there is no clear improvement on the base  $64 \times 64$  model, Lion outperforms AdamW on the text-conditional super-resolution model. It achieves a higher CLIP score and has a less noisy FID metric compared to AdamW.
4.4 Language Modeling and Fine-tuning
This section focuses on language modeling and fine-tuning. On language-only tasks, we find that tuning  $\beta_{1}$  and  $\beta_{2}$  can improve the quality for both AdamW and Lion. See Appendix M for tuning details. The masked language modeling and fine-tuning results are shown in Appendix J.
Autoregressive language modeling We first experiment on two smaller-scale academic datasets Wiki-40B [34] and PG-19 [76]. The employed Transformer spans three scales: small (110M), medium (336M), and large (731M). The architecture details can be found in Appendix E. All models are trained with  $2^{18}$  tokens per batch for 125K steps, with a learning rate schedule of 10K steps warmup followed by linear decay. The context length is set to 512 for Wiki-40B and 1,024 for PG-19. Figure 5 illustrates the token-level perplexity for Wiki-40B and word-level perplexity for PG-19. Lion consistently achieves lower validation perplexity than AdamW. It achieves 1.6x and 1.5x speedup when training the medium size model on Wiki-40B and PG-19, respectively. When the model is increased to the large size, the speedup on PG-19 further increases to 2x.
We then conduct larger experiments with the pre-training dataset similar to GLaM [28]. Following GPT-3 [12], we train three models, from 1.1B to 7.5B parameters, on 300B tokens with a batch size of 3M tokens and a context length of 1K. We evaluate them on three natural language generative (NLG) and 21 natural language understanding (NLU) tasks (see Appendix C for details). We observe no difference in perplexity throughout training. Nevertheless, Lion outperforms Adafactor on the average in-context learning ability, as shown in Table 3. Our 7.5B baseline model, trained for 300B tokens, outperforms the 8B PaLM, trained for 780B tokens, demonstrating the strength of our setup. Lion outperforms Adafactor on both NLG and NLU tasks, particularly on the NLG tasks, with an exact match improvement of  $+1.0$ ,  $+0.9$ , and  $+0.6$  for the 1.1B, 2.1B, and 7.5B models, respectively.
Our work lies in the area of AutoML and meta-learning that includes learning to learn [1, 5, 63, 64, 78, 96, 97], neural architecture search [14, 15, 57, 71, 79, 87, 93, 94, 98, 106] and hyperparameter optimization [25, 43, 44, 55], etc. There is also a long history of using evolutionary methods to search for programs, i.e., genetic programming [11, 42, 49]. Our approach builds upon a symbolic search space similar to AutoML-Zero [70, 80]. However, instead of discovering programs with fixed dimensional matrices, vector, and scalars for toy tasks, our goal is to develop programs that operate on n-dimensional arrays and can generalize to state-of-the-art tasks. Other related works include numerous handcrafted optimizers [2, 7, 27, 29, 35, 48, 58, 61, 82, 83, 86, 105].
This paper aims at discovering optimization algorithms via program search. Despite the challenges from an infinite and sparse space, and large generalization gap between the proxy and target tasks, our method discovers a simple and effective optimizer, Lion, that is memory-efficient and achieves strong generalization across architectures, datasets and tasks.
We would like to thank (in alphabetical order) Angel Yu, Boqing Gong, Chen Cheng, Chitwan Sahara, Daiyi Peng, David So, Hanxiao Liu, Hanzhao Lin, Jeff Lund, Jiahui Yu, Jingru Xu, Julian Grady, Junyang Shen, Kevin Regan, Li Sheng, Liu Yang, Martin Wicke, Mingxing Tan, Mohammad Norouzi, Qiqi Yan, Rakesh Shivanna, Rohan Anil, Ruiqi Gao, Steve Li, Vlad Feinberg, Wenbo Zhang, William Chan, Xiao Wang, Xiaohua Zhai, Yaguang Li, Yang Li, Zhuoshu Li, Zihang Dai, Zirui Wang for helpful discussions, and the Google Brain team at large for providing a supportive research environment.
[1] Marcin Andrychowicz, Misha Denil, Sergio Gómez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
[2] Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning, 2020.
[3] Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 404-413. PMLR, 10-15 Jul 2018.
[4] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[5] Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V. Le. Neural optimizer search with reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 459-468. JMLR.org, 2017.
[6] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533-1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.
[104] Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension, 2018.
[105] Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18795-18806. Curran Associates, Inc., 2020.
[106] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations (ICLR), 2017.
Use unnumbered first level headings for the acknowledgments. All acknowledgments go at the end of the paper before the list of references. Moreover, you are required to declare funding (financial activities supporting the submitted work) and competing interests (related financial activities outside the submitted work). More information about this disclosure can be found at: https://neurips.cc/Conferences/2023/PaperInformation/FundingDisclosure.
Do not include this section in the anonymized submission, only in the final paper. You can use the ack environment provided in the style file to automatically hide this section in the anonymized submission.
Table 4: Model performance when pre-trained on JFT-300M then fine-tuned on ImageNet. Those numbers correspond to Figure 1 (Left) and Figure 3. The fine-tuning resolution is  $384^2$  for ViT-B/16 and ViT-L/16, and  $392^2$  for ViT-H/14. Following Dosovitskiy et al. [26], Polyak averaging is not applied here.
Model	#Params	Epochs / Steps	Optimizer	ImageNet	ReaL	V2	A	R
ViT-B/16384	86.86M	7 / 517,791	AdamW	84.24	89.04	74.89	27.39	53.71
Lion	84.72	89.14	75.83	29.65	55.86
ViT-L/16384	304.72M	7 / 517,791	AdamW	86.69	89.95	78.03	40.55	64.47
Lion	87.32	90.43	79.29	47.13	68.49
14 / 1,035,583	AdamW	87.29	90.11	78.91	42.56	64.34
Lion	88.09	90.62	80.48	51.55	70.72
ViT-H/14392	632.72M	14 / 1,035,583	AdamW	88.02	90.27	80.10	53.14	69.48
Lion	88.78	90.68	81.41	58.21	73.09
Table 5: Model performance when pre-trained on JFT then fine-tuned on ImageNet. Two giant ViT models are pre-trained on JFT-3B while smaller ones are pre-trained on JFT-300M. The ViT-G/14 results are directly from Zhai et al. [102].
Model	ViT-L/16512	ViT-H/14518	ViT-g/14518	ViT-G/14518
#Params	305.18M	633.47M	1.04B	1.88B
Optimizer	AdamW	Lion	AdamW	Lion	Adafactor	Lion	Adafactor	Lion
ImageNet	87.72	88.50	88.55	89.09	90.25	90.52	90.45	90.71 / 90.71*
ReaL	90.46	90.91	90.62	91.02	90.84	91.11	90.81	91.06 / 91.25*
V2	79.80	81.13	81.12	82.24	83.10	83.39	83.33	83.54 / 83.83*
A	52.72	58.80	60.64	63.78	-	-	-	-
R	66.95	72.49	72.30	75.07	-	-	-	-
* We observe overfitting in fine-tuning, therefore report both the last and oracle results.
A Pseudocode for AdamW and Lion
Algorithm 1 AdamW Optimizer
given  $\beta_{1},\beta_{2},\epsilon ,\lambda ,\eta ,f$
initialize  $\theta_0, m_0 \gets 0, v_0 \gets 0, t \gets 0$
while  $\theta_t$  not converged do
$t\gets t + 1$
$g_{t}\gets \nabla_{\theta}f(\theta_{t - 1})$
update EMA of  $g_{t}$  and  $g_{t}^{2}$
$m_{t}\gets \beta_{1}m_{t - 1} + (1 - \beta_{1})g_{t}$
$v_{t}\gets \beta_{2}v_{t - 1} + (1 - \beta_{2})g_{t}^{2}$
bias correction
$\hat{m}_t \gets m_t / (1 - \beta_1^t)$
$\hat{v}_t \gets v_t / (1 - \beta_2^t)$
update model parameters
$\theta_{t}\gets \theta_{t - 1} - \eta_{t}(\hat{m}_{t} / (\sqrt{\hat{v}_{t}} +\epsilon) + \lambda \theta_{t - 1})$
return  $\theta_{t}$
Algorithm 2 Lion Optimizer (ours)
given  $\beta_{1},\beta_{2},\lambda ,\eta ,f$
initialize  $\theta_0, m_0 \gets 0$
$c_{t}\gets \beta_{1}m_{t - 1} + (1 - \beta_{1})g_{t}$
$\theta_t \gets \theta_{t-1} - \eta_t(\mathrm{sign}(c_t) + \lambda \theta_{t-1})$
update EMA of  $g_{t}$
$m_{t}\gets \beta_{2}m_{t - 1} + (1 - \beta_{2})g_{t}$
B Image Classification Tasks
Our evaluation covers various benchmarks: ImageNet, ImageNet Real [8], ImageNet V2 [81], ImageNet A [39], ImageNet R [38], ImageNet Sketch [92], ObjectNet [4], CIFAR-100 [50], and Oxford-IIIT Pet [69].
Figure 6: Zero-shot image-text retrieval results on MSCOCO (Top) and Flickr30K (Bottom) for LiT-B/16-B. Recall@K is calculated based on if the ground truth label of the query appears in the top-K retrieved examples.
C NLP Tasks
This section shows all the natural language generation (NLG) and natural language understanding (NLU) tasks where we evaluate the large-scale language models in Section 4.4. Those tasks include Open-Domain Question Answering, Cloze and Completion Tasks, Winograd-Style Tasks, Common Sense Reasoning, In-Context Reading Comprehension, SuperGLUE, and Natural Language Inference.
- NLG: TriviaQA [45], Natural Questions [51], Web Questions [6].
- NLU: HellaSwag [101], StoryCloze [66], Winograd [53], Winogrande [85], RACE [52], PIQA [9], ARC [19], OpenbookQA [65], BoolQ [18], Copa [33], RTE [20], WiC [73], Multirc [47], WSC [53], ReCoRD [104], CB [22], Adversarial NLI [68].
Program 5: Algorithm with a better regularization. It dynamically calculates the dot product between the weight and gradient, before computing the weight decay.
$\mathfrak{m} =$  interp  $(\mathfrak{m},\mathfrak{g},0.16)$
$\mathbf{v} =$  interpolate(v，g2，0.001)
v753 = dot(g, w)
wd = v753 * w
update  $=$  sinupdate
$\mathrm{lr} = \mathrm{lr} * 0.0216$
$\mathbf{v} = \sin (\mathbf{v})$
Program 6: Algorithm that tracks the second moment without EMA decay, which is the same as AdaGrad.
m = interp(m, g, 0.1)
g2 = v + g2
$\mathbf{v} =$  interp(v, g2, 0.0015)
v70 = get(pi()
$\mathbf{v} = \min (\mathbf{v},\mathbf{v70})$
update  $=$  sinh(update)
lr = lr * 0.0606
Program 7: Algorithm uses the difference between gradient and momentum to track the second moment, resembling AdaBelief.
g = g - m
$\mathbf{v} =$  interp(v, g2, 0.001)
wd = w * 0.0238
$\mathbf{lr} = \mathbf{lr} * 0.03721$
D Other Discovered Programs
By varying the task setting, different types of algorithms can be discovered. For example, if we reduce the amount of data in the proxy task, we are more likely to discover algorithms with better regularization (Program 5), and if we reduce the search progress, we are likely to find simple variants of AdamW (Program 6 and 7). Future work can explore this potential to discover optimizers specialized for different tasks.