Symbolic Learning to Optimize: Towards Interpretability and Scalability
Authors: Wenqing Zheng
Tianlong Chen
Ting-Kuei Hu
Zhangyang Wang
Venue: ICLR 2022 Poster (2022)
Keywords: Symbolic Regression
Learning To Optimize
Interpretability

Abstract: Recent studies on Learning to Optimize (L2O) suggest a promising path to automating and accelerating the optimization procedure for complicated tasks. Existing L2O models parameterize optimization rules by neural networks, and learn those numerical rules via meta-training. However, they face two common pitfalls: (1) scalability: the numerical rules represented by neural networks create extra memory overhead for applying L2O models, and limits their applicability to optimizing larger tasks; (2) interpretability: it is unclear what each L2O model has learned in its black-box optimization rule, nor is it straightforward to compare different L2O models in an explainable way. To avoid both pitfalls, this paper proves the concept that we can "kill two birds by one stone", by introducing the powerful tool of symbolic regression to L2O. In this paper, we establish a holistic symbolic representation and analysis framework for L2O, which yields a series of insights for learnable optimizers. Leveraging our findings, we further propose a lightweight L2O model that can be meta-trained on large-scale problems and outperformed human-designed and tuned optimizers. Our work is set to supply a brand-new perspective to L2O research. Codes are available at: https://github.com/VITA-Group/Symbolic-Learning-To-Optimize.

Published as a conference paper at ICLR 2022
SYMBOLIC LEARNING TO OPTIMIZE: TOWARDS IN- TERPRETABILITY AND SCALABILITY
Wenqing Zheng', Tianlong Chen', Ting-Kuei Hu', Zhangyang Wang'
'Te University of Texas at Austin, Texas A&M University {w- zheng, tianlong chen, at wut.aasseutexs...o..... edu, tkhu@tamu.edo
ABSTRACT
Recent studies on Learning to Optimize (L20) suggest a promising path to automat- ing and accelerating the optimization procedure for complicated tasks. Existing L,20 models parameterize optimization rules by neural networks, and learn those numerical rules via meta-training. However, they face two common pitfalls: (1) scalability: the numerical rules represented by neural networks create extra mem- ory overhead for applying L20 models, and limit their applicability to optimizing larger tasks; (2) interpretability: it is unclear what an L20 model has learned in its black-box optimization rule, nor is it straightforward to compare different L20 models in an explainable way. To avoid both pitfalls, this paper proves the concept that we can "kill two birds by one stone", by introducing the powerful tool of symbolic regression to L20. In this paper, establish holistic sym- bolic representation and analysis framework for L20, which yields a series insights for learnable optimizers. Leveraging our findings, further propose a lightweight L,20 model that can meta-trained on large-scale problems and outperformed human-designed and tuned optimizers. Our work is set to sup- ply a brand-new perspective to L20 research. Codes are available at: https //github. www.www............ http://ITA--roo///yyaaaaa Symbo ic- Lee-Lerrning To- Tott-TT---piiiii Optimitiiiiie
INTRODUCTION
Efficient and scalable optimization algorithms (a.k.a., optimizers) are the cornerstone almost all computational fields. In many practical applications, will repeatedly perform certain type of optimization tasks over a specific distribution of data. Each time, the inputs that define the optimization problem new but similar to previous related tasks. We say such application has a narrow task distribution. Conventional optimizers are theory-driven, so they obtain performance guarantees over the classes of problems specified by the theory. Instead of manually designing and tuning task-specific optimizers, one may be naturally interested to pursue more general-purpose and data-driven optimization frameworks.
Learning optimize (I.20) (Almeida al. 2021 Andrychowicz et al. 2016 Chen al. 2017
Li& Malik 2017a) is an emerging paradigm that automatically develops an optimizer by learning set of optimization tasks. This data-driven procedure generates optimizers
from its performance that efficiently solve problems similar to those in the training tasks. Although often lacking a solid theoretical basis, the training process shapes the learned optimizer according to problems of interest. When the task distribution is concentrated, the learned optimizer can specifically fit the tasks by discovering "shortcuts" that classic optimizers do not take (Li Malik 2017a b). I.20 has demonstrated many practical benefits including faster convergence and/or better solution quality
(Chn et al. 2021a), even saving clock time (Li ct al. 2020b) and hardware energy [Li ct al. 2020a).
With promising progress being made, the questions persist: how far L20 from becoming principled science? To what extent can we trust L20 as being mature for real applications? As fast-growing new field, fascinating open challenges remain concerning L20, among which two particular barriers will be focused on in this work:
Interpretability. The learned optimizers are often hard to interpret. Traditional algorithm design follows analytic principles and admits "white box" symbolic forms, which clear

---

Published as a conference paper at ICLR 2022
to explain, analyze, troubleshoot, and prove. In contrast, existing L20 approaches fit update models using sophisticated data-driven predictors such as neural networks. As a result, is unclear what rules have been learned in those "black boxes", prohibiting their further explainable analysis and comparison.
Scalability. The learned optimizers can be hard to scale. L20 methods use learnable predictors fit update rules, which incur extra overhead in memory that grows proportionally with the amount of optimization task variables. Moreover, the training of a learned optimizer often involves backpropagating through full unrolled sequence for the parameter updates in the optimization tasks, and the memory cost grows fast with the unroll length. The scale of that growth results in complexities that current L20 methods cannot handle.
Our proposed solution, centered at addressing the two research gaps, is a brand-new symbolic representation and interpretation framework for L20, While benefiting from large modeling capacity, L20 update rules fit by numerical predictors are inexplainable and memory-heavy. In contrast, the symbolic representations of traditional optimizers enjoy low memory overhead and clear explainability. We aim to integrate the strengths of symbolic forms into L20. We propose to convert a numerical L20 predictor into symbolic form that preserves the same problem-specific performance and is still tunable. Those symbolic rules are light, scalable, and interpretable. They can even be made tunable and subject further meta testing. by lightweight trainable re-parameterization. Based on them, we will further develop a set of L20 interpretability metrics.
Our main technical innovations are summarized below:
We build the first symbolic learning to optimize (symbolic L20) framework that is generally applicable to existing L20 approaches. That includes defining the optimizer symbolic space and the fitting approach to convert numeric L20 to its symbolic counterpart.
Under the symbolic 1.20 framework, we establish a unified scheme for 1.20 interpretability. Our synergistic analysis in the symbolic domain uncovers a number of new insights that are otherwise difficult to observe, including what "shortcuts" L20 methods essentially learn and how they differs across various methods.
The lightweight symbolic representation also allows L20 to scale better, and introduce a lightweight trainable re-parameterization to enable further fine-tuning of symbolic rules. For the first time, we obtain a symbolic optimizer that can be meta-trained on ResNet-50 level (23.5 million parameters) optimizee within 30 GB GPU memory, and achieved the state-of-the-a of-ttheart (SOTA) performance when applied to training larger (ResNet- 1522 58.2 million parameters) or very different deep models (MobileNet v2 (Sandler et al.) (018)) on various datasets, outperforming existing manually designed and tuned optimizers.
2 RELATED WORKS
L20 lies at the intersection of ML and optimization research. It overlaps with meta-learning (Vilalta& & Drissi 2002 Hospedales et al., 202T), which contributes a significant portion of L200 development (Andrychowicz
al. [0166] Li & Malik 2017a). L20 captures two main aspects of meta learning: rapid learning within each task, and transferable learning across many similar tasks. Using data-driven predictors to fit update rules, recent L20 approaches have shown success for a wide variety of problems, from convex optimization (Gregor & LeCun 2010), nonconvex optimization (Andrychowicz et al., (016)) and minmax optimization (Shen et al. 2222), 1). to black-box optimization (Chnn et al.) 2017) and combinatorial optimization (Khalil et al. 2017). Application areas benefiting the most from L20 methods include signal processing and communication (Borgerding et al. 2017
Balatsoukas- Stiiaasnn-tiinnggnaaa & Studer, 2019; You et al. 2020), image processing (Zhang & Ghanem 2018
Corbineau et al. 2019 Chen et al. 2020b). medical imaging (Liang et al. 202020200 Yin et al. 2021 and
computational biology (Cao al.
2019 Chen et al. 2019).
The pioneering L20 work (Andrychowicz et al. 2016) leverages long short-term memory (LSTM)
as the coordinate-wine predictor, which is fed with the optimizee gradients and outputs the optimizee parameter updates. Chen et al. (2017) takes the optimizee's objective value history the input state of a reinforcement learning agent, which outputs the updates as actions. To enhance L20 generalization. (Lv et al. 2017) proposes random scaling and convex function regularizers tricks. (Wichrowska et al.
2

---

Published as a conference paper at ICLR 2022
2017)
introduces hierarchical RNN capture the relationship across the optimizee parameters and trains it via meta-learning on the ensemble of small representative problems.
Training -STM-based L.20 models is however notoriously difficult, that is rooted in the LSTTMsstyl
unrolling during L20 meta-training (Tallec & Ollivier [01 Metz et al. [0118] Pascanu et al. 2013
Parmas et al. 2019). A practical optimizer may need to take thousands or more of iterations. However,
naively unrolling LSTM to this full length is impractical for both memory cost and trainability by
gradient propagation. Most LSTM- based L20 methods (Andrychowicz et al. (2016) Chen et al.) [011)]
al. 2017 Wichrowska al. 2017 Metz et al.] 2018: Cao al.) 2019) hence take advantage
of truncating the unrolled optimization, yet at the price of the so-called "truncation bias" (L.v et al.
2017) that hampers their learned optimizer's generalization unseen tasks (Chen al. 2020a).
Most L20 approaches are "black boxes" with neither theoretical backup nor clear explanation on what they learn. A handful of efforts have been made on "demystifying" L20, mainly about linking or reducing their behaviors those of some analytic, better understood optimization algorithms. Maheswaranathan et al. (2020) analyzes and visualizes the learned optimizers, to discover that they have learned canonical mechanisms including momentum, gradient clipping, learning rate schedules, etc. Another well-known success was on interpreting the learned sparse coding algorithm (Greoor & LeCun 2010). as related specific matrix factorization of the dictionary's Gram matrix (Moreau & Bruna 2017). to the projected gradient descent trade-off between convergence speed and reconstruction accuracy (Grryes et al.) 2018), or simply reduced to the original iterative algorithm with data-driven hyperparameter estimation (Chen et al. 2018 Liu et al. 2019: Chen et al. [b) Other examples include analyzing the learned alternating minimization on graph recovery (Shrivastava et al. 2020), and analogy between deep-unfolded gradient descent and Chebyshev step-sizes (Takabe & Wadayama 2020). However, they are all restricted to some specific objective or algorithm type.
3 TECHNICAL APPROACH
3.1 NOTATION AND PRELIMINARIES
Without loss of generality, let us consider an optimization problem ming f(x) where x E Rd. Here, f (x) is termed the optimization problem or optimizee, is the variable to be optimized, and the algorithm to solve this problem is termed the optimizer. A classic optimizer usually iteratively updates based handcrafted rule. For example, the first-order gradient descent algorithm takes an update at iteration based on the local gradient at the instantaneous point X, X++1 = X a (x+). where a is the step size. As a new learnable optimizer, L20 has much greater room to find flexible update rules. We define the input of L20 as z, In general, Ze contains the optimization historical information available at time t, such as the current/past iterates x, X2, and/or their gradients Vf(z)).... fxx), etc. L20 models an update rule by predictor function g of Z₁ X = X; gZ, Φ), where the mapping of y parameterized by Φ. Finding an optimal update rule can be formulated mathematically as searching for good & over the parameter space of Practically, is often neural network (NN). Since NNs are universal approximators (Hornik et al. 1989). 1.20 has the potential discover completely new update rules without relying on existing rules.
In order to find a desired @ associated with fast optimizer, (Andrychowicz et al. minimize the weighted sum of the objective function fxx) over a time span T:
2016 proposed to
T
Σ we wxx(xx))
min BSET
with X++1 = X: g(zez( Φ), t 0, O,..,,T-
(1)
where Wop WT-1 are the weights whose choices depend on empirical settings. also called the unrolling length. represents sample optimization task from ensemble T that represent the target task distribution. Note that & determines the objective value through determining the iterates x,. 1.20 solves the problem [T) for desirable á and correspondingly the update rule g(2,, Ø).
A typical L20 workflow is divided into two stages (Chen et al. 202 meta-training stage that learns the optimizer with a set of similar optimization tasks from the task distribution; and a meta- testing stage that applies the leamned optimizer to new unseen optimization tasks. The meta-training process often occurs offline and is time consuming. However, the online application of the method at meta-testing is (aimed to be) time saving.
3

---

Published as a conference paper at ICLR 2022
3.2 MOTIVATION AND CHALLENGES
If we take a unified mathematical perspective, the execution of an optimizer can be represented either through symbolic rule, by numeric computation. Current 1.20 approaches parameterize their update rules g by numerical predictors that can be learned from data via meta-training, They often choose sophisticated predictive models such as LTSMs (Andrychowicz et al. 2016; Lv al., [017 Chen et al.) 2020a Wichrowska et al. 2017). that predict the next update based on the current and historical optimization variables.
Despite the blessing of large learning capacity, the NN-based update rules are uninterpretable "black boxes". is never well understood what rules existing I.20 models have learned; nor is it easy to compare different L20 models and assess which has discovered more advanced rules. Moreover, numerical predictors, especially RNN-based ones, limit L20 scalability through severe memory bottlenecks (Metz et al. 2019; Wu et al. 2018). Their meta-training involves backpropagating through a full unrolled sequence of length T, for the parameter updates in their optimization task. At meta-testing time (though requiring less memory than meta--raining), one has to store RNN's own parameters & and hidden states, which are still far from negligible.
Those important gaps urge us to look back at symbolic update rules. Almost all traditional optimizers can be seen as formula-based symbolic methods within our framework, such as stochastic gradient descent (SGD). Adam, RMSprop, and so on. Their forms hinge on manual discovery and expert crafting. A symbolic optimizer has little memory overhead, and can be easily interpreted or transferred like an equation or computer program (Bello et al. 2017: Runarsson & Jonsson 2000; Orchard & Wang] 2016: Bengio et al., 1994). A handful of L20 methods tried to search for good symbolic reinforcement learning (Bello et al. 2017: Real et al. 2020 Runarsson & Jonsson 2000 Orchard Wang 2016 Bengio et al.) 1994). Unfortunately, those direct search methods quickly become inefficient when the search space of symbols becomes large. The open question remains as : how to strike the balance between the tractability of finding an update rule, and the effectiveness, memory efficiency and interpretability of the found rule?
rules from scratch, using evolutionary
In what follows, we will unify symbolic and numerical optimizers with synergistic representation and analysis framework. In brief, we stick to numerical predictors at meta-training for its modeling flexibility and ease of optimization. We then convert numerical predictors to symbolic rules, which preserve almost the same effectiveness yet have negligible memory overhead and clear explainability. Facilitated by the new symbolic representations, we will develop a new suite of L20 interpretability metrics, and can meanwhile scale up L20 to much larger optimization tasks.
3.3 A SYMBOLIC DISTILLATION FRAMEWORK FOR L20
We will first demonstrate how to convert the update predictor g: Xt+1 = Xx g(ze, Φ) into symbolic forms that preserve the same effectiveness. Contrasting with numerical forms, symbolic rules bear two advantages: (i) being white-box functions and enabling interpretability (see Section 3.3); and (ii) being much lighter, e.g., unlike RNNs that have to carry their own weights and hidden states, which removes the memory bottleneck for scaling up to large optimization tasks (see Section 3.4).
To achieve this goal, we leverage classical tool of symbolic regression (SR) (Cranmer 2020), type of regression analysis that searches the space of mathematical expressions find an equation that best fits a dataset, both in terms of accuracy and simplicity. Different from conventional regression techniques that optimize the parameters for a pre-specified model structure, SR infers both model structures and parameters from data, Popular algorithms rely on genetic programming to evolve from scratch (Runarsson & Jonsson 2000; Gustafson et al. 2005 Orchard & Wang 2016). However, SR algorithms are slow on large problems and rely on many heuristics, if evolving from scratch (Runarsson Jonsson 2000 Gustafson et al. 2005 Orchard & Wang 2016). They will become highly inefficient if further being entangled with the rule search, like existing attempts demonstrated (Real et al. 2020 Bello et al., 2017 Runarsson & Jonsson 2000; Orchard & Wang 2016).
Inspired by the idea of "knowledge distillation" (Gou et al. 202I), we propose a symbolic distillation approach that decouples the rule search step (in the numeric domain) and the subsequent repre- sentation step (to the symbolic domain). We will first learn a numerical predictor using existing L20 methods (e.g.. RNN-based), and then apply SR to fit symbolic form that approximates the input-output relationship captured by g. In this way, the former step can stay with the more efficient gradient-based search in continuously parameterized space.
4

---

Published as a conference paper at ICLR 2022
Specifically, will generate an off-line database D of (input, out) pairs, then to train SR to minimize a fitting loss on D. Each pair is obtained from running the learned g on an optimization task: input is a sequence of z, generated during optimization within the fixed unrolling length T; and oulpud is the corresponding sequence of the RNN teacher g(Z,, ②). The training procedure of SR iteratively mutate population of equation candidates, and filter out the better performing ones to come up with better equations. The final output of SR is a population of best equations composed of variables from input that best fit output of its RNN teacher gZz, +) on the target task distribution, using operaters coming from the pre-defined search space.
3.4 CONCRETIZING THE SYMBOLIC DISTILLATION FRAMEWORK
To make symbolic distillation for L20 executable, two open questions have to be defined: (i) "what to fit",, i.c., what symbolic operators we use as building blocks compose the final equation: and (ii) "fit on what"" i.e., what information we consider as SR input operands. These questions are open-ended and problem-dependens and can rather inconsistent among different L20 methods. To make our approach more generalizable, we choose relatively straightforward options for both.
For the first question, the selection of symbolic operators to use. we adopt the following set: {+,-,x, /..x://()) v., exp(+), _., tanh, arcsinh, sinh, relu, erfe). The first eight operators are adopted since they are the basic building blocks of several traditional optimizers such as Adam/- Momentum, and the rests are adopted since they are either the nonlinear activation functions used by typical L20 numerical predictors, convey certain (update) thresholding effects which often expected and empirically used in existing deep learning optimization (Sun 2019). For a few oper- ators whose input ranges are constrained to non-negative only, we extend them to be valid for any real-valued input (e.g., Vē is extended to sign(x)
Vizl, and xD is extended sign(x) |xr|, etc).
For the second question, we use the exact input variables of the numerical L20 (e.g.. coordinate wise gradients) as a default option throughout this paper. To allow for further flexibility, we also enable to expand the input operand set, to provide additional optimization-relinn SR procedure decide which to use or not. On high level, we model the obtained symbolic equation Finite Impulse Response (FIR) filter, and set maximum horizon T of the input variables. In the following parts, unless otherwise specified, T is set to 20 (e.g.. the latest 20 steps of input variables are visible to the rule). To facilitate capturing inter-variable relationships, we re-scale all input variables so that all of them have the uniform scale variances of 1.
features and let the
We conduct proof-of concept experiment to distill a learned rule using the LSTM- L20 method (Andrychowicz et al. 2016). The input sequence is {V (x,, (x;,1)) 1.,-1),.., ([(x-T++)) iee. the current and past gradients within an unrolling length T = 20. The L.20 meta-training was performed on multiple randomly-initializee LeNets on the MNIST dataset. To fit SR, we adopt classical genetic programming based SR approach (Koza) 1994). The SR algorithm can score the equation complexity level function of the operator number, operator type diversity, and input variable number used (Cranmer et al. 2020). Hence, we can apply a range of complexity levels, and obtain a group of SR fitting rules from simple complicated. One example of our distilled symbolic rules is displayed as (c,, i = O.... 19, are scalars, omitted for simplicity):
19
Ar.- 0.013e 835
erfe sinh (V.(x,)) + Σ cs tanh (V -1
(2)
Time dece-dec step size
"momentum"
3.5 INTERPRETABILITE FROM SYMBOLIC L20 REPRESENTATIONS
For general ML models, the interpretability is referred to as the ability of the model to explain or to present in understandable terms human (Doshi-Vedl Velez & Kim 2017). When it specifically comes to an L20 model, its interpretability will be uniquely linked to the immense wealth of domain knowledge in optimization, from rigorous guarantees to empirical observations (Nocedal & Wright 2006). We hence define L20 interpretability as how well a learned optimizer's 's behavior can align with the optimization domain knowledge, be understandable to the optimization practitioners.
5

---

Published as a conference paper at ICLR 2022
In fact, one may easily notice that the symbolic forms of L20 possess certain levels of "baked-in" interpretability. For example, the example update rule 2 immediately supplies a few interesting observations: L20 discovers momentum-lik, like historic weight averaging mechanism, a time-decaying step size, as well as several normalization-lik or (nonlinear) gradient clipping operations such as erfe, sinh and tanh. All these are unddrnn-adnest as they represent well-accepted practices in deep learning optimization (Sun 2019).
Our target is to establish a more principled and generally applicable L20 interpretability framework. The new framework will be dedicated to capturing 1.20 characteristics, quantifying their behaviors, and meaningfully comparing different learned optimizers. We propose two novel metrics that dissect L,20 along two dimensions: (i) Temporal Perception Field (TPF), i.e., how long input sequence 2 (past optimization trajectory) the learned optimizer "effectively" takes advantage of; and (ii) Mapping Complexity (MC), i.e., how complicated predictor model g needs to "effectively" be.
The estimations of TPF and MC are both facilitated in the symbolic domain. For TPF, take Eqn. (2) for example again: the predicted update at time involves a weighted (nonlinear) summation of past gradients V (x)=:))), 0,- U..... .T (set as 20). e; can be seen as importance indicators for V fxx)),,, and a preliminary idea to define TPF is simply the weighted mean of historical lengths:
Temporal Perception Field
(3)
21=0 Cc
One challenge will be to estimate TPF when more complicated symbolic forms arise, e.g., the update rule has more heterogeneous compositions such as a, (Vf(x=))))" + b, (Vf(x,-)))* e,tanh (V (x) In such situations, more generic designs of TPF will have to be investigated and validated such as D le/|+bb/|+le
we leave as future work.
Eis |e]]+|be+++
For MC, the SR approach in (Cranmer et al. (020)) has defined a complexity score for any resultant equation, that is calculated based on the total number and types of operators as well as input variables used for fitting it. Based on that tool, we set a threshold on the maximally allowable difference between the original numerical predictor and its symbolically distilled form (over some (imput, out) validation set), and choose the hinimum-ccopleexe its complexity score is used to indicate the intrinsic mapping complexity.
symbolic equation that falls under this threshold:
The above two definitions are in no way unique nor perfect: to quantify the understanding of L20 behaviors. We will extend the above ideas to comprehensively examining the existing L20 approaches. We aim at observing the relationship between the two metrics, and their empirical connections to L20's convergence speed, stability, as well as transferrability under task distribution shifts. Extensive results will be reported in Section 4 from which we conclude few hypotheses, including: (i) an L20 with larger TPF will converge faster and more stably on instances from the same task distribution, since it exploits more global optimization trajectory structure for this class of problems; and (ii) an L20 with smaller MC will be more transferable under distribution shifts, meta-tuning (adaptation) is performed, since tuning a less complicated predictor model might be more data-efficient. We defer more details later in section 4.2 and Appendices A
they are intended as pilot study effort
From the general taxonomy of interpretable deep learning Li et al. 2021). our approach belongs to the category of using surrogate models, i.c., fitting a simple and interpretable model (symbolic equation) from the original complicated one (numerical predictor), and using this surrogate model's explanations to explain the original predictions. Our new interpretability metrics and analyses will be the must-of-its-kiin for L.0.. The findings will provide troubleshooting tools for existing L.20 methods, and insights for designing new L20 methods.
3.6 SCALABLE L20 TUNING WITH SYMBOLIC REPRESENTATIONS
Symbolic L20 forms can not only be used as "frozen" " update rules at testing, but also be tweaked to provide weight meta-tuning ability. Taking Eqn. (2) for example again, we can keep the equation form but set all e to be trainable coefficients, and continue to fit them on new data, using differentiable meta- meta-trainin training or other hyperparameter optimization (HPO) methods (Feurer & Hutter 2019). This "tunability" " is useful for adapting a learned optimizer to a shifted task distribution, such as larger optimization tasks since the optimizer overhead now is much lighter compared to the

---

Published as a conference paper at ICLR 2022
Figure 1: The proposed symbolie regression workflow for L20 models: The meta training step enjoys the ease of optimization in neural network function representation space, and come up with a numerical L20 model; the symbolic regression step distills a light weight surrogate symbolic equation from the numerical L20 model; the meta tuning step makes the distilled symbolic equation again amendable for re-paramcterization
In balze Numenca 120
Evaluzte Complexiy Metric
Meta Training
Numenzal L20
Learned
Symbois .20
blets Tuned Symbbalic L2D
Symbolic Regression
Meta Tuning
Evaluaz Opcininzion Pesfomances
original numerical predictor. The overall procedure of training numerical 1.20 -> distill symbolic rule - meta tuning on new task are illustrated in Fig. [
4 INTERPRETABILITY AND SCALABILITY EVALUATIONS FOR SYMBOLIC L20
In this section, we systematically discuss our experimental settings and findings. Section 4.1 verifies symbolic regression's 's ability to recover underlying optimization rules through sanity checks. Section 4.2 leverages the interpretability of symbolic L20 to quantify several properties of the numerical
L.20 models. The scalability evaluation of the proposed symbolic L220 models is given in section 4.3
4.1 SANITY CHECKS FOR SYMBOLIC REGRESSION ON OPTIMIZATION TASKS
As discussed in section 3.3 with the help of few adaptative approaches (e.g., operator selection and modification, injection of inductive bias with processed features, variable magnitude re-scaling, etc.), the symbolic regression is capable to recover the underlying rules of the optimizers. To further verify the reliability of the recovery, we conduct sanity checks by regressing known optimizers, whose results are presented in [able[[ The gradients are acquired from the training of ResNet-500 on Cifar-10, and the most recent 20 steps are kept for the symbolic regression inputs. The hyperparameters for the symbolic regressions are tuned our best efforts: details could be found in Appendix C As demonstrated in table I the symbolic regression faithfully reproduces the known equations.
Table 1: Sanity checks for SR's ability to retrieve known equations. "Seconds" corresponds to the computation time accomplished via a 2.6 GHz Intel Core 17 CPU with 16 GB 2400 MHz DDR4 Memory.
Target Equation
Reowered Equation
XP score
Secords
-0001 g
de Stw. 0.8iina 72 0.20 dows 0.4baas to- Weut 0.20
o/ Souu
0,9979! Btenhhll o7g- g- 1.0007
g 0.+00- 0.3gg 0.22g
0. 0.0.eeniiyy Br-a)
B.9
WI
Man00.
Ride- Aystialifss--11 Zong ool (Thagh some moiyy coinl)) appears, overall remsins high)
4.2 INTERPRETABILITY OF TRADITIONAL OPTIMIZERS AND LEARNED OPTIMIZERS
In this section, we quantify the interpretability of the learned optimizers via two metrics TPF and MC which was defined in section 3.5 Without loss of generality, we consider three different problems (optimization tasks) and six representative numerical L20 backbones for the evaluation, described in Tables and and The three problems are:
Pi : miminn a
||Ax + b b + 0.5cT cos(x) Pa/P3 : min CrossEntropy (labels, (data; x)). (4)
2
Pi is non-convex generalized Rastrigin function used in Cao et al. 2019), where A E 10 and I E IR 10, Note that the elements in A, b, c are i.i.d. and sampled from N (0, 1) for simplicity. For P2/Ps, we acquire the data and labels from the MNIST dataset, and f denotes MLP with Relu activation. The x indicats the parameters of the MLP and the size of the neurons for each layer is (50, 20) for Pa and (50, 20, 20, 12) for P3, respectively.
7

---

Published as a conference paper at ICLR 2022
On the other hand, the six numerical L20 backbones considered are: DM (Andrychowicz al. 2016), RP (Lv et al. 2017). DM+CL+IL (DM with imitation learning and curriculum learning proposed in (Chen et al., 2020aa), RP+CL+IL, Rpismall) (architecture same as RP, but with fewer coefficients: the dimension of its input projection is reduced from 20 to 6, and the number of layers of the LSTTM is reduced from 2 to 1), and (P(enta) (architecture same as Rpismall, but with extra augmenting Adam-type gradient features into the input feature set).
Empirical observations. We report the values of TPF and MC of the learned L,20 models in table 3 and two observations could be made. First, the larger TPF, the faster convergence speed is expected (DM+CL+IL is faster than DM, and RP+CL+IL is faster than RP). Second, with more diverse features, the learned models performs better even with lower MC (RP achieves the best performance among the variants of RPs and DMs, and its input features are the most diverse).
Table 2: The comparison of input feature set, mapping function, temporal perception field and mapping complexity across several optimizers.
Optimiaass
SCID
Adomn 88)
DM
RP dir, de
Irput Festress Set
de
my Orne + (I Plah
Wiibe] -e du definee in Rig
Mapping Franction
(gg)) -og-
(Ug) -mmy
M(g))
-o() R1 Yome 2.
Temponil Perception Field
See Takle
See Table
Mapping
to fG where / = aus
I (ohen viewing ih, on iepari
See Tabie
See Table
Top half: the Temporal Perception Field (TPF) and Mapping Complexity (MC) values of learned 1.20 models. The values are averaged across three different problems Pa/PPpPss The tuples indicate the those models that have more than one type of inputs. Bottom half: the performance of numerical L20 and their symbolic distillation counterparts which are learned on P2 and evaluated on P3. The interpretations and the plots of optimization trajectory could be found in the Appendix B
Table 3:
Numerical L20 Models
DM
DM-CL+IL
RP
RP+CL-IL
Rponsti
Rplerat)
Temporal Perception Field
25.2
27.4
(543.
(3.7,
(..5
(4.6, 2.0, 4.1) 67.0
Mapping Complexity
182.9
220.6
104.7
121.4
80.3
Eva Loss by oumerical L20
0.13
0.16
0.12 0.07
0.11 # 0.04
0.09 # 0.05
0.08 0.05
Eva Loss distilled equation
0.33 + 0.26
0.33
0.14 0.09
0.02
0.10 0.06
0.09 0.04
4.3 SCALABILITY EVALUATIONS
In this section, we evaluate the scalability of symbolic L20 on large-scale optimizees. Due to the memory burden of LSTM-based L20 models, meta-training on large-scale problems are challenging for ASTM-based L.20 models (Chen et al.] 2020a; Bello et al., 2017). and most L20 models that work well with small-scale optimizees fails to scale up to large-scal scale optimizees, such Re (Chen et al. 2020a; Bello et al.) 2017: AAnrrychwiccz et al. 2016 Lv et al., 2017). Thanks to the simple form and the hidden-states- free property, symbolic I. L20 models
able to implement fast inference with little memory overhead during the training, which brings potentials to scale up to these larger-scale tasks. In the following, we show the feasibility that a symbolic L20 model could achieve better performance than handcrafted tuned optimizers by meta-tuning it on large-scale optimizees.
In our experiments, the best performing numerical I.20 model, i.e., RP is selected as the teacher to perform symbolic distillation. We first meta-pre-train a RP/small) model on P1 to get a better initialization. Then, we distill into a symbolic equation followed by mina-nine- the distilled symbolic equation on the large-scale optimizees.
Meta-line-tuning for distilled symbolic equations. Recall that Rp(small) features. The first two features are the same as the RP model's (Lv et al. 2017):
(extra) has three types of input
mr = m. 1/2 and 9. = = 1/2
(5)
where Sr denotes the gradients, m, [8] [8, + (1 (1-81))111 9))g./(111 B1,, and v, = [82u (1 32) - (3). The third type of input features ñn contains a few trainable parameters, i.e., ky, k2, 11 12, 01, and 02g. The form of n, is similar to vi 's except that 9 in m, is replaced with 1 kigb kigt-a kagt-a 1-a1, and of in v2 is replaced with l 2t22 129.-0z As discussed in table 3 the
8

---

Published as a conference paper at ICLR 2022
(extra) model has the lowest MC and TPF values, and its performance is empirically verified by the output of symbolic distillation. We summarize the the output symbolic form as follows:
(G;W) - ΣΣ
W -7 tanh(gtrr/7))
(6)
gEST=0
where 9 9 {ri,, ( Gt. 0z) is the set of input features, W E 3xL and y is the trainable coefficients. Note that the coefficient y is only tuned during the meta-finettuning stage. The accuracy of the distilled equation 6 is verified since the R² score between the distilled equation and the original (P(ootl)) model is large enough (R² 0.88). The performance of the distilled equation is also reported in table 3 Additionally, the plotted input-output curve can be referred in Appendix B
For the evaluation, we pick three large-scale optimizees, i.e., ResNet-50, ResNet-152 and Mo- bileNetV2, on the Cifar-10 and Cifar-100 datasets. Due to the scale of the optimizees, it poses difficulty for the previous L20 models execute training on a single GPU. Thanks the simple form and the hidden- seaenn-sttts-ffee property, the meta-training process for our symbolic L20 only takes less than 30GB GPU memory over ResNet-50. In addition, thanks to the initialization from pre-trained (extra) only takes one pass for the 200-epochs' meta training before the symbolic L.20 is ready.
The evaluation results are reported in 2 and Table 4 For the compar- ison, we include the following base- lines: SGD optimizers with learning rate selected from (0.1, 0.01, 0.001), with momentum selected from (0.5, 0.9, 0.99), with and without the Nes- terov momentum, with and without cosine learning rate annealing, and the Adam optimizers with learning rate selected from (0.01, 0.001). In the legend of Fig, default values are: learning rate = 0.1, momentum = 0.9, without using Nesterov momentum and without using cosine learning rate annealing. The setting of different hyperparameter selection plotted in the legend of Fig. 2 Note that the LSTM-based L20 requires significantly higher memory cost for large-scale optimizees, thus we do not include them in this section.
Table 4: Comparison of the meta-tuned symbolic rule and the traditional optimizers. Evaluation results are shown for optimizing a ResNet50 on Cifar10.
Optimizers
momentum=0.5
momentum-00
Ir-0.001
1m-0001
I--0.1
1e-0.001
Im0.01
1-01
SGD (D0 costne decay)
94.33
93.70
91.10
95.20
51.82
95.31
SGD (with cosine in decny!
93.25
94.37
94.87
92.12
93.57
94.03
Symbolic L20
95.40
As we observed in Fig 2 the proposed symbolic 1.20 model outperforms most of the human- engineered optimizers (SGD, Adam, Nesterov, cosine learning rate decay, etc.) in three tasks, in terms of not only convergence speed but also accuracy.
Figure 2: The performance comparison of the meta-fines fine-tuned symbolic 1.20 and the baseline optimizers, The symholic 1.20 model achieves even better performance than the laboriously tuned traditional optimizers.
ResNet50 on Cifar10
ResNet152 on Cifar100
MobileNetV2 on Cifar100
43 39.
Test Acc.
ALC A3 Syrboic 120
Tost ACC.
Axx Symrbokr L20 520 com--cca-mmmm
Ac: $4.33 550--cs
Acc Symos
40
580
SSDerrss 24.771 5S0 2ot2
23.02
inca
Auc 33.03 Jaaa
Apt 22, 10 (alrrkm00)))
Arr 1.00 (Atamenee
52
103
125
152
175
200
75
120
125
150
275
55
200
:50
2i0
250
202
Epachs
Epochs
Epachs
5 CONCLUSION
Current L.20 methods face important limitations in terms of their interpretability and scalability. In this work, we attempt to overcome those limitations, by unify symbolic and numerical optimizers with a synergistic representation and analysis framework. We first numerically learn a neural network based L20 model, then distill it into symbolic equation to link the wealth of optimization domain knowledge; the distilled equation can be meta- meta-tuned further on testing problems. Symbolic L20 may hopefully reveal new path to enhancing L20 to reaching the oftee-o-the--ee art practical performance. We hope our results will boost the wider adoption of L220 methods, that can eventually replace the laborious manual design or case-by-case tuning of optimizers.

---

Published as a conference paper at ICLR 2022
REFERENCES
Diogo Almeida, Clemens Winter, Jie Tang, and Wojeiech Zaremba. A generalizable approach to learning optimizers. arXiv preprint arXiv:2/06.0000 100.00988 2021.
Marein Andrychowiez, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, pp. 3981 199899, 2016.
Alexios -aiimiinnnniiiniggggg and Christoph Studer. Deep unfolding for communications systems: A survey and some new directions. In 2019 IEEE International Workshop on Signal Processing Systems (SiPS), pp. 266 271. IEEE, 2019.
Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V Le. Neural optimizer search with reinforce- ment learning. arXiv preprint 1rxii::709.07440 2017.
Samy Bengio, Yoshua Bengio, and Jocelyn Cloutier. Use of genetic programming for the search of new learning rule for neural networks. In Proceedings of the First IEEE Conference Evolutionary Computation. IEEE World Congress on Computational Intelligence, pp. 324-327. IEEE, 1994.
Mark Borgerding, Philip Schniter, and Sundeep Rangan. Amp-inspired deep networks for sparse linear inverse problems. IEEE Transactions on Signal Processing, 65(16):4293-4308, Aug 2017. ISSN 1941-0476. doi: 10.109/TSP201::2708000.
Yue Cao, Tianlong Chen, Zhangyang Wang, and Yang Shen. Learning optimize in swarms. In Advances in Neural Information Processing Systems, pp. 15018-15028, 2019.
Tianlong Chen, Weiyi Zhang, Zhou Jingyang, Shiyu Chang, Sijia Liu, Lisa Amini, and Zhangyang Wang. Training stronger baselines for learning to optimize. Advances in Neural Information Processing Systems, 33, 2020a.
Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang, and Wotao Yin. Learning to optimize: A primer and benchmark. arXiv preprint 110..228...22 2021a.
Xiaohan Chen, Jialin Liu, Zhangyang Wang, and Wotao Yin. Theoretical linear convergence of unfolded ista and its practical weights and thresholds. Advances in Neural Information Processing Systems, 31, 2018.
Xiaohan Chen, Jialin Liu, Zhangyang Wang, and Wotao Yin. Hyperparameter tuning is all you need for lista. Advances in Neural Information Processing Systems, 34, 2021b.
Xinshi Chen, Yu Li, Ramzan Umarov, Xin Gao, and Le Song. Rna secondary structure prediction by learning unrolled algorithms. International Conference on Learning Representations, 2019.
Xuxi Chen, Wuyang Chen, Tianlong Chen, Ye Yuan, Chen Gong, Kewei Chen, and Zhangyang Wang. Self-pu: Self boosted and calibrated positive-unlabeled training. In International Conference on Machine Learning, pp. 1510-1519, PMLR, 2020b.
Yutian Chen, Matthew W Hoffman, Sergio Gómez Colmenarejo, Misha Denil, Timothy P Lillicrap, Matt Botvinick, and Nando Freitas. Learning to learn without gradient descent by gradient descent. In International Conference on Machine Learning, pp.
PMLR, 2017.
M.-C. Corbineau, C. Bertocchi, E. Chouzenoux, M. Prato, and J.-C. Pesquet. Learned image deblurring by unfolding a proximal interior point algorithm. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 4664-4668. IEEE, Sep 2019. ISBN 978-1-5386-6249-6, doi: 10. 10.1109//.... 8803438.
Miles Cranmer. Pysr: Fast & parallelized symbolic symboli in python/julia, regrsionnn September 2020. URL
https:: /ttp:://do.. http:///...or.0../0///////////// 4052869
Miles Cranmer, Alvaro Sanchez-Gonzalez Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. NeurIPS 2020, 2020.
10

---

Published as a conference paper at ICLR 2022
Finale Doshi- Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint Stti::1702.080000 2017.
Matthias Feurer and Frank Hutter. Hyperparameter optimization. In Automated Machine Learning, pp. 3-33. Springer, Cham, 2019.
Raja Giryes, Yonina C. Eldar, Alex M. Bronstein, and Guillermo Sapiro. Tradeoff's between con- vergence speed and reconstruction accuracy in inverse problems. JEEE Transactions on Signal Processing, 66(7):1676-166) 1690, Apr 2018. ISSN 1941-0476. doi: 10. 10.1109/TSP2221122779999
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6) 1789-1898, 2021.
Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th international conference on international conference on machine learning, pp- 399-406, 2010.
Steven Gustafson, Edmund K Burke, and Natalio Krasnogor. On improving genetic programming for symbolic regression. In 2005 IEEE Congress on Evolutionary Computation, volume 1, pp. 912-919. IEEE, 2005.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neurul networks, 2(5):359-366, 1989.
Timothy M Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Elias Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning combinatorial optimization algorithms over graphs. Advances in neural information processing systems, 30: 6348 6358, 2017.
John R Koza. Genetic programming as means for programming computers by natural selection. Statistics and computing, 4(2):87-112, 1994.
Chaojian Li, Tianlong Chen, Haoran You, Zhangyang Wang, and Yingyan Lin. Halo: Hardware-aware learning optimize. In Proceedings of the European Conference on Computer Vision (ECCV), September 2020a.
Ke Li and Jitendra Malik. Learning to optimize. International Conference on Learning Representa- tions (ICLR), 2017a.
Ke Li and Jitendra Malik. Learning to optimize neural nets. arXiv preprint arXiv:1703.0044 2017b.
Maojia Li, Jialin Liu, and Wotao Yin. Learning combine quasi-newton methods. NerulPS workshop on on Optimization for Machine Learning, 2020b.
Xuhong Li, Haoyi Xiong, Xingjian Li, Xuanyu Wu, Xiao Zhang, Ji Liu, Jiang Bian, and Dejing Dou. Interpretable deep learning: Interpretation, interpretability, trustworthiness, and beyond. arXiv preprint arXiv:2 10..100..00000 10689, 2021.
Dong Liang, Jing Cheng, Ziwen Ke, and Leslie Ying. Deep magnetic resonance image reconstruction: Inverse problems meet neural networks. IEEE Signal Processing Magazine, 37(1):141-151) 2020.
Jialin Liu, Xiaohan Chen, Zhangyang Wang, and Wotao Yin. ALISTA: Analytic weights are as good as learned weights in LISTA. In International Conference on Learning Representations, 2019.
Kaifeng Lv, Shunhua Jiang, and Jian Li. Learning gradient descent: Better generalization and longer horizons. arXiv preprint arXiv: 2017.
Niru Maheswaranathan, David Sussillo, Luke Metz, Ruoxi Sun, and Jascha Sohl-Dickstein. Re- verse engineering learned optimizers reveals known and novel mechanisms. arXiv preprint 1.00552 2020.
11

---

Published as a conference paper at ICLR 2022
Luke Metz, Niru Maheswaranathan, Jeremy Nixon, C Daniel Freeman, and Jascha Sohl-Dickstein. Understanding and correcting pathologies in the training of learned optimizers. arXiv preprint arXiv:1810.10180 2018.
Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-Dickstein. Understanding and correcting pathologies in the training of learned optimizers. In International Conference on Machine Learning, pp. 4556 4565. PMLR, 2019.
Thomas Moreau and Joan Bruna. Understanding the learned iterative soft thresholding algorithm with matrix factorization. In International Conference on Learning Representations (ICLR), 2017. URL htttt://aaaiiiii org/ abs/a//s..1..//////// 0135/
Jorge Nocedal and Stephen Wright. Numerical optimization. Springer Science & Business Media, 2006.
Jeff Orchard and Lin Wang. The evolution of generalized neural learning rule. In 2016 International Joint Conference on Neural Networks (IJCNN), pp. 4688-4694. IEFF, 2016.
Paavo Parmas, Carl Edward Rasmussen, Jan Peters, and Kenji Doya. Pipps: Flexible model-base policy search robust the curse of chaos. arXiv preprint (KK2.0992.01244 2019.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318, 2013.
Esteban Real, Chen Liang, David So, and Quoc Le. Automl-zero: Evolving machine learning
algorithms from scratch. In International Conference on Machine Learning, pp. 8007-8019. PMLR, 2020.
Thomas Philip Runarsson and Magnus Thor Jonsson. Evolution and design of distributed learning rules. In 2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks (Cat. No. 00, pp. 59 63. IEEE, 2000.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chier Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018
Jiayi Shen, Xiaohan Chen, Howard Hcaton, Tianlong Chen, Jialin Liu, Wotao Yin, and Zhangyang Wang. Learning a minimax optimizer: pilot study. In International Conference on Learning Representations, 2021. URL http:/::////peeeeeeeeeee net. wwtt//eorrrea.. www.oooidddi.wwiee........
Harsh Shrivastava, Xinshi Chen, Binghong Chen, Guanghui Lan. Srinivas Aluru, Han Liu, and Le Song. Glad: Learning sparse graph recovery. In International Conference on Learning Representations, 2020. htts:: http://s////peevvie...... wtt.:eeooooommmii-ccc-e.ea..
Ruoyu Sun. Optimization for deep learning: theory and algorithms. arXiv preprint 1912.0999,, 2019.
Satoshi Takabe and Tadashi Wadayama. Theoretical interpretation of learned step size in deep- unfolded gradient descent. arXiv:2001.055 102002.5542220 [cs, math, stat), Jan 2020. URL http::/arxxx org/abs/ 2001. 05142
arXiv: 2001.05142.
Corentin Tallec and Yann Ollivier. Unbiasing truncated backpropagation through time. arXiv preprint arXiv:1705.08209, 2017.
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial intelligence review, 18(2) 2002.
Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. In International Conference on Machine Learning, pp. 3751-3760. PMLR, 2017.
Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding short-horizon bias in stochastic meta-optimization. arXiv preprint arXiv: 1803.0000.00221, 2018.
12

---

Published as a conference paper at ICLR 2022
Tianwei Yin, Zihui Wu, He Sun, Adrian V Dalca, Yisong Yue, and Katherine L Bouman. End-to-end sequential sampling and reconstruction for mr imaging. arXiv preprint arXiv:2105.06460) 2021.
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. L2-gen: Layer-wise and learned efficient training of graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2127-2135. 2020.
Jian Zhang and Bernard Ghanem. Ista-net: Interpretable optimization-inspree deep network for image compressive sensing. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1828-1837, 2018.
A MORE DETAILS REGARDING THE INPTERPRETABILITY
A.1 EXPERIMENTAL DETAILS
As discussed in section 4.2 we consider models in the interpretability experiments: the DM, RP, and modifications of RP. The RP model differ from the DM model in two ways: RP use two gradient features (discussed in equation where DM used the pure gradient as its input. Additionally, RP has an input projection layer where DM does not.
A.2 MORE DETAILED DISCUSSIONS
The message conveyed by the RP/smtul distillation result. Among the comparison of table 3 the
winner model the (extra) which has the smallest evaluation loss. At the same time, Rp(smal)) also the simplest model. This does not mean that the optimal gradient descent rule is simplest rule, but rather, the optimal gradient descent rule is the simple combination of complex gradient features, as the input feature of ( is the most complex one compared with other model, such as RP or DM.
(extra) is
(extra)
On the intuition behind TPF and MC: should they be large or small? In table 3 the TPF (tem- poral perception field) and MC (mapping complexity) are compared together with the performance (measured in the evaluation loss) of the model. The TPF/MC are measured for certain models w.r.t. its input, hence they are blind to the concrete components of the input. We link the different behaviors of different models and their TPF and MC, with the following summaries:
Comparing among models with the same type of input (DM vs. DM + CL + IL, or RP vs. RP + CL IL), the larger TPF/MC lead to better evaluation loss on the testing set (newly sampled, unseen problems).
Comparing across model with different input features (i.e., DM vs. RP, or RM + CL + IL vs. RP + CL + IL, or RP vs RP (P(ann)) ), the model with more complex input feature set (e.g., RP) not only outperform the model with simpler input set (e.g.. DM), but they also have smaller temporal perception field and mapping complexity. Hence, improving the input feature diversity for the numerical L20 model will reduce both TPF and MC, while still leading to better performance.
The influence of pre-training problem selection on the final performance of symbolic I,20. Since obtaining the symbolic L.20 model require a meta-pre-training step, one straightforward question may arise: what if the teraairr- step selected a different problem, how will the final results be affectd? To test the sensitiveness of the symbolic L20 to its meta-pre-train initialization step, we conducted ablation studies for the experiment in section 4.3 follows.
We set up 3 different problem settings as the meta-pre-training problems, and pre-trained the Rp(small) (extra) (best performing) and the DM (worst performing) models. After that, applied the symbolic distillation procedure, read out the distilled equation skeleton, set its coefficients as trainable, then meta- meta-fine-tume the symbolic rule in the ResNet-50 are: P1,2 introduced in section section and Pa. which also minimizing the cross entropy the MI.P model, but the dataset is Cifar instead of MNIST.
Cifar10. The three different problem chose
13

---

Published as a conference paper at ICLR 2022
We distill symbolic rule from the optimization trajectory data of the ( and DM models on three different problems. For ( the distilled symbolic equations take the same skeleton, which exactly equal to equation (6), just with different coefficients. However, with the DM model, the symbolic skeletons are significantly different, as shown as follows:
Pi -asinh 0.89 (log(1 + 0..0g. + U.00ussini(j 1) 0.01usinh(ge-a))
(7a)
P: -0.02g 0.01sign(ssst sinh
Vasiuh (g.)
0.Tasinl."" 0.3asinh ((t-)) 0.2asinh (gr-4)
(7b)
Pa: -0.02 -.0.0p
Lumbie
(7c)
of +08 +99-2
Note that the most desired output from the symbolic regression step the equation skeleton, and the coefficients are subject to be further fine-tuned in the meta- seta-fine-tune stage. Hence, the message from the ablation study results that:
When the meta-pre-train problem are changed, the knowledge learned by the L20 model with more diverse imput features (Rl)) could be more aligned under the same frame, compared with the 1.20 model with fewer input features (DM). In other words, more diverse input feature diversity of the optimizer will bring better tolerance/stability to the meta-pre-train problem selection.
When the equation's 's skeleton distilled correctly, the choice of the meta-pre-train problem does not have significant effect on the resulting symbolic 1,20's performance.
The interpretations of the distilled equations and 7 The equation o is the linear combination of multiple thresholded gradient features. First, these diverse features (gradient, Adam, learned exponential combination) enable higher chance to escape from flat local minima. Second, the nonlinear thresholding potentially smoothes noisy gradient signal and made the update more stable.
Equations 7 also have interpretable patterns. In Pl, the asinh is thresholding function which clip the gradient, and inside is log(x + 1) with x being the big square root. The function y = log( + 1) is close the funetion y when the magnitude of is small, where the gradient magnitude is indeed usually small. In P2, the equation is close to SGD in the first term -0.02g.. If the sign of gr and gt-1 is the same, i.e., two most recent gradient holds the same sign, then sign(g:9t.)) descent more, otherwise it descent less. In Pı, the equation is similar to Adam with additional tanh thresholding certain components.
How much adaptation is needed during meta-fine-tune. This experiment follows the same setting of the above adaptation experiments for DM and RP RPfomull), meta-pre-train fine-tune and evaluate on the ResNet-50 on Cifar 10. To help understand how much improvement is needed fine-tuning, the evaluation performances (measured in test accuracies) before and after the fine-tuning step are provided in table 5
22/22/Pa, and meta-
From the table, can be seen that the symbolic rule from RP/small) which have smaller MC, transfers better than DM. High complexity model DM, on the other hand, are less stable for different meta-pre- training problem.
Optimizers
DM model
Rp/smalli model
sitta-pre-trained from
P
Pa
P4
P1
Pa
P
Betore meta-fine-tuu
13.4
70.0
83.2
91.3
93.8
94.4
After meta-fine fine-tune
10.0
70.7
84,5
95.4
95.3
95.5
Table 5: Comparison for before and after meta- meta-fine-tunires fininnuning. The values are test accuracies of the symbolic equation distilled from DM and Rpfamall) when meta-pre-trained on Pt/Pa/Ps.
Why could the symbolic distillation provide better interpretabilityy With regard to our claim in white-box functions have better interpretability and lighter weight, one question may
section 3.3
14

---

Published as a conference paper at ICLR 2022
arise: the interpretability and light weight properties do not seem unique to symbolic optimizers, but rather, they seem unique to simple optimizers.
Indeed, the simpler model will naturally hold better interpretability than the complex model. However, whether the rule learned by the numerical optimizer is simple or not is not revealed by the numerical optimizer itself: it is only after the proposed symbolic distillation is applied can people realize that the rule learned by RP is simple (has small MC). It is also only after the symbolic distillation uncovered this fact can people safely simplify the RP model into the Rp model, to make it lighter weight and obtain better interpretability and better performance (see table 3).
In other words, the interpretability indeed belongs to simple optimizers, but numerical optimizer can not tell the simplicity of the rules it has learned, prior to applying the proposed symbolic distillation. In this sense, the proposed symbolic distillation brings better interpretability to numerical L20.
The running time of the proposed method. The proposed symbolic distillation procedure contains the following components: the meta-pre-training. SR step, mnea--finettning and the final evaluation to optimize the large scale problem. The execution time for each step is provided in table 6 The proposed TPF and MC metrics are all simple calculations based on the SR results, hence they are instantly available once the SR results are out. The execution times for SR and computing TPF/MC are measured on the 2.6 GHz Intel Core i7 CPU with 16 GB 2400 MHz DDR4 Memory, and other other computation times measured on the Nvidia A6000 GPU.
Step
Meta-pre-train
Meta-fine-tune 20 bours
Final evaluation 8 bours (MotoleNaV2) V2- 20 hours (RasNei-1522
Comporing TPHMC:
Esecution Lime
mi (*))- 10 min (P3)
S hours
Table The computation execution times of the experiments section 4.3
B
MORE DETAILS REGARDING THE SCALABILITY
B.1 EXPERIMENTAL DETAILS
During the symbolic regression step, the outputs are a list of distilled equation candidates. For each candidate, a unique complexity value could be measured using Cranmer (2020). During the sanity check in section 4.1 the complexity levels are chosen to be the largest output, which relatively small (<5 for SGD, <50 for Momentum, 100 for Adam). During the experiments in section 4.3] we pick the equation with 100 complexity value, since this number gives the best fitting ability well as the best performance (check figure and table for details) according to our extensive experiments.
We have also tested varying the random seeds for both meta-pre-trai and the symbolic regression, and check the resulting symbolic form. Due to the low MC of the learned rule of (cxtra) the distilled equation stably shows the same form (equation 6), regardless of the random seeds.
We used 128 batch size, and in both meta-fine-tune phase and final evaluation phase, we trained the CNN optimizees for 200 epochs.
B.2 MORE DETAILED DISCUSSIONS
Verification of the symbolic rule fitting ability. We verified that the distilled equation and the original optimizer are well aligned. First, shows the fitting curve the symbolic equation against the numerical model. In this figure, we chose fixed time during the optimization, and plot the output of both original numerical L.20 and the distilled symbolic equation. The results show that the distilled equation fits the original model well. Second, in table we have distilled different optimizers, and have shown SR's 's fitting ability for each, indicated by Rs-ocoee. The high Re-score further verified the fitting ability is reliable for our tasks.
The interpretations of the symbolic L20 ultimately used in section 43 In section the best numerical model (RP(mal)) as the benchmark distill the symbolic equation, and the distilled equations for Rp in most cases take the form of equation (6). We note that this simple nonlinear thresholding function yielded good fitting accuracy, with the R2-score 0.88. We note that the simple
4.3 used
15

---

Published as a conference paper at ICLR 2022
Table 7: SR evaluations results: the fitting and optimization ability across different SR options.
mom((0.5
Adarm(0.99 0.9)
DM
RP
Distilled from
P1
P5
Ps
P)
Pa
F1
P5
Ps
R R'-score
0.83
0.85
0.25
0.31
0044
0.54
0.93
0.94
0.34
0.94
Models
Model
model
Crafted Octmbbors
Dullec
ar
1000
4:000
4.00
4.1
Figure 3: The sampled mapping functions different optimizers w.r.t their input feature sets. The colorful marked lines are the numerica/hhand-craffe equations. In the line markers, @A@B" means the mapping function is A, the input of this function is B, and at the current time step. is the only variable. It can be observed that the symbolic distillation fits the underlying optimization algorithms accurately.
models, while the black solid lines are the distilled symbolic
form has already been verified in the small mapping complexity and temporal perception field of RL.si in table
Performance comparison of the numerical and distilled symbolic rules. In table we offered the performance comparison between the numerical rules and their distilled symbolic surrogates. To further illustrate their convergence speed, we plot the optimization trajectories for the DM model (the worst performing one) and the RP/omull) model (the best performing one) in figure 4 Specifically, the lines in figure 4 correspond to the following rows/columns in table 3 the Evaluation Loss by numerical L20 and the Evaluation Loss by distilled equation, for the DM and RP ap models.
Optimization trajectory numerical L20 models
symbolic surrogates
1.4
Distiled
12-
RP_s1
Evaluation Loss
20-
8.8
0.6
9.4
3.0
400
Steps
Figure 4: The optimization trajectorics of the DM and the Rpomall) models.
The relationship between SR complexity and fitting ability/accuracy. The symbolic regression algorithm is able to generate series of equations with different complexity. Intuitively, the over- simple equations under fits the L2O model, and the over-complex equations tends to overfit. The exact fitting curve is shown in Fig. 5L In this figure, the y coordinate is the R2-score of the fitting performance, the higher the better, and the z-coordinate is the complexity of the equation. With the increase of the complexity, the fitting performance tends to first increase then decrease.
The influence of complexity to the accuracy are displayed in table 8 The experimental setting in this (esura)
table is the same setting section 4.3 and the numerical optimizer used is also Rp(small)
16

---

Published as a conference paper at ICLR 2022
Olee
DM
O4H CL+
SP 5
DIFCI DRFCLEE
Tempora Perception Field
RR-score
25-
15
0.2
169
220
200
The complexity distilled equations
Complexity of the Distilled Equation
Figure Left: the fitting performance the equations with different level complexity. The R²-score indicates the fitting accuracy level of a single distilled equation compared with the original numerical optimizer. Right: the estimated TPF evaluated over distilled equations with different level of complexities. These figure revealed the relationship between the distilled equation complexity with the fitting ability and the estimated TPF. The values come from numerical L20s meta-pre Miliia-pe--iee trained over Ps (MNIST dataset with shallow MLP).
Complexity Accuracy 93.5
100
95.4
12.0
Table 8: The evaluation accuracies of the distilled equations with different complexities from the RP (extr) model.
The experimental setting is the same section 4.3 and the values are over ResNet50 on Cifar10.
Interpreting the hyperbolic functions presented in the distilled equations. The hyperbolic func- tions appears in the distilled equations. There are two possible reasons for the occurances of the hyperbolic functions. First, the symbolic equations are learned through gradient-free random muta- tion, hence is possible that such hyperbolic thresholding effect has a higher chance to reduce the uncertainty under noisy gradient data, which makes these operators to be more easily selected and kept among other mutated results. Note that for tanh()/ainh((/ete, these functions equal to function when the input magnitude is small (the case of tanh is plotted in figure (6) Therefore, only when the inputs have large magnitude will adding these functions result in significant difference in the output.
approximately
1.78
Function Series:
0.75
Y tanh
0.5
0.2
y
y=
-128
Gradient feature g+
Figure 6: The tanh() function series. These functions are used in equation 6 where the learnable coefficient determine the region of the lincarity (within which the tanh() function is close to x)
Second, the tanh() functions in the distilled equations of LSTM models are straightforward to understand: the symbolic rules the surrogate of the L.TT model, and the LSTM uses the tanh() function as its non-linear activation function. Therefore, the tanh() function the distilled equations could be correctly reflecting the nonlincarity of the LSTM teacher's updating dynamics.
We are more interested in the distillation of a real numerical L20 model, hence have conducted new series of experiments, by comparing not including and including the hyperbolic functions. The distilled equations without hyperbolic functions take similar forms than with the hyperbolic functions,
17

---

Published as a conference paper at ICLR 2022
Task
Metric
Adam Result
SymbolicL.20 Result
SST-2
Accuracy
89.15
91.53
QQP
F1/Accuracy
86,3/89.2 22
87.8/90.8
QNLI
Accuracy
89.2
90.1
MRPC
Fl/Accuracy
83.5/78.5
87.5/83.
MNLI
Matched acc.
82.6
84.0
CoLA
Matthews corr
52.15
56,49
Table 9: GLUE fine tune performance comparisons
and are slightly different in the coefficients. The Re-score does not improve by dropping these functions. This could be explained by that even given the hyperbolic functions, the equations with and without these functions are all compared and screened during the SR step, hence the resulting equation is already the best among them, though with the hyperbolic.
Given hyperbolic (T22 wore = 0.54) 0.02g 0.00--ign(/e----1) (neinh (gu) O.Teinhh (9h-1) +0. 0.5exinhh (dr-s) 0.2aiihh" ((t-4)
(8a)
Without hyperbolic (IP score - 0.53) :- 0.02 00 (y 0.96 0.8001. 0.T0l 0.50 0.A6o + 0.3gi-d - 0.1g
(8b)
Large scale evaluations on GLUE tasks
We have evaluated the symbolic rule used in section 4.3 on BERT with GLUE tasks. We compared between symbolicL20 and the AdamW. For both optimizers, we used Ir 0.0005. For AdamW. we used 3 = 0.9, 82 = 0.999. In all these experiments, we fine-tuned the BERT (BERT base cased model English) for 100 epochs. For symbolicl.20, we fixed the coefficients, without meta-tuning them while fine-tuning BERT. The results are as follows. From the results, the learned symbolicl.20 achieved best results across all tasks.
C SYMBOLIC REGRESSION PRELIMINARIES
The psudo-code for the SR algorithm. As described in section 1 the symbolic regression works by constantly mutating a list of equations to gradually obtain the better performing one. The core mutation algorithm of SR is provided in figure 7 as pseudo-code.
The hyperparameters that chose. There several hyperparameters for the symbolic regression procedure that could potentially influence the performance. We list them as follows:
The number of iterations: it means the number of executions, among which the best equations are picked. We set it to a large enough value, 300, since it shows in the experiment that any larger values leads to the similarly good results.
The population number: it means the number of symbolic equation candidates used each iteration. is empirically observed that simpler models is enough to be accurately distilled with smaller population number. After tuning this parameter across all L20 models, used the largest best population number, 200, for all models.
Dataset size: means the number of samples in the SR. We used 5000 samples across the experiments, which is verified to be sufficient for our purpose.
18

---

Published as a conference paper at ICLR 2022
Algotithm, thm: mutation operations in Symbolic Regression function Random Mutate (dataset):
cweights is the weights used to randomize mutation type. If equation too big, don't add new operators
if n >= maxsize I| depth >= maxdepth
oweights [3] 0.0
oweights[4] 0.0
end
successful mutation false
mutationChoice rand() & zandomly select one mutation type. Motations
while (Lsuccessful _ & attempts max_attempts
tree = copyÑode (prev)
successful euccessful_mtationn mutation true
if mutationChoice cweights
tree mutateConstant (tree, temperature, options)
elseif mutarionChoice cweights
Can always mutate to the same operator
tree mitate@perator (tree, options)
elseif mutacionChoice cweights
Can potentially have situation without
tree appendRandon@ipttre,, options, afeatures) elseif mutacionchoice [weights
tree insertRandondp((rre,, options, nfeatures)
el elseif mutationChoice (weights
tree deletteRunddn@pttreee, options, afeatures)
elseif mutacionChoice ( cweights(6]
tree simpl iffple options) # Somet imes we simplify tree tree conbine0perators (tree, options)
return PoyMember (tree, beforeLoss, parent-parent__ee
elseif muta:ionChoice cweights[7]
tree genRandomTTree (5, options, nfeatures)
else 8 no mutation applied
return PopMember (tree, beforeLoss, pazenb-pacent_reef
end attempts end
if [sccessffll mutation
return PopMember (copyNode (prev), beforeLoss, parent-parent_ref, end if probChange < rand ()
return PopMember (copyNode (prev), beforetoss, parent-parent_res
return PopMember (tree, afterLoss, parent-parent_eee _ef) end
Figure 7: The pseudo-code for the core mutation operation used in the SR algorithm.
19


Reviews:
[{"review_id": "fdD1Uyo2Dp", "reviewer": "Reviewer_ALsy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While I'm not very familiar with the L2O literature, I find the paper is difficult to read not due to my lack of domain knowledge but due to the writing and presentation. I suggest the authors re-organize the paper and address the following concerns:\n - Technical preliminaries should be put into section 3 as related work does not involve any formal definitions\n - Symbolic regression is the most critical component of the proposed method. I suggest the authors provide formal definitions of the algorithm and other details (figures, pseudo-code) to show how this algorithm fits into the framework. If it is borrowed from the previous work then this part should go to preliminaries.\n - Experimental results and technical details should be put into separate sections: they are tangled up in the current draft. For example:\n  - Section 3.3, \"We conduct a proof-of-concept experiment\"\n  - Section 3.4, \"Extensive results will be reported in Section 4, from which we conclude a few hypotheses, including\"\n\t\nTerms and notations are used without explanation/definitions:\n - What is a Finite Impulse Response filter?\n - What is R^2 score in Table 1?\n\t\nMixed-use of technical terms:\n - Meta-pre-train vs meta-fine-tune vs meta-tuned vs meta-training\n\nThe faithfulness of SR:\n - Since SR learns a surrogate optimizer from a fixed snapshot of a numerical optimizer, it is critical to show that SR with the proposed operators is sufficient and accurate to recover the original optimizer.\n - However, this is demonstrated only with simple known equations in Table 1, instead of real numerical L2O models\n\nI'm also concerned about the methodology:\n - While I agree with the motivation of improving the interpretability of the L2O model, the resulting symbolic equation is difficult to understand as well. For example, in Eq.2, why hyper-parameters are set to these values? What does T=20 mean vs. setting T to other values e.g. 30?\n - To learn the symbolic surrogate optimizer, one must first pre-train a numerical L2O model. If the pre-trained model is fully trained, doesn't the pre-training phase have the same computational cost as the traditional method?\n - If only consider the inference phase, the author claims the optimizer can be further meta-tuned for better performance. It is unclear how well does it compare with directly fine-tuning the pre-trained L2O model, as the latter is more flexible.\n\n\nExperiments:\n - What does the number mean in Table 4? How can this table show that the proposed method is more scalable than the numerical ones?\n - The author could provide the running time of the proposed method showing the computational cost of all its components: L2O pre-training, SR, meta-tuning, TPF and MC computing.", "summary_of_the_paper": "This paper proposes a symbolic L2O framework that aims to improve the scalability and the interpretability over the numeral L2O methods. The proposed framework uses symbolic regression to turn a snapshot of the numerical method into a surrogate symbolic optimizer. The resulting optimizer can be further fine-tuned with less computational costs and has a certain degree of interpretability.\n", "summary_of_the_review": "In summary, this paper needs an overhaul of its writing and presentation. The proposed method is not well-justified and the important technical details are missing. The experiments are lacking and cannot support the claims made by the paper. That said, I recommend rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "ZGRcPBjSjDh", "reviewer": "Reviewer_mSQF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Section 2.2\n\nI think the paper should really discuss some of the other symbolic optimizer papers in this section. A number of these are listed in section 3.2, but should really be discussed here as prior work.\n\nAlso, this [paper](https://arxiv.org/abs/2011.02159) on interpreting learned optimizers seems worth discussing.\n\n## Section 3.1\n\n(Minor point) Section 3.1 states that an optimizer can be either represented through a “symbolic” rule or by “numeric” computation. Ultimately, isn’t every optimizer (whether parameterized by a neural network or not) symbolic on paper, but numeric when instantiated by a computer? For example, an LSTM optimizer, if I understand correctly, would be characterized as “numeric” by this paper. But the LSTM is made up of symbolic equations! Moreover, “symbolic” optimizers such as gradient descent or momentum are numerical methods when implemented using finite precision arithmetic in a computer.\n\nSection 3.1 also claims that “numerical predictors, especially RNN-based ones, limit L2O scalability through severe memory bottlenecks”. This paragraph confusingly discusses both memory limitations of meta-training an optimizer (the T copies of the model required for unrolled backprop), as well as the memory cost of running the optimizer itself (which depends on how the optimizer is parameterized, it is unclear to me whether this is a severe memory bottleneck). The difficulties of carrying out unrolled backprop over long sequences has been discussed in other papers (e.g., Metz et al 2019, [Understanding and correcting pathologies in the training of learned optimizers](http://proceedings.mlr.press/v97/metz19a.html); or Wu et al, [Understanding short-horizon bias in stochastic meta-optimization](https://arxiv.org/abs/1803.02021)), I think some discussion of this prior work is appropriate here.\n\nThis paper proposes scaling up learned optimizers by a multi-step process: training a neural-net-parameterized optimizer using unrolled backprop on small problems, simplifying that optimizer using symbolic regression, and then finally tuning the symbolic regression weights on larger scale problems. This implicitly assumes a hypothesis is true: that whatever algorithm a learned optimizer implements on small problems will be appropriate for large problems. I think this hypothesis should be clearly stated, and perhaps the authors could also comment on what _types_ of small optimization problems might be best for getting a good learned optimizer that yields symbolic regression rules that work well on larger scale problems.\n\n## Section 3.2\n\nSection 3.2 claims that symbolic optimizers are: (1) white-box functions that enable interpretability, and (2) are much lighter. I would push back against these claims, these points do not seem unique to symbolic optimizers, but rather, they seem unique to _simple_ optimizers. I can easily make up a complicated, heavy, uninterpretable symbolic optimizer.\n\n## Section 3.3\n\nWe need much more detail about the experiments performed in section 3.3. Details of the symbolic regression algorithm? How is a “complexity level” chosen? How does the performance of the symbolic regression optimizer compare against the original, for a range of different complexity values? How sensitive are these results to random seeds? These questions could be much more thoroughly answered in the paper.\n\n## Section 4.1\n\nIn section 4.1, how is the complexity level chosen to get these results? How much data do you need (how many optimization runs)? Can you provide more details about the optimization problem (ResNet on Cifar10, but what was the batch size / number of epochs)?\n\nTable 1 does provide some interesting results. For example, it looks like you often get spurious hyperbolic trig functions (tanh, asinh, etc.). Could you comment more on why you think those occur? If the symbolic regression is _not_ given hyperbolic trig functions as symbols, do the r^2 values improve?\n\nPerhaps the authors could also comment on the choice of forcing the symbolic regression to mimic the optimizer without access to _any_ state variables. This makes, for example, mimicking things like momentum much more difficult. With a state variable, momentum only needs access to the current gradient. But to approximate this, the symbolic version needs to approximate the exponential decay of coefficients for some number of gradient steps. Is it possible to give the symbolic regression access to some state variable, to make the resulting equations even more accurate and interpretable?\n\n## Section 4.2\n\nTable 2: How is the “temporal perception field” of Adam zero?\n\nIt would be nice to see a 2D scatter plot with mapping complexity on one axis (perhaps on a log scale), and the performance of the optimizer on the other (also perhaps on a log scale, depending on how you compute it).\n\n## Section 4.3\n\nMany more details are needed:\nWhy is pre-training done on P_1, as opposed to P_2 or P_3? If I understand correctly, P_2 and P_3 are more similar to the large scale problems (they also involve training neural networks).\nWhy is the performance of Adam at chance? Is there a bug, or is something badly tuned?\nAre you picking the best baseline over the grid of hyperparameters, or plotting just one example from that grid in Figure 2? The text is a little unclear\nResults would be more convincing with a more thorough baseline hyperparameter search.\nWhat is the stopping criterion for the different optimizers? Are they not all trained for a fixed number of epochs?\n\nWhat happens if you take one of these symbolic optimizers, and test them on a whole bunch of other different optimization problems? Do they generalize at all?\n\nFinally, what are the symbolic rules learned after meta-tuning on the large scale problems?!? Are they interpretable? Seems like a big opportunity is missed by not showing any of the learned symbolic rules after Table 1. If the rules are interpretable, can you explain why they have good performance?", "summary_of_the_paper": "This paper proposes using symbolic regression for making simpler, more interpretable learned optimizers.\n\nThey do this by first training a neural-network parameterized learned optimizer, and then fitting a set of symbolic equations to capture its behavior using symbolic regression. The paper shows that the method recovers aspects of the ground truth rule when applied to baseline optimizers. Then, the paper applies their technique to learn symbolic optimizers for large scale neural network training problems.", "summary_of_the_review": "Overall, this paper addresses an important problem (better understanding what learned optimizers are doing) and does so by proposing  a new technique: using symbolic regression to fit interpretable equations to pre-trained learned optimizers.\n\nI think the paper could be greatly improved by:\n- Adding many more details about the experiments and methods. In particular, the \"complexity\" regularization of the symbolic rules seems like an important parameter. It's worth showing results as this parameter is swept across different values, as (presumably) there is a simplicity-vs-performance tradeoff that occurs.\n- Show us the learned symbolic equations for the optimizers in section 4.3! The whole point is to have interpretable equations that we can stare at. Showing that these have good performance is one thing, but it would be much more interesting to tie that performance to particular parts of the symbolic equations.\n\n[[ UPDATE AFTER REBUTTAL ]]\nAfter reviewing the author response and updated manuscript, I have chosen to increase my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TlDtrZsuY06", "reviewer": "Reviewer_jXyx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Stengths\n\n* The paper tackles and important problem with a sound approach, which is presented well overall.\n* The paper improves tuned optimizers on  ResNet50 + CIFAR10 and MobileNetV2 + CIFAR100.\n* It is great to see that an optimizer on a toy task can transfer well to CIFAR10/100.\n\n# Weaknesses\n\n* What is the distilled symbolic rule (before tuning the constants) that you use in Section 4.3? Can you provide its explicit form (perhaps masking some constants would make sense for clarity purposes).\n* I'd expect more interpretability analysis about the symoblic rule used in Section 4.3.\n*  In my opinion, the paper needs experiments on other datasets to make a convincing case for scaling the method in the paper. One suggestion is to try to finetune a BERT-like architecture on GLUE, starting from the distilled symbolic rule used in Section 4.3.\n* What's the intuition behind TPF and MC? Do we want them to be large or small? From Table 3, it seems like you want them to be small for better transferability; that makes sense to me.\n* How important is the toy task to the results in section 4.3? Was it crucial that you used $\\mathcal{P}_{2,3}$ to select the best symbolic distillation? I'd expect some ablations here.\n\n## Minor.\n\n* page 3: You should use \\citet in \"(Chen et al., 2017) takes the optimizees'...\"\n* Figure 1: make the green arrows longer and reposition the \"Symbolic Regression\" and \"Meta Tuning\" texts, so that it's clear to what arrow the texts correspond ot.\n* Broken equation link in Table 2.\n* page 8: \"better performance better performance\" <- \"better performance\"\n* Table 3: can you report standard deviations as well?\n* No evidence in the main text that \"RP models generally have better performance\". Can you provide detailed results to convince the reader?\n* page 9: learn <- learning; distill <- distilling\n* It seems like the authors were not able to complete the appendix. Completing the appendix is an important task, because for this empirical paper hyperparameters and other details about the L2O methods, and ablation studies, would be very useful.", "summary_of_the_paper": "This is an ambitious paper that tackles an important problem: how to train general-purpose and data-driven optimization frameworks. The main idea of the paper is to explore symbolic learning to optimize (L2O) by distilling a numerical L2O optimization rule into a symbolic rule. The motivation for using a symbolic distillation is to provide interpretability and scalability of the trained optimizers. The contributions of the paper are as follows: introduction of the idea of symbolic distillation in the L2O literature; tools to interpret trained optimizers; results on larger-scale optimization tasks.", "summary_of_the_review": "My recommendation is 5: marginally below the acceptance threshold. I like the idea of the paper, and overall the execution looks promising. However, I think that some important details and experiments are needed before the paper is published. If the authors address most of my questions/ comments, and if no obvious red flags appear, I would be very happy to update my recommendation.\n\n--post rebuttal\n\nThank you for the clarifications and the new experiments. I will raise my score to 6. However, I recommend better tuning of AdamW for the GLUE experiments, because the baselines are rather weak, i.e. not all tasks match the results in Table 1 of the BERT paper (https://arxiv.org/pdf/1810.04805.pdf). I would also suggest that the authors add the new experiments in the appendix and fix the typos in the blue text, e.g. missing space before citation and missing closing parenthesis. Also, in the appendix when you discuss the interpretability of the equation from P_1, I think you mean \"asihn\" instead of \"sihn\".\n\nFinally, I would like to note that the paper seems acceptable only after extensive experimental details were made available in the Appendix and the blue text. It's unfortunate these details were missing originally, but I am glad they are available now :).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OcKId3pwES3", "reviewer": "Reviewer_FzGU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The two main ideas in this paper are:\n(1) One can train the neural optimizer on small problems and distill it into symbolic form. Then apply it to large problems with adaptation. This can help the interpretability and scalability of the learned optimizer.\n(2) TPF and MC are proposed as two metrics to understand the optimizer.\n\nThe ideas and the overall framework are sound. However, the major issue of this paper is that the proposed method and the claims are not well-supported by the experiments.\n\nExperiments in Sec 4.1:\n- The sanity check is a good experimental setting to verify the correctness of symbolic regression. This set of experiments are meaningful but symbolic regression is not the main contribution of this paper. The main paper didn't even describe the concrete algorithm for symbolic regression. Apart from the distilled equations for known optimizers, we are more interested in seeing more examples of distill equations for the neural optimizers (such as the symbolic L2O in Section 4.3).\n\nExperiments in Sec 4.2:\n- The reported numbers seem to be insufficient to support the claims. The authors try to show that the metrics TPF and MC are linked to optimizers behavior such as the convergence and transferability. However, why are the metrics TPF and MC reported only without numbers measuring the model accuracy, convergence, generalization abilities are reported along with these metrics?\n- To support some claims such as 'larger TPF will help converge faster' and 'models with smaller TPF and larger MC will be more transferable', one needs to measure the convergence and transferability of different optimizers and see whether such a linkage is true.\n\nExperiments in Sec 4.3:\n- What is the symbolic form of the distilled optimizer? This very important result is missed.\n- Although the authors said RP_si model is the best performing one so they choose to distill this model. It will be also interesting to see what's the performance of distilling other neural optimizers and also the expressions of those symbolic optimizers. Otherwise, we only see one example.\n- Table 4 says 'ResNet152 on Cifar10'. Is this a typo? Should it be 'ResNet50 on Cifar10'?\n- Since the symbolic optimizer is meta-fine-tuned on ResNet-50 on Cifar10, it is better to compare what is the performance before and after the adaptation in order to get an idea of how much the adaptation is needed.\n- Furthermore, it is also good to know what is the performance of training the symbolic optimizer from scratch on ResNet-50 on Cifar10. This can verify the necessity of distilling the optimizer from a neural optimizer. Especially, when the neural optimizer is trained on a different smaller problem, it is difficult to see whether there will be a performance drop if one trains the symbolic optimizer directly on large target problems.\n- It seems the symbolic L2O does not perform the best on ResNet152 on Cifar100. Could the authors elaborate on this?\n- Finally, as mentioned that the LSTM based methods require more memory so a comparison is not shown. I think for research interest it is beneficial to make such a comparison on smaller problems to see how large the performance gap is.", "summary_of_the_paper": "This paper proposed symbolic learning to optimize (L2O), in which a neural optimizer will be trained and then distilled into a symbolic form via symbolic regression. This symbolic optimizer will then be applied to solve large problems, with some adaptation and fine-tuning.\n\nThis paper also proposed two metric called TPF and MC to understand the behavior of the optimizers.", "summary_of_the_review": "The overall framework and concept are sound. This will be a good paper if the experiments can support the arguments. However, the current experiments have left out too many details and results for validating the effectiveness of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "07RL6f8TWgS", "reviewer": "Reviewer_VHkg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths: \n1. Learning a symbolic L2O model from a numerical L2O model looks novel to me.\n2. The paper shows appealing results in the comparison of other rule-based optimizers. \n\nweakness:\n1. It seems like we need to manually define a \"suitable\" function class for the symbolic regression.\n2. I think the only reason people would first train a numerical L2O model and then distill it into a symbolic version is to apply this rule to other datasets. However, the paper lacks an experiment of the generalization of the learned symbolic L2O update rule. Moreover, I also expect a comparison (I don't see it anywhere) between the performance of numerical L2O and the distilled one. ", "summary_of_the_paper": "This paper proposed to learn a symbolic interpretable L2O model from the common numerical L2O methods. The key idea is to select the proper function class for the symbolic updating function and apply the symbolic regression to distal a light, interpretable, symbolic L2O model. In their experimental results, they first demonstrate the capability of symbolic regression for some know functions and then \nthey show the good distilling performance of their proposed symbolic regression method on empirical evaluation, where the learned symbolic L2O method is better than existing baseline optimization methods.\n\n", "summary_of_the_review": "I lean to accept this paper since it is nice and appealing to distill the complicated numerical L2O model into a symbolic optimizer.\nHowever, I think the paper would be improved dramatically by showing the generalization performance of the symbolic L2O model as well as the performance difference between the numerical L2O model and the distilled symbolic one.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for learning to optimize (L2O) by distilling a numerical L2O optimization rule into a simple mathematical rule, mathematical equation, using special-purpose student learning algorithm.  The motivation for using a symbolic distillation is to provide interpretability and scalability of the trained optimizers. 

Pros
 - The paper addresses an important problem (better understanding learned optimizers).
 - The experiments give good evidence that learned black-box optimizers can be mapped to mathematical rules.

Cons
 - The symbolic regression student learning algorithm, and many details of the experimentation, remain hard to understand.

Overall, after discussion, the paper was viewed as a solid contribution by the reviews, with some slight disagreement about whether the clarity was sufficient after revisions.  However, I believe that the main points of the paper and the general approach are quite clear, and that the details of the experiments and student learner are sufficiently well-explained for other researchers to build on, at least given the appendix and the supplementary material.
Decision: Accept (Poster)