SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xinyi Chen
xinyic@princeton.edu
Elad Hazan
ehazan@princeton.edu
Choosing the optimal hyperparameters, including the learning rate and momentum, for specific optimization instances is a significant yet nonconvex challenge. This makes conventional iterative techniques such as hypergradient descent insufficient in obtaining global optimality guarantees in general.
We consider the more general task of meta-optimization – online learning of the best optimization algorithm given problem instances. For this task, a novel approach based on control theory is introduced. We show how meta-optimization can be formulated as an optimal control problem, departing from existing literature that use stability-based methods to study optimization. Our approach leverages convex relaxation techniques in the recently-proposed nonstochastic control framework to overcome the challenge of nonconvexity, and obtains regret guarantees vs. the best offline solution. This guarantees that in meta-optimization, we can learn a method that attains convergence comparable to that of the best optimization method in hindsight from a class of methods.
The performance of optimization methods, in particular for the application of training deep neural networks, crucially depends on their hyperparameters. Bengio [2012] notes that the learning rate is the "single most important parameter" for efficient training. However, this parameter is also notoriously hard to tune without intensive hyperparameter search. An even more difficult problem is adapting all aspects of an optimization algorithm, including momentum, preconditioner, weight decay, and more, to potentially changing tasks. This is the motivating problem of our paper.
The problem of learning the best optimizer, even for the special case of hyperparameter tuning of the learning rate, is difficult because optimizing these hyperparameters is nonconvex. We tackle this challenging problem by a novel framework that is based on several components:
- We formalize the problem of "finding the best algorithm" as the task of meta-optimization. In this task, the player is given a sequence of optimization problems. Her goal is to solve them as fast as the best algorithm in hindsight from a family of possible methods. This setting generalizes hyperparameter tuning to encompass several attributes, including the preconditioning matrix and momentum.
- We propose a new metric for meta optimization called meta-regret, which measures the total cost compared to the cost of the best algorithm in hindsight in meta-optimization. We give an efficient method for minimizing the meta-regret that attains sublinear guarantee, which is tight. The implication of this efficient method with optimal meta-regret bound is that we can compete with the best hyperparameters in hindsight (although not necessarily find them).
- We give a novel control formulation for meta-optimization based on the recently proposed framework of online nonstochastic control. This control formulation differs in several
aspects to previous control methods applied to optimization: it is based on optimal control (rather than Lyapunov stability), and requires robustness to nonstochastic perturbations.
- The online control formulation gives rise to our efficient method which is based on the Gradient Perturbation Controller (GPC), a new type of online feedback-loop controller that requires only gradient access from the objective functions.
While the method itself is based on recent advancements in control, its analysis requires new technical components. Meta-optimization requires learning between optimization episodes, which necessitates the analysis of feedback control in changing environments without bounded disturbances. This presents a challenge of bounding the magnitude of the state, which we address for the first time in the online control literature.
1.1 The setting of meta-optimization
In meta-optimization, we are given a sequence of optimization problems, called episodes. The goal of the player is not only to minimize the total cost in each episode, but also to compete with a set of available optimization methods.
Specifically, in each of the  $N$  episodes, we have a sequence of  $T$  optimization steps that are either deterministic, stochastic, or online. Throughout the paper, we use  $(t,i)$  to denote time  $t$  in episode  $i$ . At the beginning of an episode, the iterate is "reset" to a given starting point  $x_{1,i}$ . In the most general formulation of the problem, at time  $(t,i)$ , an optimization algorithm  $\mathcal{A}$  chooses a point  $x_{t,i} \in \mathcal{K}$ , in a convex domain  $\mathcal{K} \subseteq \mathbb{R}^d$ . It then suffers a cost  $f_{t,i}(x_{t,i})$ . Let  $x_{t,i}(\mathcal{A})$  denote the point chosen by  $\mathcal{A}$  at time  $(t,i)$ , the protocol of this setting is formally defined in Algorithm 1.
Algorithm 1 Meta-optimization
1: Input:  $N, T, \mathcal{A}, \mathcal{K}$ , reset points  $\{x_{1,i}\}_{i=1}^{N}$ .  
2: for  $i = 1, \ldots, N$  do  
3: for  $t = 1, \ldots, T$  do  
4: Play  $x_{t,i} = \mathcal{A}(f_{1,1}, \ldots, f_{t-1,i}) \in \mathcal{K}$  if  $t > 1$ ; else play  $x_{1,i}$ .  
5: Receive  $f_{t,i}$ , pay  $f_{t,i}(x_{t,i})$ .  
6: end for  
7: end for
The standard goal in optimization, either deterministic, stochastic, or online, is to minimize the cost of each episode in isolation. In meta-optimization, the goal is to minimize the cumulative cost in both each episode, and overall in terms of the choice of the algorithm. We thus define meta-regret to be
$$
\operatorname {M e t a R e g r e t} (\mathcal {A}) = \sum_ {i = 1} ^ {N} \sum_ {t = 1} ^ {T} f _ {t, i} (x _ {t, i} (\mathcal {A})) - \min  _ {\mathcal {A} ^ {*} \in \Pi} \sum_ {i = 1} ^ {N} f _ {t, i} (x _ {t, i} (\mathcal {A} ^ {*})),
$$
where  $\Pi$  is the benchmark algorithm class. The meta-regret is the regret incurred for learning the best optimization algorithm in hindsight, and captures both efficient per-episode optimization as well as competing with the best algorithm in  $\Pi$ .
Meta-regret can be studied under various optimization settings. In the deterministic setting, the objectives are the same across time steps and episodes; in the stochastic setting, the objectives are drawn from distributions that can vary at each  $t$ ; finally, in the adversarial setting, the objectives appear online. Meta-regret guarantees can be refined for each setting; for example, in the deterministic setting, meta-regret translates to a bound on the cost of the average iterate.
Why is meta-optimization hard? It is natural to apply standard techniques from optimization, such as local search or gradient based methods, to meta-optimization. Indeed, this has been studied in numerous previous works under related settings, e.g. Baydin et al. [2017], Chandra et al. [2019].
However, the resulting optimization problem is nonconvex and local gradient optimization may reach suboptimal solutions. For example, a natural application of meta-optimization is to learn hyperparameters of a class of optimization algorithms. In particular, finding the optimal gradient descent learning rate for a specific objective takes the form of the following minimization problem:
$\min_{\eta}f(x_T)$  , subject to  $x_{t + 1} = x_t - \eta \nabla f(x_t)$
We can unroll the minimization objective from the initial point  $x_{1}$
$$
{f (x _ {T}) = f (x _ {1} - \eta \nabla f (x _ {1}) - \eta \sum_ {t = 2} ^ {T - 1} \nabla f (x _ {t}))} {= f (x _ {1} - \eta \nabla f (x _ {1}) - \eta \nabla f (x _ {1} - \eta \nabla f (x _ {1})) - \ldots)}
$$
From this expression, it is clear that  $f(x_{T})$  is a nonconvex function of  $\eta$ .
The setting of meta-optimization relates to and/or generalizes previous approaches for hyperparameter tuning in optimization. For example, it is the online learning analogue of the average case analysis framework of optimization algorithms [Pedregosa and Scieur, 2020, Paquette et al., 2021], and generalizes control-based approaches for analyzing a single optimization instance [Lessard et al., 2016, Casgrain and Kratsios, 2021].
1.2 Main theorem statements
Applying iterative optimization methods directly to the meta-optimization problem is unlikely to result in global guarantees, since the problem is nonconvex. We take a different approach: we formulate meta-optimization as an online optimal control problem. Since the control formulation has disturbances that are not stochastic, as shown in Section 2, we adopt the recently-developed nonstochastic control framework, and use techniques therein to develop efficient methods for unconstrained meta-optimization of quadratic and convex smooth functions.
The following are informal statements of our results, where the algorithm referenced is detailed in Section 3. Due to space constraints, we include the bandit algorithm in Appendix D
Theorem 1.1 (Informal). For quadratic losses, Algorithm  $\boxed{2}$  satisfies
$$
\operatorname {M e t a R e g r e t} (\mathcal {A}) = \sum_ {i = 1} ^ {N} \sum_ {t = 1} ^ {T} f _ {t, i} \left(x _ {t, i} ^ {\mathcal {A}}\right) - \min  _ {\mathcal {A} ^ {*} \in \Pi} \sum_ {i = 1} ^ {N} \sum_ {t = 1} ^ {T} f _ {t, i} \left(x _ {t, i} ^ {\mathcal {A} ^ {*}}\right) = \tilde {O} (\sqrt {N T}).
$$
Theorem 1.2 (Informal). For convex smooth losses, a bandit variant of Algorithm  $\boxed{2}$  satisfies
$$
\mathbb {E} \left[ M e t a R e g r e t (\mathcal {A}) \right] = \tilde {O} ((N T) ^ {3 / 4}).
$$
In both statements,  $\tilde{O}$  hides factors polynomial in parameters of the problem and logarithmic in  $T$ ,  $N$ . The sublinear scaling in both  $N$  and  $T$  implies that with more episodes, the average performance approaches that of the best optimizer from a family of optimization methods in hindsight. Our theorem holds for the most general optimization setting, the adversarial setting where  $f_{t,i}$  appears in an online fashion. The theorem can be refined for the deterministic and stochastic settings, for more details see Appendix F.
Online convex optimization and nonstochastic control. Our methods for meta-optimization are based on iterative gradient (or higher order) methods with provable regret guarantees. These have been developed in the context of the online convex optimization framework. In this framework, a decision maker iteratively chooses a point  $x_{t} \in \mathcal{K}$ , for a convex set in Euclidean space  $\mathcal{K} \subseteq \mathbb{R}^{d}$ , and receives loss according to a convex loss function  $f_{t}: \mathcal{K} \mapsto \mathbb{R}$ . The goal of the decision maker is to minimize her regret, defined as
$$
\operatorname {R e g r e t} = \sum_ {t = 1} ^ {T} f _ {t} \left(x _ {t}\right) - \min  _ {x \in \mathcal {K}} \sum_ {t = 1} ^ {T} f _ {t} (x).
$$
Notably, the best point  $x^{*} \in \mathcal{K}$  is defined only in hindsight, since the  $f_{t}$ 's are unknown a priori. For more information on this setting as well as an algorithmic treatment see [Hazan et al., 2016].
Techniques from online convex optimization were instrumental in developing an online control theory that permits nonstochastic disturbances and is regret-based. Deviating from classical control theory,
the online nonstochastic control framework treats control as an interactive optimization problem, where the objective is to make decisions that compete with the best controller in hindsight. For a detailed exposition see [Hazan and Singh 2022]. In general, computing the best controller in hindsight is a nonconvex optimization problem if we allow general convex costs. However, online nonstochastic control has provable guarantees in this setting despite the nonconvexity, since it uses policies in a larger policy class via convex relaxation. This is crucial for our guarantees: meta-optimization can be a nonconvex problem, but we can hope to use feedback control to bypass the nonconvexity and obtain meta-regret bounds.
Average case analysis of optimization. A closely related framework for analyzing optimization methods is the average-case analysis framework developed in [Pedregosa and Scieur, 2020]. This framework studies the expected performance of optimization algorithms when the problem is sampled from a distribution, and allows for more fine-grained results than typical worst-case analysis. Average-case optimal first-order methods for minimizing quadratic objectives are proposed in [Pedregosa and Scieur, 2020], and [Domingo-Enrich et al., 2021] extends the study to bilinear games. The meta-optimization setting is relevant to, but significantly more general than the average-case analysis framework, since we do not assume known stochastic distribution of the optimization problems, and we compete with the best algorithm in hindsight.
Hypergradient descent and hyperparameter tuning for optimizer parameters. Hyperparameter optimization is a significant problem in deep learning practice, and thus was intensively studied. The work of Baydin et al. [2017] apply local iterative methods to the problem of optimizing the learning rate from an empirical standpoint, and [Chandra et al., 2019] gives better practical methods. However, even the simple case of optimizing the learning rate can be nonconvex, and it is unlikely that iterative methods will yield global optimality guarantees. Certain provable guarantees for quadratic functions and scalar learning rate are presented in [Wang et al., 2021b].
Control for optimization. The connections between control and optimization go back to Lyapunov's work and its application to the design and analysis of optimization algorithms. We survey the various approaches in detail in Appendices A and B. Lessard et al. [2016] apply control theory to the analysis of optimization algorithms on a single problem instance. They give a general framework, using the notion of Integral Quadratic Constraints from robust control theory, for obtaining convergence guarantees for a variety of gradient-based methods. Casgrain and Kratsios [2021] study the characterization of the regret-optimal algorithm given an objective function, using a value function-based approach motivated by optimal control. In contrast, we aim to learn the optimal algorithm from a sequence of optimization problems, with potentially different objectives and characteristics.
Learning to optimize. Learning to Optimize (L2O) is a related framework with a focus on empirical methods. The goal is to learn an optimizer, usually parameterized by a deep neural network, automatically on a set of training problems that can also generalize to unseen problem instances. Andrychowicz et al. [2016] propose learning updates in a coordinate-wise fashion using an LSTM architecture. Wichrowska et al. [2017] later suggest several modifications to improve the generalization capabilities of learned optimizers. More recently, Metz et al. [2022] proposes a learned optimizer trained on a wide variety of tasks, using evolutionary strategies for estimating gradient updates to the meta-optimizer.
Meta-learning. Meta-learning Thrun and Pratt [1998], Finn et al. [2017] aims to learn algorithms or model parameters that can adapt rapidly to new tasks. Regret guarantees over model parameters for online meta-learning were developed in Finn et al. [2019], and guarantees for adaptive gradient-based meta-learning were given in Balcan et al. [2019]. The latter work provides guarantees for meta-learning optimization algorithms, but the performance metric in this context is an upper bound on the actual performance. In contrast, meta-regret measures the performance gap directly and minimizing it is a nonconvex problem, necessitating the use of techniques from online nonstochastic control theory.
1.4 Structure of the paper
In Section 2, we introduce the new control formulation of meta-optimization, and elaborate on the assumptions needed to apply the nonstochastic control framework. In Section 3, we state the algorithm and main results, as well as the benchmark algorithm class. Illustrative experimental
results of our meta-optimization methods are given in Section 4. In Appendix A and B, we give more background on the historical use of control theory in optimization. We describe the recent framework of online nonstochastic control in Appendix C and why it is important for meta-optimization. Generalizations of our dynamical system formulation to capture smooth convex meta-optimization is given in section D. More expressive optimizers are given in Appendix E. The extension to the stochastic and deterministic settings is given in F. Finally, we include more experiments, their descriptions and settings in Appendix J.
2 Online control formulation of meta-optimization
In this section, we explain our control formulation of the meta-optimization problem, representing a first step to developing an efficient control-based method. Different from historical perspectives, which we outline in Appendices A and B our formulation gives rise to an optimal control problem rather than a stability analysis.
In its simplest form, an optimal control problem is a minimization problem over the controls:
$$
\min  _ {u _ {1}, \dots , u _ {T}} \quad \sum_ {t = 1} ^ {T} c _ {t} \left(z _ {t}, u _ {t}\right) \tag {1}
$$
$$
\begin{array}{l l} \text {s . t .} & z _ {t + 1} = A z _ {t} + B u _ {t} + w _ {t}. \end{array}
$$
The states are denoted as  $z_{t}$ , and for a linear dynamical system they evolve according to the dynamics determined by  $A$  and  $B$ . The control signals and disturbances are denoted as  $u_{t}$  and  $w_{t}$ , respectively. The control objective is the sum of control costs  $c_{t}$ , which are functions of the states and controls. This optimization problem has explicit solutions under certain conditions, if we know the cost functions a-priori; however, since meta-optimization is inherently online, we turn to the more recent regret-based online control setting.
In online control, the convex cost functions can appear in an adversarial fashion, and the controller can choose the controls iteratively as the cost functions are revealed. The goal in online control is to minimize policy regret:
$$
\sum_ {t = 1} ^ {T} c _ {t} \left(z _ {t}, u _ {t}\right) - \min  _ {\pi \in \Pi_ {C}} \sum_ {t = 1} ^ {T} c _ {t} \left(z _ {t} ^ {\pi}, u _ {t} ^ {\pi}\right), \tag {2}
$$
where  $\Pi_C$  denotes the comparator policy class,  $z_t^\pi$ ,  $u_t^\pi$  denote the state and controls under the policy  $\pi$ , respectively. The policy regret is the excess total cost of the control algorithm, compared to that of the best controller in  $\Pi_C$  in hindsight. In the following discussion, we specify how meta-optimization can be formulated as an online control problem.
For simplicity, consider the online convex optimization setting where all objective functions  $f_{t,i} : \mathbb{R}^d \to \mathbb{R}$  are convex quadratic. Denote the location-independent Hessian as  $\nabla^2 f_{t,i}(x) = H_{t,i} \in \mathbb{R}^{d \times d}$ . This setting can be generalized to convex smooth functions, as we detail in Appendix D. We assume that the functions have bounded Hessian, and the gradients satisfy:
Assumption 1.  $\forall i\in [N],t\in [T],x\in \mathbb{R}^d,\| \nabla f_{t,i}(x)\| \leq 2\beta \| x\| +b.$
The dynamical system. The state of the dynamical system at time  $(t,i)$  is given by the vector below consisting of the current optimization iterate, the previous iterate, and the previous gradient,
$$
z _ {t, i} = \left[ \begin{array}{c} x _ {t, i} \\ x _ {t - 1, i} \\ \nabla f (x _ {t - 1, i}) \end{array} \right].
$$
For an extension of the control formulation to include more iterates and gradients, see Appendix E.
Let  $\delta > 0$  be a regularization parameter, and  $\eta > 0$  be the base learning rate, consider the system:
$$
\left[ \begin{array}{c} x _ {t + 1, i} \\ x _ {t, i} \\ \nabla f _ {t, i} (x _ {t, i}) \end{array} \right] = \left[ \begin{array}{c c c} (1 - \delta) I & 0 & - \eta I \\ I & 0 & 0 \\ H _ {t, i} & - H _ {t, i} & 0 \end{array} \right] \times \left[ \begin{array}{c} x _ {t, i} \\ x _ {t - 1, i} \\ \nabla f _ {t - 1, i} (x _ {t - 1, i}) \end{array} \right] + \left[ \begin{array}{c c c} I & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{array} \right] \times u _ {t, i} + \left[ \begin{array}{c} 0 \\ 0 \\ \nabla f _ {t, i} (x _ {t - 1, i}) \end{array} \right], \tag {3}
$$
which is a valid representations of the optimization process given the relation
$$
\nabla f _ {t, i} (x _ {t, i}) = \nabla f _ {t, i} (x _ {t - 1, i}) + H _ {t, i} (x _ {t, i} - x _ {t - 1, i}).
$$
The disturbance in the system evolution includes  $\nabla f(x_{t - 1,i})$  as a component, which is not stochastic in nature. Consequently, classical control approaches such as  $\mathcal{H}_2$  optimal control are not applicable, and we adopt the online nonstochastic control framework [7]. This formulation also enables us to capture the state reset when a new episode begins by using adversarial disturbances.
Without the control signal  $u_{t,i}$ , the system describes the partially time-delayed gradient descent update with  $\ell_2$  regularization:  $x_{t+1,i} = (1 - \delta)x_{t,i} - \eta\nabla f(x_{t-1,i})$ . Several results show that under mild conditions, the regret using delayed gradients are constant factors away from online gradient descent using fresh gradients [Quandrud and Khashabi, 2015; Langford et al., 2009]. The parameter  $\delta$  is user-specified to ensure stability, and can be arbitrarily small.
The control signal  $u_{t,i}$  contributes only to the update of  $x_{t,i}$ , and does not affect the rest of the state. Observe that we can simulate any optimizer's update by using the control signal, and therefore we can learn the best optimizer in hindsight by optimizing over the  $u_{t,i}$ 's.
In meta-optimization, the dynamical system evolves according to (3) during an episode. However, each episode starts with the optimization iterate at an arbitrary initialization  $x_{1,i}$ , and we need to reset the system before a new episode begins. To transition the system state to the new initialization, consider the following reset disturbance,
$$
w _ {T, i} = \left[ \begin{array}{c} x _ {1, i + 1} - ((1 - \delta) x _ {T, i} - \eta \nabla f _ {T - 1, i} (x _ {T - 1, i}) + \bar {u} _ {T, i}) \\ x _ {1, i + 1} - x _ {T, i} \\ \nabla f _ {T, i} (x _ {T - 1, i}) - \nabla f _ {T, i} (x _ {T, i}) \end{array} \right], \tag {4}
$$
where  $\bar{u}_{T,i}$  is the top  $d$  entries of the control signal  $u_{T,i}$ . Under this reset disturbance, the initial state of an episode is  $z_{1,i} = \left[x_{1,i}^{\top} x_{1,i}^{\top} 0\right]^{\top}$ , which is consistent with the meta-optimization protocol. Finally, we assume that the initial point in each episode has bounded norm.
Assumption 2. For all  $i$ ,  $x_{1,i}$  satisfies  $\| x_{1,i}\| \leq R$ .
Online control formulation. Consider the online control problem (2), where the system evolves according to (3) and the control costs are the optimization objectives,
$$
\sum_ {i = 1} ^ {N} \sum_ {t = 1} ^ {T} f _ {t, i} \left(x _ {t, i}\right) - \min  _ {\pi \in \Pi_ {C}} \sum_ {i = 1} ^ {N} \sum_ {t = 1} ^ {T} f _ {t, i} \left(x _ {t, i} ^ {\pi}\right). \tag {5}
$$
Note that the cost functions are functions of the states:  $c_{t,i}(z_{t,i},u_{t,i}) = f_{t,i}(Sz_{t,i}) = f_{t,i}(x_{t,i})$ , where  $S$  is a matrix that selects the first  $d$  entries of  $z_{t,i}$ . In addition, observe that even though the cost functions do not explicitly depend on the controls  $u_{t,i}$ , they still affect the states through the control policy. This online control problem directly corresponds to meta-regret minimization, where the benchmark algorithm class  $\Pi$  consists of optimizers whose updates match the dynamics of system (3) under controllers in  $\Pi_C$ .
Stability. In nonstochastic control, a stable system or access to a stabilizing controller is necessary to achieve regret that is polynomial in the state dimension [Chen and Hazan 2021]. The notion of stability in this context is stronger than the conventional notion, and for linear time-varying (LTV) dynamical systems, it is referred to as sequential stability. We assume our system satisfies this condition.