SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Definition 2.1 (Sequentially stable). An LTV system whose dynamics at time  $t$  is  $z_{t+1} = A_t z_t + B_t u_t + w_t$  is  $(\kappa, \gamma)$  sequentially stable if for all intervals  $I = [r, s] \subseteq [T]$ ,  $\| \prod_{t=s}^{r} A_t \| \leq \kappa^2 (1 - \gamma)^{|I|}$ .
Assumption 3. For settings where the functions  $f_{t,i}$  are changing, we assume that the resulting dynamical system is sequentially stable with  $\kappa \geq 1$ .
<sup>1</sup>Notice that the regret bounds we prove are under the same sequence of disturbances, and apply to the gradients along the taken trajectory. This is similar to the nature of adaptive gradient methods: the best regularization in hindsight depends on the observed gradients, rather than the ones that would appear had we used a different algorithm.
The above assumption is standard in the literature of nonstochastic control for time-varying systems [Gradu et al. 2020b], [Minasyan et al. 2021]. If the dynamical system is linear time-invariant (LTI), then Assumption 3 is satisfied if the system is strongly stable, whose definition is given below.
Definition 2.2 (Strong stability). A system  $z_{t+1} = Az_t + Bu_t + w_t$  is  $(\kappa, \gamma)$  strongly stable if there exist matrices  $P, Q$ , such that  $A = PQP^{-1}$ , and  $\|Q\| \leq 1 - \gamma$ ,  $\|P\|, \|P^{-1}\| \leq \kappa$ .
In the deterministic meta-optimization setting, we have an LTI system. We show in the appendix that with  $\eta$  scaling inversely as the smoothness of the function, and  $\delta$  arbitrarily small, the LTI system is strongly stable for some  $(\kappa, \gamma)$ . The condition on  $\eta$  is natural, since gradient descent with learning rate larger than  $\frac{1}{\beta}$  diverges.
3 Algorithm and main theorem statements
In this section, we give an efficient algorithm that minimizes meta-regret and state accompanying guarantees. Our main algorithm for quadratic meta-optimization is given in Algorithm 2. This algorithm views meta-optimization as a single-trajectory control problem, and uses the Gradient Perturbation Controller (GPC) method adapted from Agarwal et al. [2019]. The GPC method considers a specific class of controllers, namely disturbance-feedback controllers, that are linear in the past disturbances, as defined below.
Definition 3.1 (Disturbance-feedback controller). A disturbance-feedback controller (DFC),  $\pi(M)$  is parameterized by a sequence of  $L$  matrices  $\{M^l\}_{l=1}^L$ , where  $M^l$  denotes the  $l$ -th matrix, instead of a matrix power. It outputs the control signal  $u_{t,i} = \sum_{i=1}^{L} M^l w_{t-l,i}$ .
The GPC method uses the class of DFC controllers, and makes gradient updates on the parameters of interest,  $M$ , to compete with the best DFC in hindsight.
Similar to the GPC method, at each time step, Algorithm 2 outputs the control signal generated by  $\pi(M_{t,i})$ , and receives an objective function. Then, the disturbances are computed according to the dynamical system formulation (3), and a cost function for updating  $M$  is constructed on Line 9. Finally, a gradient update is executed on  $M$ .
The surrogate cost on Line 9 has the following expression:
$$
g _ {t, i} \left(M _ {t - L, i}, \dots , M _ {t - 1, i}\right) = c _ {t, i} \left(y _ {t, i} \left(M _ {t - L, i}, \dots , M _ {t - 1, i}\right)\right), \tag {6}
$$
where the surrogate state  $y_{t,i}$  is the state reached at time  $(t,i)$ , if  $z_{t - L,i} = 0$  and we execute the sequence of policies  $M_{t - L,i},\ldots ,M_{t - 1,i}$ . Note that the function  $g_{t,i}(M) = g_{t,i}(M,\dots ,M)$  is a function of only  $M$ , instead of a sequence of time-varying controls. As a result, when we take a gradient of  $g_{t,i}(M)$ , the gradient captures the effects of changing  $M$  over multiple time steps.
For smooth convex optimization, we cannot directly observe the dynamics  $A_{t,i}$ . There are two consequences for our algorithm: we cannot access the full gradient with respect to  $M$ , and we cannot compute  $y_{t,i}$  directly. However, a bandit variant of Algorithm 2 overcomes both difficulties, as the algorithm approximates  $y_{t,i}(M,\dots,M)$ , and estimates the gradient of  $M$  with zeroth-order methods. The algorithm is given in Appendix D.
3.1 Guarantees for quadratic and smooth meta-optimization
We first state the regret guarantees of our algorithms, and describe the benchmark algorithm class in more detail in the next subsection. All proofs are in the full version at https://arxiv.org/abs/2301.07902.
Theorem 3.2 (Quadratic). Under Assumptions  $\boxed{1}, \boxed{2}, \boxed{3}$ , Algorithm  $\boxed{2}$  with  $\eta \leq 1, \delta \in (0, \frac{1}{2}]$ ,  $\eta_{g} = \Theta(\sqrt{NT})^{-1}$ , and  $L = \Theta(\log NT)$  satisfies
$$
M e t a R e g r e t = \sum_ {i = 1} ^ {N} \sum_ {t = 1} ^ {T} f _ {t, i} (x _ {t, i}) - \min  _ {\mathcal {A} \in \Pi} \sum_ {i = 1} ^ {N} \sum_ {t = 1} ^ {T} f _ {t, i} (x _ {t, i} ^ {\mathcal {A}}) \leq \tilde {O} (\sqrt {N T}),
$$
where  $\Theta$  contains polynomial factors in  $\gamma^{-1},\beta ,\kappa ,R,b,d,$  and  $\tilde{O}$  contains these polynomial factors as well as logarithmic factors in  $T,N$
Algorithm 2 Gradient perturbation controller for meta-optimization
1: Input:  $N, T, z_{1,1}, \eta, \delta, \eta_g$ , starting points  $\{x_{1,i}\}_{i=1}^{N}, \kappa, \gamma$   
2: Set:  $\mathcal{M} = \{M = \{M^1, \dots, M^L\} : \|M^l\| \leq \kappa^3 (1 - \gamma)^l\}$ , and initialize any  $M_{1,1} \in \mathcal{M}$ .  
3: for  $i = 1, \dots, N$  do  
4: If  $i > 1$ , set  $z_{1,i} = z_{T+1,i-1}$ ,  $M_{1,i} = M_{T+1,i-1}$ .  
5: for  $t = 1, \dots, T$  do  
6: Choose  $u_{t,i} = \sum_{l=1}^{L} M_{t,i}^l w_{t-l,i}$  and observe  $z_{t+1,i}$ .  
7: Receive  $f_{t,i}$ , compute  $\nabla f_{t,i}(x_{t,i})$ ,  $\nabla f_{t,i}(x_{t-1,i})$ . If  $t = T$ , compute  $w_{T,i}$  according to (4).  
8: Suffer control cost  $c_{t,i}(z_{t,i}) = f_{t,i}(x_{t,i})$ .  
9: Construct ideal cost  $g_{t,i}(M) = g_{t,i}(M, \dots, M)$  according to (6).  
10: Perform gradient update on the controller parameters, where  $\Pi$  denotes projection:  
[ M_{t+1,i} = \Pi_{\mathcal{M}}(M_{t,i} - \eta_g\nabla g_{t,i}(M_{t,i})). ]  
11: end for  
12: end for
It follows from lower bounds in online convex optimization [Hazan et al. 2016] that the dependence of meta-regret on  $N, T$  are optimal in Theorem 3.2. The next theorem holds for meta-optimization over convex smooth objectives, and the additional assumptions are given in Appendix D. Algorithm D.2 and the proof of Theorem 3.3 are based on nonstochastic control techniques in the bandit setting [Gradu et al. 2020a].
Theorem 3.3 (Smooth). Under Assumptions  $\boxed{2}$ $\boxed{3}$ $\boxed{4}$ $\boxed{5}$  Algorithm  $\boxed{D.2}$  with  $\eta \leq 1$ ,  $L = \Theta (\log NT)$ , there exists a setting of  $\delta_{M}$ ,  $\{\eta_{t,i}^{g}\}$  that satisfies
$$
\mathbb {E} \left[ M e t a R e g r e t \right] = \mathbb {E} \left[ \sum_ {i = 1} ^ {N} \sum_ {t = 1} ^ {T} f _ {t, i} (x _ {t, i}) \right] - \min _ {\mathcal {A} \in \Pi} \sum_ {i = 1} ^ {N} \sum_ {t = 1} ^ {T} f _ {t, i} (x _ {t, i} ^ {\mathcal {A}}) \leq \tilde {O} ((N T) ^ {3 / 4}),
$$
where  $\tilde{O}$  contains polynomial factors in  $\gamma^{-1},\beta ,\kappa ,R,b,d,$  and logarithmic factors in  $T,N$
Remark 3.4. Stronger performance metrics that are more suitable for changing environments, such as adaptive regret and dynamic regret, were explored in the context of control of time varying dynamical systems, see [Gradu et al. 2020b Minasyan et al. 2021]. These latter results can potentially be adapted to give stronger performance guarantees in meta-optimization.
Analysis overview and technical novelty. The regret of Algorithm 2 is the excess total cost of our controller compared to that of the best fixed DFC in hindsight. Since the system is strongly stable, the control cost at  $(t,i)$  depends minimally on the controls executed before  $L$  time steps, e.g.  $M_{t - L,i}, M_{t - L - 1,i}\dots$ . Therefore, we can focus on a surrogate cost that is a function of only the past  $L$  control signals, where  $L$  is logarithmic in the horizon  $NT$ . Bounding the regret then reduces to obtaining low regret on such surrogate costs, which was studied in the online learning with memory (OCOWM) framework [Anava et al., 2015].
Crucially, this analysis relies on universal upper bounds on the states and disturbances, which is the main technical challenge of this work. Indeed, this assumption does not hold in our setting, since the disturbances include  $\nabla f_{t,i}(x_{t - 1,i})$ . Under the smoothness assumption,  $\| w_{t,i}\|$  can grow proportionally with  $\| x_{t - 1,i}\|$ . Furthermore, the reset disturbance can be on the same scale as the states themselves.
We overcome this challenge by showing that due to system stability, the effect of the reset disturbance attenuates exponentially. In each episode, after the initial large reset disturbance, the state and disturbance have an upper bound that decreases with time and eventually converges to a constant value. Moreover, if we scale the objective functions properly, we can bound the contribution from  $w_{t,i}$  to the size of the state because of smoothness. To make this intuition formal, we use several inductive arguments to address the interdependence between the states and disturbances.
3.2 The benchmark algorithm class
We introduce the benchmark algorithm class we compete against, which consist of optimizers that correspond to a class of DFCs. Informally, our guarantee is competitive with optimizers that are linear functions of past gradients.
The comparator policy class we consider for control is  $\Pi_{DFC}$ , the set of all disturbance-feedback policies defined by  $M$  such that  $M \in \mathcal{M}$  defined in Algorithm 2. The GPC method obtains sublinear regret against  $\Pi_{DFC}$  under a fixed sequence of disturbances, cost functions, and dynamics. However, if these sequences are fixed in meta-optimization, the gradient component in the state is no longer the true gradient; instead, they are pseudo-gradients. Let  $x_{t,i}^{M}$  denote the optimization iterate at time  $t$  in epoch  $i$  under controller  $\pi(M) \in \Pi_{DFC}$ . Define the pseudo-gradient at time  $(t,i)$  as
$$
\hat {\nabla} f _ {t, i} ^ {M} (\cdot) = \nabla f _ {t, i} (\cdot) - \nabla f _ {t, i} (x _ {t - 1, i} ^ {M}) + \nabla f _ {t, i} (x _ {t - 1, i}),
$$
where  $x_{t - 1,i}$  is the optimization iterate of our algorithm. Then the states under  $\pi (M)$  are
$$
z _ {1, i} ^ {M} = \left[ \begin{array}{c} x _ {1, i} \\ x _ {1, i} \\ 0 \end{array} \right], z _ {2, i} ^ {M} = \left[ \begin{array}{c} x _ {2, i} ^ {M} \\ x _ {1, i} \\ \nabla f _ {1, i} (x _ {1, i}) \end{array} \right], z _ {t, i} ^ {M} = \left[ \begin{array}{c} x _ {t, i} ^ {M} \\ x _ {t - 1, i} ^ {M} \\ \hat {\nabla} f _ {t - 1, i} ^ {M} (x _ {t - 1, i} ^ {M}) \end{array} \right] \mathrm {f o r} t \geq 3.
$$
Observe that each policy in  $\Pi_{DFC}$  corresponds to an optimization algorithm. Writing out the control signals, the benchmark algorithm class consists of optimizaers with updates:
$$
x _ {t + 1, i} ^ {M} = (1 - \delta) x _ {t, i} ^ {M} - \eta \hat {\nabla} f _ {t - 1, i} ^ {M} (x _ {t - 1, i} ^ {M}) + \sum_ {l = 1} ^ {L} M ^ {l} w _ {t - l, i}.
$$
This benchmark algorithm class has a more straightforward interpretation for deterministic meta-optimization. It can capture common algorithms on time-delayed pseudo-gradients, including gradient descent, momentum, and preconditioned methods. For details on the range of hyperparameters it contains, see Appendix G for a concrete example of learning the learning rate, see Appendix H
We evaluate the empirical performance of our meta-optimization algorithm on a suite of tasks, including regression and classification. For an illustrative example, we consider regression with the objective  $\frac{1}{2}\| Ax - b\|^2$ . We first generate  $A$  and  $b$  element-wise using the standard Gaussian distribution. To simulate the stochastic meta-optimization setting, we have  $N$  episodes and  $T$  time steps, and the optimization iterate is re-initialized to the same point at the beginning of each episode. The initial value is randomly sampled. For each  $t \in [T]$ , and  $i \in [N]$ , we add entry-wise standard Gaussian noise to  $b$ . We set  $d = 100$ ,  $T = 500$ ,  $N = 10$ , and compare with gradient descent, momentum, Marchenko-Pastur, and nesterov accelerated gradient. The Marchenko-Pastur (MP) algorithm is optimal in the average-case sense for regression tasks whose design matrix has entries sampled from an i.i.d Gaussian distribution, and therefore should perform well in our setting without noise [Pedregosa and Scieur (2020)]. We tune the learning rate and momentum hyperparameters for all algorithms, and the sweep details are in Appendix []. For meta-optimization, we use  $L = 3$ ,  $\delta = 0$ , and a base learning rate of 0.001. We plot the moving average of the objective values in Figure [7] with a window size of 50.
The plot shows that initially, meta-optimization has similar performance to other algorithms. However, with an increasing number of episodes, the algorithm gradually learns and outperforms baselines both at the beginning and, eventually, towards the end of each episode. This illustrative experiment shows that for this particular problem, meta-optimization can improve and learn updates that minimize the objectives quickly.
For a proof-of-concept nonconvex task, we consider MNIST classification with a neural network. The network has 3 layers and is fully connected, with layer sizes [784, 512, 10]. We run for 5 episodes with 20 epochs in each episode. Since this problem is high-dimensional, we choose to learn scalar weights of past gradients. We take  $L = 20$ , and the network is trained using stochastic gradients from batches of size 256. The hyperparameters of the baselines are tuned, and the sweep details are in Appendix J. We also include Adam in our baseline, since it is a popular optimizer.
Figure 1: Comparison plot for the performance of meta-optimization and other baselines.
Figure 2: Comparison plot for the training performance of meta-optimization and other baselines with a suboptimal base learning rate.
In this experiment, the base learning rate  $\eta$  for the meta optimizer is set to 0.5 instead of the optimal learning for SGD, which is 1.0. We show the training curves of SGD with learning rates of both 1.0 and 0.5 for comparison. Examining the plot, it is clear that the meta optimizer outperforms SGD with learning rate 0.5 by a wide margin, and continues to converge to the performance of momentum over episodes. We include more experiments in Appendix J.
5 Conclusion.
This manuscript proposes a framework for optimization whose goal is to learn the best optimization algorithm from experience, and gives an algorithmic methodology using feedback control for this meta-optimization problem. In contrast to well-studied connections between optimization and Lyapunov stability, our approach builds upon new techniques from the regret-based online nonstochastic control framework. We derive new efficient algorithms for meta-optimization using recently proposed control methods, and prove regret bounds for them.
An interesting direction for future investigation is extending the work on nonstochastic control of time varying and nonlinear dynamical systems to our setting of meta-optimization. In particular, it is interesting to explore the implications of low adaptive and dynamic regret in meta-optimization. Another area of investigation is the connection between meta-optimization using control, and adaptive methods: do the solutions proposed by the control approach have connections to adaptive regularization-based algorithms? Many intriguing problems arise in this new intersection of optimization and nonstochastic control.
Elad Hazan acknowledges funding from the ONR award N000142312156, the NSF award 2134040, and Open Philanthropy.
Kent Quanrud and Daniel Khashabi. Online learning with adversarial delays. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
R Tyrell Rockafellar. Linear-quadratic programming and optimal control. SIAM Journal on Control and Optimization, 25(3):781-814, 1987.
Max Simchowitz, Karan Singh, and Elad Hazan. Improper learning for non-stochastic control. In Conference on Learning Theory, pages 3320-3436. PMLR, 2020.
Jean-Jacques E Slotine, Weiping Li, et al. Applied nonlinear control, volume 199. Prentice hall Englewood Cliffs, NJ, 1991.
Weijie Su, Stephen Boyd, and Emmanuel Candes. A differential equation for modeling nesterov's accelerated gradient method: theory and insights. Advances in neural information processing systems, 27, 2014.
Russ Tedrake. Underactuated Robotics: Algorithms for Walking, Running, Swimming, Flying, and Manipulation (Course Notes for MIT 6.832). 2020.
Sebastian Thrun and Lorien Pratt, editors. Learning to Learn. Kluwer Academic Publishers, USA, 1998. ISBN 0792380479.
Jun-Kun Wang, Chi-Heng Lin, and Jacob D Abernethy. A modular analysis of provable acceleration via polyak's momentum: Training a wide relu network and a deep linear network. In International Conference on Machine Learning, pages 10816-10827. PMLR, 2021a.
Xiang Wang, Shuai Yuan, Chenwei Wu, and Rong Ge. Guarantees for tuning the step size using a learning-to-learn approach. In International Conference on Machine Learning, pages 10981-10990. PMLR, 2021b.
Olga Wichrowska, Niru Maheswaranathan, Matthew W. Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3751-3760. PMLR, 06-11 Aug 2017.
Ashia Wilson. Lyapunov arguments in optimization. University of California, Berkeley, 2018.