SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pengfei Li
University of California Riverside
Riverside, CA, USA
pli081@ucr.edu
jyang239@ucr.edu
We study a challenging form of Smoothed Online Convex Optimization, a.k.a. SOCO, including multi-step nonlinear switching costs and feedback delay. We propose a novel machine learning (ML) augmented online algorithm, Robustness-Constrained Learning (RCL), which combines untrusted ML predictions with a trusted expert online algorithm via constrained projection to robustify the ML prediction. Specifically, we prove that RCL is able to guarantee  $(1 + \lambda)$ -competitiveness against any given expert for any  $\lambda > 0$ , while also explicitly training the ML model in a robustification-aware manner to improve the average-case performance. Importantly, RCL is the first ML-augmented algorithm with a provable robustness guarantee in the case of multi-step switching cost and feedback delay. We demonstrate the improvement of RCL in both robustness and average performance using battery management for electrifying transportation as a case study.
This paper studies Smoothed Online Convex Optimization (SOCO), a model that has seen application in a wide variety of settings. The goal of SOCO is to minimize the sum of a per-round hitting cost and a switching cost that penalizes temporal changes in actions. The added (even single-step) switching cost creates substantial algorithmic challenges, and has received more than a decade of attention (see [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] and the references therein). While there have been various competitive online algorithms, e.g., ROBD, to guarantee the worst-case performance robustness for SOCO [13, 5, 2, 6, 8, 13], their average performance is typically far from optimal due to the conservativeness needed to address potentially adversarial instances. In contrast, machine learning (ML) based optimizers can improve the average performance by exploiting rich historical data and statistical information [14, 1, 12, 7, 15], but they sacrifice the strong robustness in terms of provable competitive bounds needed by safety-critical applications, especially when there is a distributional shift [16, 17], the ML model capacity is limited, and/or inputs are adversarial [18, 19].
More recently, ML-augmented online algorithms have emerged as potential game changers in classic online problems such as ski rental and caching systems [20, 21, 22, 23, 24]. The goal is to obtain the best of both worlds by utilizing good ML predictions to improve the average performance while ensuring bounded competitive ratios even when ML predictions are arbitrarily bad. In the context of SOCO, there has been initial progress on ML-augmented algorithms in the past year [1, 14, 7, 25]. However, these studies target the simplest case of SOCO where there is no feedback delay and the
switching costs are linear. Crucially, their specific designs make it difficult, if not impossible, to apply to more general and practical settings where there is hitting cost feedback delay and multi-step nonlinear memory in the switching cost. In addition, with a few exceptions [12, 25], a common assumption in the existing SOCO studies is that the ML model is pre-trained as a black box without awareness of the downstream operation, which creates a mismatch between training and testing and degrades the average performance.
Even without ML predictions, addressing the hitting cost feedback delay and multi-step nonlinear memory is already challenging, as the agent must make decisions semi-blindly without receiving the immediate hitting cost feedback and the decision at each step can affect multiple future decisions in a complex manner due to multi-step nonlinear switching costs [6, 26]. Incorporating ML predictions into the decision process adds substantial challenges, requiring novel algorithmic techniques beyond those used in the simple SOCO setting with single-step memory and no feedback delay [1, 14, 7].
Contributions. We propose a novel ML-augmented algorithm, called Robustness-Constrained Learning (RCL) that, for the first time, provides both robustness guarantees and good average performance for SOCO in general settings with hitting cost feedback delay and multi-step nonlinear memory in the switching cost. The foundation of RCL is to utilize an existing online algorithm (referred to as expert) as well as a novel reservation cost to hedge against future risks while closely following the ML predictions. Without having the immediate hitting cost feedback, RCL robustifies untrusted ML predictions at each step by judiciously accounting for the hitting cost uncertainties and the non-linear impact of the current decision on future switching costs. Importantly, by design, the resulting cost of RCL is no greater than  $(1 + \lambda)$  times the expert's cost for any  $\lambda > 0$  and any problem instance, while a larger  $\lambda$  allows RCL to better explore the potential of good ML predictions.
Our main technical results provide bounds on both the worst-case and average-case performance of RCL. In particular, we prove a novel worst-case cost bound on RCL in Theorem 4.1 and a bound on the average-case performance in Theorem 4.2. Our cost bound is proven by utilizing a new reservation cost as the core of RCL. The form of the reservation cost allows us to develop a new proof approach (potentially of independent interest) that decouples the dependency of the online action on the history and constructs a new sufficient robustness constraint to bound the distance between the actions and ML predictions. Importantly, this approach enables us to account for multi-step non-linear memory in the switching cost and arbitrarily delayed hitting cost feedback in SOCO, which cannot be addressed by the existing algorithms [1, 14, 7, 12, 25]. We also provide a first-of-its-kind condition for simultaneously achieving both finite robustness and 1-consistency, which has been shown to be impossible in general [14]. Finally, we evaluate the performance of RCL using a case study focused on battery management in electric vehicle (EV) charging stations. Our results highlight the advantage of RCL in terms of robustness guarantees compared to pure ML-based methods, as well as the benefit of training a robustification-aware ML model.
In summary, our work makes significant contributions to the growing SOCO literature. First, we propose a novel ML-augmented algorithm that provides the first worst-case cost bounds in a general SOCO setting with hitting cost feedback delay and multi-step non-linear switching costs. Our algorithm design and analysis (Theorem 4.1) are new and significantly differ from those used in simple SOCO settings [1, 14, 7, 12, 25]. Second, we provide a first sufficient condition under which finite robustness and 1-consistency are simultaneously achieved (Corollary 4.1.1). Finally, we introduce and analyze the first algorithm that allows robustification-aware training, highlighting its advantage over the commonly-assumed robustification-oblivious training in terms of the average cost.
SOCO has been actively studied for over a decade under a wide variety of settings [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]. For example, designed based on classic algorithmic frameworks, expert online algorithms include online gradient descent (OGD) [27], online balanced descent (OBD) [9], regularized OBD (ROBD) [13, 5], among many others. These algorithms are judiciously designed to have bounded competitive ratios and/or regrets, but they may not perform well on typical instances due to the conservative choices necessary to optimize the worst-case performance. Assuming the knowledge of (possibly imperfect) future inputs, algorithms include standard receding horizon control (RHC) [10] committed horizon control (CHC) [28], and receding horizon gradient descent (RHGD) [29, 4]. Nonetheless, the worst-case performance is still unbounded when the inputs have large errors.
By tapping into historical data, pure ML-based online optimizers, e.g., recurrent neural networks, have been studied for online problems [30, 31, 32]. Nonetheless, even with (heuristic) techniques such as distributionally robust training and/or addition of hard training instances (e.g., adversarial samples) [17, 16], they cannot provide formal worst-case guarantees as their expert counterparts.
By combining potentially untrusted ML predictions with robust experts, ML-augmented algorithms have emerged as a promising approach [22, 24, 33, 20]. The existing ML-augmented algorithms for SOCO [25, 1, 14, 7, 15, 12] only focus on simple SOCO settings where the hitting cost is known without delays and the switching cost is linear. Extending these algorithms [25, 1, 14, 7, 15, 12] to the general SOCO setting requires substantially new designs and analysis. For example, [25] utilizes the simple triangle inequality for linear switching costs in the metric space to achieve robustness, whereas this inequality does not hold given (multi-step) non-linear memory in terms of squared switching costs [14] even when there is no feedback delay. In fact, even without considering ML predictions, the general SOCO setting with feedback delays and multi-step non-linear switching costs presents significant challenges that need new techniques [26, 6] beyond those for the simple SOCO setting. Thus, RCL makes novel contributions to the growing ML-augmented SOCO literature. Customizing ML to better suit the downstream operation to achieve a lower cost has been considered for a few online problems [34, 35, 25]. In RCL, however, we need implicit differentiation through time to customize ML by considering our novel algorithm designs in our general SOCO setting.
In online learning with expert predictions [36, 37, 38], experts are dynamically chosen with time-varying probabilities to achieve a low regret compared to the best expert in hindsight. By contrast, RCL considers a different problem setting with feedback delay and multi-step non-linear memory, and focuses on constrained learning by bounding the total cost below  $(1 + \lambda)$  times of the expert's cost for any instance and any  $\lambda > 0$ . Finally, RCL is also broadly relevant to conservative bandits and reinforcement learning [39]. Specifically, conservative exploration focuses on unknown cost functions (and, when applicable, transition models) and uses a baseline policy to guide the exploration process. But, its design is fundamentally different in that it does not hedge against future uncertainties when choosing an action for each step. Additionally, constrained policy optimization [40, 41] focuses on constraining the average cost, whereas RCL focuses on the worst-case cost constraint.
3 Model and Preliminaries
In a SOCO problem, an agent, a.k.a., decision maker, must select an irrevocable action  $x_{t}$  from an action space  $\mathcal{X} \subseteq \mathbb{R}^{n}$  with size  $|\mathcal{X}|$  at each of time  $t = 1,\dots,T$ . Given the selected action, the agent incurs the sum of (i) a non-negative hitting cost  $f(x_{t},y_{t}) \geq 0$  parameterized by the context  $y_{t} \in \mathcal{Y} \subseteq \mathbb{R}^{m}$ , where  $f(\cdot):\mathbb{R}^n\to \mathbb{R}_{\geq 0}$ , and (ii) a non-negative switching cost  $d(x_{t},x_{t - p:t - 1}) = \frac{1}{2}\| x_t - \delta (x_{t - p:t - 1})\|^2$ , where the constant  $\frac{1}{2}$  is added for the convenience of derivation,  $\| \cdot \|$  is the  $l_{2}$  norm by default, and  $\delta (\cdot):\mathbb{R}^{p\times n}\rightarrow \mathbb{R}^n$  is a (possibly non-linear) function of  $x_{t - p:t - 1} = (x_{t - p},\dots ,x_{t - 1})$ . We make the following standard assumptions.
Assumption 1. At each  $t$ , the hitting cost  $f(x_{t},y_{t})$  is non-negative,  $\alpha_{h}$ -strongly convex, and  $\beta_{h}$ -smooth in  $x_{t} \in \mathcal{X}$ . It is also Lipschitz continuous with respect to  $y_{t} \in \mathcal{Y}$ .
Assumption 2. In the switching cost  $d(x_{t}, x_{t - p:t - 1}) = \frac{1}{2} \| x_{t} - \delta(x_{t - p:t - 1})\|^{2}$ , the function  $\delta(x_{t - p:t - 1})$  is  $L_{i}$ -Lipschitz continuous in  $x_{t - i}$  for  $i = 1, \dots, p$ , i.e., for any  $x_{t - i}, x_{t - i}' \in \mathcal{X}$ , we have  $\| \delta(x_{t - p}, \dots, x_{t - i}, \dots, x_{t - 1}) - \delta(x_{t - p}, \dots, x_{t - i}', \dots, x_{t - 1}) \| \leq L_{i} \| x_{t - i} - x_{t - i}' \|$ .
The convexity of the hitting cost is standard in the literature and needed for competitive analysis, while smoothness (i.e., Lipschitz-continuous gradients) guarantees that bounded action differences also result in bounded cost differences [26]. A common example of the hitting cost is  $f(x_{t},y_{t}) = \| x_{t} - y_{t}\|^{2}$  as motivated by object tracking applications, where  $y_{t}$  is the online moving target [6, 26]. In the switching cost term, the previous  $p$ -step actions  $x_{t - p:t - 1}$  are essentially encoded by  $\delta (x_{t - p:t - 1})$  [6]. Let us take drone tracking as an example. The switching cost can be written as  $d(x_{t},x_{t - 1}) = \frac{1}{2}\| x_t - x_{t - 1} + C_1 + C_2\cdot |x_{t - 1}|\cdot x_{t - 1}\| ^2$  and hence  $\delta (x_{t - 1}) = x_{t - 1} - C_1 - C_2\cdot |x_{t - 1}|\cdot x_{t - 1}$  is nonlinear, where  $x_{t}$  is the drone's speed at time  $t$  and the constants of  $C_1$  and  $C_2$  account for gravity and the aerodynamic drag [42]. For additional examples of switching costs in other applications, readers are further referred to [26].
For the convenience of presentation, we use  $y = (y_{1},\dots ,y_{T})\in \mathcal{V}^{T}$  to denote a problem instance, while noting that the initial actions  $x_{-p + 1:0} = (x_{-p + 1},\dots ,x_0)$  are also provided as an additional
input. Further, for  $1 \leq t_1 \leq t_2 \leq T$ , we also rewrite  $\sum_{\tau = t_1}^{t_2} f(x_\tau, y_\tau) + d(x_\tau, x_{\tau - p: \tau - 1})$  as  $\mathrm{cost}(x_{t_1:t_2})$ , where we suppress the context  $y_t$  without ambiguity.
For online optimization, the key challenge is that the switching cost couples online actions and the hitting costs are revealed online. As in the recent literature on SOCO [26], we assume that the agent knows the switching cost, because it is determined endogenously by the problem itself and the agent's previous actions. The agent also knows the smoothness constant  $\beta_{h}$ , although the hitting cost function itself is revealed online subject to a delay as defined below.
3.1 Feedback Delay
There may be feedback delay that prevents immediate observation of the context  $y_{t}$  (which is equivalent to delayed hitting cost function) [43, 6]. For example, in assisted drone landing, the context parameter  $y_{t}$  can represent the target velocity at time  $t$  sent to the drone by a control center, but the communications between the drone and control center can experience delays due to wireless channels and/or even packet losses due to adversarial jamming [26, 42].
To model the delay, we refer to  $q \geq 0$  as the maximum feedback delay (i.e., context  $y_{t}$  can be delayed for up to  $q \geq 0$  steps), and define  $q$ -delayed time set of arrival contexts.
Definition 1 (q-delayed time set of arrival contexts). Given the maximum feedback delay of  $q \geq 0$ , for each time  $t = 1, \dots, T$ , the q-delayed time set of arrival contexts contains the time indexes whose contexts are newly revealed to the agent at time  $t$  and is defined as  $\mathcal{D}_t^q \subseteq \{\tau \in \mathbb{N} | t - q \leq \tau \leq t\}$  such that  $\{\tau \in \mathbb{N} | \tau \leq t - q\} \subseteq (\mathcal{D}_1^q \bigcup \dots \bigcup \mathcal{D}_t^q)$ .
Naturally, given the maximum delay of  $q \geq 0$ , we must have  $\{\tau \in \mathbb{N} | \tau \leq t - q\} \subseteq (\mathcal{D}_1^q \bigcup \dots \bigcup \mathcal{D}_t^q)$ , i.e., at time  $t$ , the agent must have already known the contexts  $y_{\tau}$  for any  $\tau = 1, \dots, t - q$ .
It is worth highlighting that our definition of  $\mathcal{D}_t^q$  is flexible and applies to various delay models. Specifically, the no-delay setting corresponds to  $q = 0$  and  $D_t^{q=0} = \{t\}$ , while  $q = T$  captures the case in which the agent may potentially have to choose actions without knowing any of the contexts  $y_1, \dots, y_T$  throughout an entire problem instance. Given the maximum delay  $q \geq 0$ , the delayed contexts can be revealed to the agent in various orders different from their actual time steps, i.e., the agent may receive  $y_{t_1}$  earlier than  $y_{t_2}$  for  $t_1 > t_2$ . Also, the agent can receive a batch of contexts  $y_{t - q}, \dots, y_t$  simultaneously at time  $t$ , and receive no new contexts some other time steps.
In online optimization, handling delayed cost functions, even for a single step, is challenging [6, 44, 26]. Adding ML predictions into online optimization creates further algorithmic difficulties.
3.2 Performance Metrics
Our goal is to minimize the sum of the total hitting costs and switching costs over  $T$  time steps:  $\min_{x_1,\dots x_T}\sum_{t = 1}^T f(x_t,y_t) + d(x_t,x_{t - p:t - 1})$ . We formally define our two measures of interest.
Definition 2 (Competitiveness). An algorithm  $ALG_{1}$  is said to be CR-competitive against another baseline algorithm  $ALG_{2}$  if  $cost(ALG_{1}, y) \leq CR \cdot cost(ALG_{2}, y)$  is satisfied for any problem instance  $y \in \mathcal{Y}^{T}$ , where  $cost(ALG_{1}, y)$  and  $cost(ALG_{2}, y)$  denote the total costs of  $ALG_{1}$  and  $ALG_{2}$ , respectively.
Definition 3 (Average cost). The average cost of an algorithm  $ALG$  is  $\overline{cost}(ALG) = \mathbb{E}_{y \in \mathbb{P}_y}[\text{cost}(ALG, y)]$ , where  $cost(ALG, y)$  denotes the cost of  $ALG$  for a problem instance  $y$ , and  $\mathbb{P}_y$  is the exogenous probability distribution of  $y = (y_1, \dots, y_T) \in \mathcal{Y}^T$ .
Our definition of competitiveness against a general baseline algorithm is commonly considered in the literature, e.g., [7]. The two metrics measure an online algorithm's robustness in the worst case and expected performance in the average case, which are both important in practice.
We consider an expert (online) algorithm  $\pi$  which chooses  $x_{t}^{\pi}$  at time  $t$  and an ML model  $h_W$  which, parameterized by  $W$ , produces  $\tilde{x}_t = h_W$  ( $\tilde{x}_{t - p:t - 1},\{y_\tau |\tau \in \mathcal{D}_t^q\})$  at time  $t$ . As in the existing ML-augmented online algorithms [7, 14], RCL chooses an actual online action  $x_{t}$  by using the two actions  $x_{t}^{\pi}$  and  $\tilde{x}_t$  as advice. In general, it is extremely challenging, if not impossible, to simultaneously optimize for both the average cost and the competitiveness. Here, given a robustness requirement  $\lambda >0$ , we focus on minimizing the average cost while ensuring  $(1 + \lambda)$ -competitiveness against
the expert  $\pi$ . Crucially, the optimal expert for our setting is iROBD, which has the best-known competitiveness against the offline optimal  $OPT$  with complete information [26]. Thus, by using iROBD as the expert  $\pi$ ,  $(1 + \lambda)$ -competitiveness of RCL against  $\pi$  immediately translates into a scaled competitive ratio of  $(1 + \lambda) \cdot CR_{iROBD}$  against  $OPT$ , where  $CR_{iROBD}$  is iROBD's competitive ratio against  $OPT$ .
4 RCL: The Design and Analysis
In this section, we present RCL, a novel ML-augmented that combines ML predictions (i.e., online actions produced by an ML model [1, 45, 12, 15]) with a robust expert online algorithm to the worst-case cost while leveraging the benefit of ML predictions for average performance.
4.1 Robustness-Constrained Online Algorithm
Our goal is to "robustify" ML predictions, by which we mean that we want to ensure a robustness bound on the cost of no greater than  $(1 + \lambda)$  times of the expert's cost, i.e., for any problem instance  $y$ , we have  $\mathrm{cost}(x_{1:T}) \leq (1 + \lambda) \mathrm{cost}(x_{1:T}^{\pi})$ , where  $\lambda > 0$  is a hyperparameter indicating the level of robustness we would like to achieve. Meanwhile, we would like to utilize the benefits of ML predictions to improve the average performance.
Because of the potential feedback delays, RCL needs to choose an online action  $x_{t}$  without necessarily knowing the hitting costs of the expert's action  $x_{t}^{\pi}$  and ML prediction  $\tilde{x}_{t}$ . Additionally, the action  $x_{t}$  can affect multiple future switching costs due to multi-step non-linear memory. Thus, it is very challenging to robustify ML predictions for the SOCO settings we consider. A simple approach one might consider is to constrain  $x_{t}$  such that the cumulative cost up to each time  $t$  is always no greater than  $(1 + \lambda)$  times of the expert's cumulative cost, i.e.,  $\mathrm{cost}(x_{1:t}) \leq (1 + \lambda) \mathrm{cost}(x_{1:t}^{\pi})$ . However, even without feedback delays, such an approach may not even produce feasible actions for some  $t = 1, \dots, T$ . We explain this by considering a single-step switching cost case. Suppose that  $\mathrm{cost}(x_{1:t}) \leq (1 + \lambda) \mathrm{cost}(x_{1:t}^{\pi})$  is satisfied at a time  $t < T$ , and we choose an action  $x_{t} \neq x_{t}^{\pi}$  different from the expert. Then, at time  $t + 1$ , let us consider a case in which the expert algorithm has such a low cost that even choosing  $x_{t + 1} = x_{t + 1}^{\pi}$  will result in  $\mathrm{cost}(x_{1:t}) + f(x_{t + 1}^{\pi}, y_{t + 1}) + d(x_{t + 1}^{\pi}, x_{t}) > (1 + \lambda) [\mathrm{cost}(x_{1:t}^{\pi}) + f(x_{t + 1}^{\pi}, y_{t + 1}) + d(x_{t + 1}^{\pi}, x_{t}^{\pi})]$ . This is because the actual switching cost  $d(x_{t + 1}^{\pi}, x_{t})$  can be significantly higher than the expert's switching cost  $d(x_{t + 1}^{\pi}, x_{t}^{\pi})$  due to  $x_{t} \neq x_{t}^{\pi}$ . As a result, at time  $t + 1$ , it is possible that there exist no feasible actions that satisfy  $\mathrm{cost}(x_{1:t + 1}) \leq (1 + \lambda) \mathrm{cost}(x_{1:t + 1}^{\pi})$ . Moreover, when choosing an online action close to the ML prediction, extra caution must be exercised as the hitting costs can be revealed with delays of up to  $q$  steps.
To address these challenges, RCL introduces novel reservation costs to hedge against any possible uncertainties due to hitting cost feedback delays and multi-step non-linear memory in the switching costs. Concretely, given both the expert action  $x_{t}^{\pi}$  and ML prediction  $\tilde{x}_{t}$  at time  $t$ , we choose  $x_{t} = \arg \min_{x\in \mathcal{X}_{t}}\frac{1}{2}\| x - \tilde{x}_{t}\|^{2}$  by solving a constrained convex problem to project the ML prediction  $\tilde{x}_{t}$  into a robustified action set  $x\in \mathcal{X}_t$  where  $x_{t}$  satisfies:
$$
\begin{array}{l} \sum_ {\tau \in \mathcal {A} _ {t}} f (x _ {\tau}, y _ {\tau}) + \sum_ {\tau = 1} ^ {t} d (x _ {\tau}, x _ {\tau - p: \tau - 1}) + \sum_ {\tau \in \mathcal {B} _ {t}} H (x _ {\tau}, x _ {\tau} ^ {\pi}) + G (x _ {t}, x _ {t - p: t - 1}, x _ {t - p: t} ^ {\pi}) \\ \leq (1 + \lambda) \left(\sum_ {\tau \in \mathcal {A} _ {t}} f \left(x _ {\tau} ^ {\pi}, y _ {\tau}\right) + \sum_ {\tau = 1} ^ {t} d \left(x _ {\tau} ^ {\pi}, x _ {\tau - p: \tau - 1} ^ {\pi}\right)\right), \tag {1} \\ \end{array}
$$
in which  $\lambda > 0$  is the robustness hyperparameter,  $\mathcal{A}_t^q = (\mathcal{D}_1^q \bigcup \dots \bigcup \mathcal{D}_t^q)$  and  $\mathcal{B}_t^q = \{1, \dots, t\} \setminus \mathcal{A}_t$  are the sets of time indexes for which the agent knows and does not know the context parameters up to time  $t$ , respectively. Most importantly, the two novel reservation costs  $H(x_{\tau}, x_{\tau}^{\pi})$  and  $G(x_{t}, x_{t - q:t - 1}, x_{t - q:t}^{\pi})$  are defined as
$$
\begin{array}{l} H \left(x _ {\tau}, x _ {\tau} ^ {\pi}\right) = \frac {\beta_ {h}}{2} \left(1 + \frac {1}{\lambda_ {0}}\right) \| x _ {\tau} - x _ {\tau} ^ {\pi} \| ^ {2}, (2) \\ G \left(x _ {t}, x _ {t - p: t - 1}, x _ {t - p: t} ^ {\pi}\right) = \frac {\left(1 + \frac {1}{\lambda_ {0}}\right) \left(1 + \sum_ {k = 1} ^ {p} L _ {k}\right)}{2} \sum_ {k = 1} ^ {\tilde {p}} \left(L _ {k} \| x _ {t} - x _ {t} ^ {\pi} \| ^ {2} + \sum_ {i = 1} ^ {p - k} L _ {k + i} \| x _ {t - i} - x _ {t - i} ^ {\pi} \| ^ {2}\right), (3) \\ \end{array}
$$