Title: Robust Learning for Smoothed Online Convex Optimization with Feedback Delay
Venue: NeurIPS 2023
Accepted: yes
Keywords: Online optimization

competitive algorithm

switching cost
Abstract: We study a general form of Smoothed Online Convex Optimization, a.k.a. SOCO, including multi-step switching costs and feedback delay. We propose a novel machine learning (ML) augmented online algorithm, Robustness-Constrained Learning (RCL), which combines untrusted ML predictions with a trusted expert online algorithm via constrained projection to robustify the ML prediction. Specifically, we prove that RCL is able to guarantee $(1+\lambda)$-competitiveness against any given expert for any $\lambda>0$, while also explicitly training the ML model in a robustification-aware manner to improve the average-case performance. Importantly, RCL is the first ML-augmented algorithm with a provable robustness guarantee in the case of multi-step switching cost and feedback delay. We demonstrate the improvement of RCL in both robustness and average performance using battery management as a case study.

Robust Learning for Smoothed Online Convex Optimization with Feedback Delay

Abstract

We study a challenging form of Smoothed Online Convex Optimization, a.k.a. SOCO, including multi-step nonlinear switching costs and feedback delay. We propose a novel machine learning (ML) augmented online algorithm, RobustnessConstrained Learning ( RCL ), which combines untrusted ML predictions with a trusted expert online algorithm via constrained projection to robustify the ML prediction. Specifically, we prove that RCL is able to guarantee (1+ λ ) -competitiveness against any given expert for any λ > 0 , while also explicitly training the ML model in a robustification-aware manner to improve the average-case performance. Importantly, RCL is the first ML-augmented algorithm with a provable robustness guarantee in the case of multi-step switching cost and feedback delay. We demonstrate the improvement of RCL in both robustness and average performance using battery management for electrifying transportation as a case study.

1 Introduction

This paper studies Smoothed Online Convex Optimization (SOCO) , a model that has seen application in a wide variety of settings. The goal of SOCO is to minimize the sum of a per-round hitting cost and a switching cost that penalizes temporal changes in actions. The added (even single-step) switching cost creates substantial algorithmic challenges, and has received more than a decade of attention (see [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] and the references therein). While there have been various competitive online algorithms, e.g., ROBD, to guarantee the worst-case performance robustness for SOCO [13, 5, 2, 6, 8, 13], their average performance is typically far from optimal due to the conservativeness needed to address potentially adversarial instances. In contrast, machine learning (ML) based optimizers can improve the average performance by exploiting rich historical data and statistical information [14, 1, 12, 7, 15], but they sacrifice the strong robustness in terms of provable competitive bounds needed by safety-critical applications, especially when there is a distributional shift [16, 17], the ML model capacity is limited, and/or inputs are adversarial [18, 19].

More recently, ML-augmented online algorithms have emerged as potential game changers in classic online problems such as ski rental and caching systems [20, 21, 22, 23, 24]. The goal is to obtain the best of both worlds by utilizing good ML predictions to improve the average performance while ensuring bounded competitive ratios even when ML predictions are arbitrarily bad. In the context of SOCO, there has been initial progress on ML-augmented algorithms in the past year [1, 14, 7, 25]. However, these studies target the simplest case of SOCO where there is no feedback delay and the

switching costs are linear. Crucially, their specific designs make it difficult, if not impossible, to apply to more general and practical settings where there is hitting cost feedback delay and multi-step nonlinear memory in the switching cost. In addition, with a few exceptions [12, 25], a common assumption in the existing SOCO studies is that the ML model is pre-trained as a black box without awareness of the downstream operation, which creates a mismatch between training and testing and degrades the average performance.

Even without ML predictions, addressing the hitting cost feedback delay and multi-step nonlinear memory is already challenging, as the agent must make decisions semiblindly without receiving the immediate hitting cost feedback and the decision at each step can affect multiple future decisions in a complex manner due to multi-step nonlinear switching costs [6, 26]. Incorporating ML predictions into the decision process adds substantial challenges, requiring novel algorithmic techniques beyond those used in the simple SOCO setting with single-step memory and no feedback delay [1, 14, 7].

Contributions. We propose a novel ML-augmented algorithm, called Robustness-Constrained Learning ( RCL ) that, for the first time, provides both robustness guarantees and good average performance for SOCO in general settings with hitting cost feedback delay and multi-step nonlinear memory in the switching cost. The foundation of RCL is to utilize an existing online algorithm (referred to as expert) as well as a novel reservation cost to hedge against future risks while closely following the ML predictions. Without having the immediate hitting cost feedback, RCL robustifies untrusted ML predictions at each step by judiciously accounting for the hitting cost uncertainties and the non-linear impact of the current decision on future switching costs. Importantly, by design, the resulting cost of RCL is no greater than (1 + λ ) times the expert's cost for any λ > 0 and any problem instance, while a larger λ allows RCL to better explore the potential of good ML predictions.

Our main technical results provide bounds on both the worst-case and average-case performance of RCL . In particular, we prove a novel worst-case cost bound on RCL in Theorem 4.1 and a bound on the average-case performance in Theorem 4.2. Our cost bound is proven by utilizing a new reservation cost as the core of RCL . The form of the reservation cost allows us to develop a new proof approach (potentially of independent interest) that decouples the dependency of the online action on the history and constructs a new sufficient robustness constraint to bound the distance between the actions and ML predictions. Importantly, this approach enables us to account for multi-step non-linear memory in the switching cost and arbitrarily delayed hitting cost feedback in SOCO, which cannot be addressed by the existing algorithms [1, 14, 7, 12, 25]. We also provide a first-of-its-kind condition for simultaneously achieving both finite robustness and 1-consistency, which has been shown to be impossible in general [14]. Finally, we evaluate the performance of RCL using a case study focused on battery management in electric vehicle (EV) charging stations. Our results highlight the advantage of RCL in terms of robustness guarantees compared to pure ML-based methods, as well as the benefit of training a robustification-aware ML model.

In summary, our work makes significant contributions to the growing SOCO literature. First, we propose a novel ML-augmented algorithm that provides the first worst-case cost bounds in a general SOCO setting with hitting cost feedback delay and multi-step non-linear switching costs. Our algorithm design and analysis (Theorem 4.1) are new and significantly differ from those used in simple SOCO settings [1, 14, 7, 12, 25]. Second, we provide a first sufficient condition under which finite robustness and 1-consistency are simultaneously achieved (Corollary 4.1.1). Finally, we introduce and analyze the first algorithm that allows robustification-aware training, highlighting its advantage over the commonly-assumed robustification-oblivious training in terms of the average cost.

2 Related Work

SOCO has been actively studied for over a decade under a wide variety of settings [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]. For example, designed based on classic algorithmic frameworks, expert online algorithms include online gradient descent (OGD) [27], online balanced descent (OBD) [9], regularized OBD (ROBD) [13, 5], among many others. These algorithms are judiciously designed to have bounded competitive ratios and/or regrets, but they may not perform well on typical instances due to the conservative choices necessary to optimize the worst-case performance. Assuming the knowledge of (possibly imperfect) future inputs, algorithms include standard receding horizon control (RHC) [10] committed horizon control (CHC) [28], and receding horizon gradient descent (RHGD) [29, 4]. Nonetheless, the worst-case performance is still unbounded when the inputs have large errors.

By tapping into historical data, pure ML-based online optimizers, e.g., recurrent neural networks, have been studied for online problems [30, 31, 32]. Nonetheless, even with (heuristic) techniques such as distributionally robust training and/or addition of hard training instances (e.g., adversarial samples) [17, 16], they cannot provide formal worst-case guarantees as their expert counterparts.

By combining potentially untrusted ML predictions with robust experts, ML-augmented algorithms have emerged as a promising approach [22, 24, 33, 20]. The existing ML-augmented algorithms for SOCO [25, 1, 14, 7, 15, 12] only focus on simple SOCO settings where the hitting cost is known without delays and the switching cost is linear. Extending these algorithms [25, 1, 14, 7, 15, 12] to the general SOCO setting requires substantially new designs and analysis. For example, [25] utilizes the simple triangle inequality for linear switching costs in the metric space to achieve robustness, whereas this inequality does not hold given (multi-step) non-linear memory in terms of squared switching costs [14] even when there is no feedback delay. In fact, even without considering ML predictions, the general SOCO setting with feedback delays and multi-step non-linear switching costs presents significant challenges that need new techniques [26, 6] beyond those for the simple SOCO setting. Thus, RCL makes novel contributions to the growing ML-augmented SOCO literature. Customizing ML to better suit the downstream operation to achieve a lower cost has been considered for a few online problems [34, 35, 25]. In RCL , however, we need implicit differentiation through time to customize ML by considering our novel algorithm designs in our general SOCO setting.

In online learning with expert predictions [36, 37, 38], experts are dynamically chosen with timevarying probabilities to achieve a low regret compared to the best expert in hindsight. By contrast, RCL considers a different problem setting with feedback delay and multi-step non-linear memory, and focuses on constrained learning by bounding the total cost below (1 + λ ) times of the expert's cost for any instance and any λ > 0 . Finally, RCL is also broadly relevant to conservative bandits and reinforcement learning [39]. Specifically, conservative exploration focuses on unknown cost functions (and, when applicable, transition models) and uses a baseline policy to guide the exploration process. But, its design is fundamentally different in that it does not hedge against future uncertainties when choosing an action for each step. Additionally, constrained policy optimization [40, 41] focuses on constraining the average cost, whereas RCL focuses on the worst-case cost constraint.

3 Model and Preliminaries

In a SOCO problem, an agent, a.k.a., decision maker, must select an irrevocable action x t from an action space X ⊆ R n with size |X| at each of time t = 1 , . . . , T . Given the selected action, the agent incurs the sum of (i) a non-negative hitting cost f ( x t , y t ) ≥ 0 parameterized by the context y t ∈ Y ⊆ R m , where f ( · ) : R n → R ≥ 0 , and (ii) a non-negative switching cost d ( x t , x t -p : t -1 ) = 1 2 ∥ x t -δ ( x t -p : t -1 ) ∥ 2 , where the constant 1 2 is added for the convenience of derivation, ∥ · ∥ is the l 2 norm by default, and δ ( · ) : R p × n → R n is a (possibly non-linear) function of x t -p : t -1 = ( x t -p , · · · , x t -1 ) . We make the following standard assumptions.

Assumption 1. At each t , the hitting cost f ( x t , y t ) is non-negative, α h -strongly convex, and β h -smooth in x t ∈ X . It is also Lipschitz continuous with respect to y t ∈ Y .

Assumption 2. In the switching cost d ( x t , x t -p : t -1 ) = 1 2 ∥ x t -δ ( x t -p : t -1 ) ∥ 2 , the function δ ( x t -p : t -1 ) is L i -Lipschitz continuous in x t -i for i = 1 · · · p , i.e., for any x t -i , x ′ t -i ∈ X , we have ∥ δ ( x t -p , · · · , x t -i , · · · x t -1 ) -δ ( x t -p , · · · , x ′ t -i , · · · x t -1 ) ∥ ≤ L i ∥ x t -i -x ′ t -i ∥ .

The convexity of the hitting cost is standard in the literature and needed for competitive analysis, while smoothness (i.e., Lipschitz-continuous gradients) guarantees that bounded action differences also result in bounded cost differences [26]. A common example of the hitting cost is f ( x t , y t ) = ∥ x t -y t ∥ 2 as motivated by object tracking applications, where y t is the online moving target [6, 26]. In the switching cost term, the previous p -step actions x t -p : t -1 are essentially encoded by δ ( x t -p : t -1 ) [6]. Let us take drone tracking as an example. The switching cost can be written as d ( x t , x t -1 ) = 1 2 ∥ x t -x t -1 + C 1 + C 2 · | x t -1 | · x t -1 ∥ 2 and hence δ ( x t -1 ) = x t -1 -C 1 -C 2 · | x t -1 | · x t -1 is nonlinear, where x t is the drone's speed at time t and the constants of C 1 and C 2 account for gravity and the aerodynamic drag [42]. For additional examples of switching costs in other applications, readers are further referred to [26].

For the convenience of presentation, we use y = ( y 1 , · · · , y T ) ∈ Y T to denote a problem instance, while noting that the initial actions x -p +1:0 = ( x -p +1 , · · · , x 0 ) are also provided as an additional

input. Further, for 1 ≤ t 1 ≤ t 2 ≤ T , we also rewrite ∑ t 2 τ = t 1 f ( x τ , y τ ) + d ( x τ , x τ -p : τ -1 ) as cost( x t 1 : t 2 ) , where we suppress the context y t without ambiguity.

For online optimization, the key challenge is that the switching cost couples online actions and the hitting costs are revealed online. As in the recent literature on SOCO [26], we assume that the agent knows the switching cost, because it is determined endogenously by the problem itself and the agent's previous actions. The agent also knows the smoothness constant β h , although the hitting cost function itself is revealed online subject to a delay as defined below.

3.1 Feedback Delay

There may be feedback delay that prevents immediate observation of the context y t (which is equivalent to delayed hitting cost function) [43, 6]. For example, in assisted drone landing, the context parameter y t can represent the target velocity at time t sent to the drone by a control center, but the communications between the drone and control center can experience delays due to wireless channels and/or even packet losses due to adversarial jamming [26, 42].

To model the delay, we refer to q ≥ 0 as the maximum feedback delay (i.e., context y t can be delayed for up to q ≥ 0 steps), and define q -delayed time set of arrival contexts.

Definition 1 ( q -delayed time set of arrival contexts) . Given the maximum feedback delay of q ≥ 0 , for each time t = 1 , . . . , T , the q -delayed time set of arrival contexts contains the time indexes whose contexts are newly revealed to the agent at time t and is defined as D q t ⊆ { τ ∈ N | t -q ≤ τ ≤ t } such that { τ ∈ N | τ ≤ t -q } ⊆ ( D q 1 ⋃ · · · ⋃ D q t ) .

Naturally, given the maximum delay of q ≥ 0 , we must have { τ ∈ N | τ ≤ t -q } ⊆ ( D q 1 ⋃ · · · ⋃ D q t ) , i.e., at time t , the agent must have already known the contexts y τ for any τ = 1 , · · · , t -q .

It is worth highlighting that our definition of D q t is flexible and applies to various delay models. Specifically, the no-delay setting corresponds to q = 0 and D q =0 t = { t } , while q = T captures the case in which the agent may potentially have to choose actions without knowing any of the contexts y 1 , · · · , y T throughout an entire problem instance. Given the maximum delay q ≥ 0 , the delayed contexts can be revealed to the agent in various orders different from their actual time steps, i.e., the agent may receive y t 1 earlier than y t 2 for t 1 > t 2 . Also, the agent can receive a batch of contexts y t -q , · · · , y t simultaneously at time t , and receive no new contexts some other time steps.

In online optimization, handling delayed cost functions, even for a single step, is challenging [6, 44, 26]. Adding ML predictions into online optimization creates further algorithmic difficulties.

3.2 Performance Metrics

Our goal is to minimize the sum of the total hitting costs and switching costs over T time steps: min x 1 , ··· x T ∑ T t =1 f ( x t , y t ) + d ( x t , x t -p : t -1 ) . We formally define our two measures of interest.

Definition 2 (Competitiveness) . An algorithm ALG 1 is said to be CR -competitive against another baseline algorithm ALG 2 if cost ( ALG 1 , y ) ≤ CR · cost ( ALG 2 , y ) is satisfied for any problem instance y ∈ Y T , where cost ( ALG 1 , y ) and cost( ALG 2 , y ) denote the total costs of ALG 1 and ALG 2 , respectively.

Definition 3 (Average cost) . The average cost of an algorithm ALG is cost ( ALG ) = E y ∈ P y [ cost ( ALG,y )] , where cost ( ALG,y ) denotes the cost of ALG for a problem instance y , and P y is the exogenous probability distribution of y = ( y 1 , · · · , y T ) ∈ Y T .

Our definition of competitiveness against a general baseline algorithm is commonly considered in the literature, e.g., [7]. The two metrics measure an online algorithm's robustness in the worst case and expected performance in the average case, which are both important in practice.

We consider an expert (online) algorithm π which chooses x π t at time t and an ML model h W which, parameterized by W , produces ˜ x t = h W (˜ x t -p : t -1 , { y τ | τ ∈ D q t } ) at time t . As in the existing MLaugmented online algorithms [7, 14], RCL chooses an actual online action x t by using the two actions x π t and ˜ x t as advice. In general, it is extremely challenging, if not impossible, to simultaneously optimize for both the average cost and the competitiveness. Here, given a robustness requirement λ > 0 , we focus on minimizing the average cost while ensuring (1 + λ ) -competitiveness against

the expert π . Crucially, the optimal expert for our setting is iROBD, which has the best-known competitiveness against the offline optimal OPT with complete information [26]. Thus, by using iROBD as the expert π , (1+ λ ) -competitiveness of RCL against π immediately translates into a scaled competitive ratio of (1 + λ ) · CR iROBD against OPT , where CR iROBD is iROBD's competitive ratio against OPT .

4 RCL : The Design and Analysis

In this section, we present RCL , a novel ML-augmented that combines ML predictions (i.e., online actions produced by an ML model [1, 45, 12, 15]) with a robust expert online algorithm to the worst-case cost while leveraging the benefit of ML predictions for average performance.

4.1 Robustness-Constrained Online Algorithm

Our goal is to 'robustify' ML predictions, by which we mean that we want to ensure a robustness bound on the cost of no greater than (1 + λ ) times of the expert's cost, i.e., for any problem instance y , we have cost( x 1: T ) ≤ (1 + λ )cost( x π 1: T ) , where λ > 0 is a hyperparameter indicating the level of robustness we would like to achieve. Meanwhile, we would like to utilize the benefits of ML predictions to improve the average performance.

̸

Because of the potential feedback delays, RCL needs to choose an online action x t without necessarily knowing the hitting costs of the expert's action x π t and ML prediction ˜ x t . Additionally, the action x t can affect multiple future switching costs due to multi-step non-linear memory. Thus, it is very challenging to robustify ML predictions for the SOCO settings we consider. A simple approach one might consider is to constrain x t such that the cumulative cost up to each time t is always no greater than (1 + λ ) times of the expert's cumulative cost, i.e., cost( x 1: t ) ≤ (1 + λ )cost( x π 1: t ) . However, even without feedback delays, such an approach may not even produce feasible actions for some t = 1 , · · · , T . We explain this by considering a single-step switching cost case. Suppose that cost( x 1: t ) ≤ (1+ λ )cost( x π 1: t ) is satisfied at a time t < T , and we choose an action x t = x π t different from the expert. Then, at time t +1 , let us consider a case in which the expert algorithm has such a low cost that even choosing x t +1 = x π t +1 will result in cost( x 1: t ) + f ( x π t +1 , y t +1 ) + d ( x π t +1 , x t ) > (1 + λ ) [ cost( x π 1: t ) + f ( x π t +1 , y t +1 ) + d ( x π t +1 , x π t ) ] . This is because the actual switching cost d ( x π t +1 , x t ) can be significantly higher than the expert's switching cost d ( x π t +1 , x π t ) due to x t = x π t . As a result, at time t + 1 , it is possible that there exist no feasible actions that satisfy cost( x 1: t +1 ) ≤ (1 + λ )cost( x π 1: t +1 ) . Moreover, when choosing an online action close to the ML prediction, extra caution must be exercised as the hitting costs can be revealed with delays of up to q steps.

̸

To address these challenges, RCL introduces novel reservation costs to hedge against any possible uncertainties due to hitting cost feedback delays and multi-step non-linear memory in the switching costs. Concretely, given both the expert action x π t and ML prediction ˜ x t at time t , we choose x t = arg min x ∈X t 1 2 ∥ x -˜ x t ∥ 2 by solving a constrained convex problem to project the ML prediction ˜ x t into a robustified action set x ∈ X t where x t satisfies:

in which λ > 0 is the robustness hyperparameter, A q t = ( D q 1 ⋃ · · · ⋃ D q t ) and B q t = { 1 , · · · , t }\A t are the sets of time indexes for which the agent knows and does not know the context parameters up to time t , respectively. Most importantly, the two novel reservation costs H ( x τ , x π τ ) and G ( x t , x t -q : t -1 , x π t -q : t ) are defined as

where β h is the smoothness constant of the hitting cost in Assumption 1, L k is the Lipschitz constant of δ ( x t -p : t -1 ) in Assumption (2), and λ 0 ∈ (0 , λ ) with the optimum being λ 0 = √ 1 + λ -1 (Theorem 4.1). The computational complexity for projection into (1) is tolerable due to convexity.

The interpretation of H ( x τ , x π τ ) and G ( x, x t -q : t -1 , x π t -q : t ) is as follows. If RCL 's action x τ deviates from the expert's action x π τ at time τ and the hitting cost is not known yet due to delayed y τ , then it is possible that RCL actually experiences a high but unknown hitting cost. In this case, to guarantee the worst-case robustness, we include an upper bound of the cost difference as the reservation cost such that f ( x τ , y τ ) -(1 + λ ) f ( x π τ , y τ ) ≤ H ( x τ , x π τ ) regardless of the delayed y τ ∈ Y . If y τ has been revealed at time t (i.e., τ ∈ A t ), then we use the actual costs instead of the reservation cost. Likewise, by considering the expert's future actions as a feasible backup plan in the worst case, the reservation cost G ( x, x t -p : t -1 , x π t -p : t ) upper bounds the maximum possible difference in the future switching costs (up to future p steps) due to deviating from the expert's action at time t .

With H ( x τ , x π τ ) and G ( x, x t -q : t -1 , x π t -q : t ) as reservation costs in (1), RCL achieves robustness by ensuring that following the expert's actions in the future is always feasible, regardless of the delayed y t . The online algorithm is described in Algorithm 1, where both the expert and ML model have the same online information { y τ | τ ∈ D q t } at time t and produce their own actions as advice to RCL .

4.2 Analysis

We now present our main results on the cost bound of RCL , showing that RCL can indeed maintain the desired (1 + λ ) -competitiveness against the expert π while exploiting the potential of ML predictions.

Theorem 4.1 (Cost bound) . Consider a memory length p ≥ 1 and the maximum feedback delay of q ≥ 0 . Given a context sequence y = ( y 1 , · · · , y T ) , let cost (˜ x 1: T ) and cost ( x π 1: T ) be the costs of pure ML predictions ˜ x 1: T and expert actions x π 1: T , respectively. For any λ > 0 , by optimally setting λ 0 = √ 1 + λ -1 , the cost of RCL is upper bounded by

where ∆( λ ) = ∑ T i =1 [ ∥ ˜ x t -x π t ∥ 2 -2( √ 1+ λ -1) 2 ( β h + α 2 ) cost π t ] + in which cost π t = ( ∑ τ ∈D q t f ( x π τ , y τ ) ) + d ( x π t , x π t -p : t -1 ) is the total of revealed hitting costs and switching cost for the expert at time t , β h is the smoothness constant of the hitting cost (Assumption 1), and α = 1 + ∑ p i =1 L i with L 1 . . . L p being the Lipschitz constants in the switching cost (Assumption 2). □

Theorem 4.1 is the first worst-case cost analysis for ML-augmented algorithms in a general SOCO setting with delayed hitting costs and multi-step switching costs. Its proof is available in the appendix and outlined here. We prove the competitiveness against the expert π based on our novel reservation cost H ( x τ , x π τ ) and G ( x t , x t -q : t -1 , x π t -q : t ) by induction. It is more challenging to prove the competitiveness against the ML prediction, because x t implicitly depends on all the previous actions of ML predictions and expert actions up to time t . To address this challenge, we utilize a novel technique by first removing the dependency of x t on the history. Then, we construct a new sufficient robustness constraint that allows an explicit expression of another robustified action, whose distance to the ML prediction ˜ x t is an upper bound by the distance between x t and ˜ x t due to the projection of ˜ x t into (1). Finally, due to the smoothness of the hitting cost function and the switching cost, the distance bound translates into the competitiveness of RCL against ML predictions.

The two terms inside the min operator in Theorem 4.1 show the tradeoff between achieving better competitiveness and more closely following ML predictions. To interpret this, we note that the first

term inside min operator shows (1+ λ ) -competitiveness of RCL against the expert π , while the second term inside min operator shows RCL can also exploit the potential of good ML predictions. A smaller λ > 0 means that we want to be closer to the expert for better competitiveness, while a larger λ > 0 decreases the term ∆( λ ) = ∑ T i =1 [ ∥ ˜ x t -x π t ∥ 2 -2( √ 1+ λ -1) 2 ( β h + α 2 ) cost π t ] + and hence makes RCL follow the ML predictions more closely.

The term ∆( λ ) in (4) essentially bounds the total squared distance between the actual online action and ML predictions. Intuitively, RCL should follow the ML predictions more aggressively when the expert does not perform well. This insight is also reflected in ∆( λ ) in Theorem 4.1. Concretely, when the expert π 's total revealed cost π t is higher, ∆( λ ) also becomes smaller, pushing RCL closer to ML. On the other hand, when the expert's cost is lower, RCL stays closer to the better-performing expert for guaranteed competitiveness.

When ∥ ˜ x t -x π t ∥ 2 is larger (i.e., greater discrepancy between the expert's action and ML prediction), it is naturally more difficult to follow both the expert and ML prediction simultaneously. Thus, given a robustness requirement of λ > 0 , we see from ∆( λ ) that a larger ∥ ˜ x t -x π t ∥ 2 also increases the second term in the min operator in Theorem 4.1, making it more difficult for RCL to exploit the potential of ML predictions. Moreover, deviating from the expert's action at one step can have impacts on the switching costs in future p steps. Thus, the memory length p creates some additional friction for RCL to achieve a low cost bound with respect to ML predictions: the greater p , the greater α = 1 + ∑ p i =1 L i , and hence the greater the second term in (4).

Robustness and consistency. It remains to show the worst-case competitiveness against the offline optimal algorithm OPT , which is typically performed for two extreme cases - when ML predictions are extremely bad and perfect - which are respectively referred to as robustness and consistency in the literature [1, 20, 22] and formally defined below.

Definition 4 (Robustness and consistency) . The robustness of RCL is CR ( ∞ ) if RCL is CR ( ∞ ) -competitive against OPT when the ML's competitiveness against OPT is arbitrarily large (denoted as ˜ CR → ∞ ) ; and the consistency of RCL is CR (1) if RCL is CR (1) -competitive against OPT when the ML's competitiveness against OPT is ˜ CR = 1 .

Robustness indicates the worst-case performance of RCL for any possible ML predictions, whereas consistency measures the capability of RCL to retain the performance of perfect ML predictions. In general, the tradeoff between robustness and consistency is unavoidable for online algorithms [20, 14, 46]. The state-of-the-art expert algorithm iROBD recently proposed in [26] has the bestknown competitive ratio under the assumption of identical delays for each context (i.e., D q t = { t -q } - each context is delayed by q steps). The identical-delay model essentially ignores any other contexts y τ for τ ∈ { τ ∈ N | t -q +1 ≤ τ ≤ t } . Thus, it is the worst case of a general q -step delay setting, whose competitive ratio is upper bounded by that of iROBD. Consequently, due to (1 + λ ) -competitiveness against any expert π in Theorem 4.1, we immediately obtain a finite robustness bound for RCL by considering iROBD as the expert.

Nonetheless, even for the simplest SOCO setting with no feedback delay and a switching cost of d ( x t , x t -1 ) = 1 2 ∥ x t -x t -1 ∥ 2 , a recent study [14] has shown that it is impossible to simultaneously achieve 1-consistency and finite robustness. Consequently, in general SOCO settings, the finite robustness of RCL given any λ > 0 means the impossibility of achieving 1-consistency by following perfect ML predictions without further assumptions.

Despite this pessimistic result due to the fundamental challenge of SOCO, we find a sufficient condition that can overcome the impossibility, which is formalized as follows.

Corollary 4.1.1 (1-consistency and finite robustness) . Consider iROBD as the exert π , whose competitive ratio against OPT is denoted as CR iROBD [26]. If the expert's switching cost always satisfies d ( x π t , x π t -p : t -1 ) ≥ ϵ > 0 for any time t = 1 , · · · , T , then by setting λ ≥ |X| 2 ( α 2 + β h ) 2 ϵ + √ 2 |X| 2 ( α 2 + β h ) ϵ ∼ O ( 1 ϵ ) and optimally using λ 0 = √ 1 + λ -1 , RCL achieves (1 + λ ) · CR iROBD -robustness and 1-consistency simultaneously. □

Corollary 4.1.1 complements the impossibility result for SOCO [14] by providing the first condition under which finite robustness and 1-consistency are simultaneously achievable. The intuition is that if the expert has a strictly positive switching cost no less than ϵ > 0 at each time, then its per-step

cost is also no less than ϵ , which provides RCL with the flexibility to choose a different action than the expert's action x π t due to the (1 + λ ) cost slackness in the competitiveness requirement. Therefore, by choosing a sufficiently large but still finite λ ∼ O ( 1 ϵ ) , we can show that ∆( λ ) = 0 in (4) in Theorem 4.1, which means RCL can completely follow the ML predictions. Without this condition, it is possible that the expert's cost is zero in the first few steps, and hence RCL must follow the expert's actions at the beginning to guarantee (1 + λ ) -competitiveness in case the expert continues to have a zero cost in the future - even though ML predictions are perfect and offline optimal, RCL cannot follow them at the beginning because of the online process and (1 + λ ) -competitiveness requirement.

Importantly, our sufficient condition is not unrealistic in practice. For example, in moving objecttracking applications, the condition d ( x π τ , x π τ -p : τ -1 ) ≥ ϵ > 0 is satisfied if the expert's action ˜ x t keeps changing to follow the moving object over time or alternatively, we ignore the dummy time steps with no movement.

4.3 MLModel Training in RCL

We present the training details and highlight the advantage of training the ML model in a robustification-aware manner to reduce the average cost.

4.3.1 Architecture, loss, and dataset

Because of the recursive nature of SOCO and the strong representation power of neural networks, we use a recurrent neural network with each base model parameterized by W ∈ W (illustrated in Fig. 1 in the appendix). Such architectures are also common in ML-based optimizers for other online problems [17, 31]. With historical data available, we can construct a training dataset S that contains a finite number of problem instances. The dataset can also be enlarged using data augmentation techniques (e.g., adding adversarial samples) [16, 17].

Robustification-oblivious. The existing literature on ML-augmented algorithms [22, 1, 45, 7] has commonly assumed that the ML model h W is separately trained in a robustification-oblivious manner without being aware of the downstream algorithm used online. Concretely, the parameter W of a robustification-oblivious ML model is optimized for the following loss

where ˜ x t = h W (˜ x t -p : t -1 , { y τ | τ ∈ D q t } ) is the ML prediction at time t .

Robustification-aware. There is a mismatch between the actual objective cost( x 1: T ) and the training objective cost(˜ x 1: T ) of a robustification-oblivious ML model. To reconcile this, we propose to train the ML model in a robustification-aware manner by explicitly considering the robustification step in Algorithm 1. For notational convenience, we denote the actual action as x t = Rob λ ( h W ) = Rob λ (˜ x t ) , which emphasizes the projection of ˜ x t into the robust action set (2). Thus, the parameter W of a robustification-aware ML model is optimized to minimize the following loss

which is different from (5) that only minimizes the cost of pre-robustification ML predictions.

4.3.2 Average cost

We bound the average cost of RCL given an ML model h W .

Theorem 4.2 (Average cost) . Assume that the ML model is trained over a dataset S drawn from the training distribution P ′ y . With probability at least 1 -δ, δ ∈ (0 , 1) , the average cost E y [cost( x 1: T )] of RCL over the testing distribution y ∼ P y is upper bounded by

where cost S ( Rob λ ( h W )) = 1 |S| ∑ S cost ( x 1: T ) is the empirical average cost of robustified ML predictions in S , Rad S ( Rob λ ( W )) defined in Definition 5 in the appendix is the Rademacher

complexity with respect to the ML model space parameterized by W with robustification on the training dataset S , the scaling coefficient inside O for Rad S ( Rob λ ( W )) is Γ x = √ T |X| [ β h + 1 2 (1 + ∑ p i =1 L i )(1 + ∑ p i =1 L i ) ] with |X| being the size of the action space X and β h , L i , and p as the smoothness constant, Lipschitz constant of the nonlinear term in the switching cost, and the memory length as defined in Assumptions 1 and 2, and the coefficient for √ log( 2 δ ) |S| is 3¯ c with ¯ c being the upper bound of the total cost for an episode.

Theorem 4.2 bounds the average cost of RCL by the minimum of two bounds. The first bound (1 + λ ) cost ( π ) further highlights the guaranteed (1 + λ ) -competitiveness of RCL with respect to the expert's average cost AV G ( π ) . The second bound includes a term cost S ( Rob λ ( h W )) = 1 |S| ∑ S cost ( x 1: T ) , which is the empirical average cost of RCL given an ML model h W and decreases when λ > 0 increases. The reason is that with a larger λ > 0 , the robust action set (1) is enlarged and RCL has more freedom to follow ML predictions due to the less stringent competitiveness constraint. Given an ML model h W , Theorem 4.1 shows how the upper bound on cost S ( Rob λ ( h W )) varies with λ . Note that h W ∗ in (5) and h ˆ W ∗ in (6) minimize cost S ( h W ∗ ) and cost S ( Rob λ ( h ˆ W ∗ )) , respectively, while the post-robustication cost (i.e., cost S ( Rob λ ( h ˆ W ∗ )) ) is the actual cost of RCL . Thus, we can further reduce the average cost by using the optimal robustification-aware ML model h ˆ W ∗ , compared to a robustification-oblivious model h W ∗ .

The other terms inside the second bound in (7) are related to the training dataset: the larger dataset, the smaller Rademacher complexity Rad S ( Rob ( W )) and √ log(1 /δ ) |S| . Note that the Rademacher complexity Rad S ( Rob ( W )) of RCL is no greater than that of using the ML model alone (i.e. Rad S ( Rob ( W )) ≤ Rad S ( W ) , shown in the appendix). The intuition is that RCL limits the action space for robustness. Thus, Theorem 4.1 provides the insight that, given λ > 0 , robustification in RCL is more valuable in terms of bounding the average cost when the ML model h W is not well trained (e.g., due to inadequate training data). In practice, the hyperparameter λ > 0 can be set to improve the empirical average performance subject to the robustness constraint based on a held-out validation dataset along with the tuning of other hyperparameters (e.g., learning rates).

4.3.3 Robustification-aware training and experimental verification

Despite the advantage in terms of the average cost, it is non-trivial to train a robustification-aware ML model using standard back-propagation. This is because the operator of projecting ML predictions into a robust set (1) is a recursive implicit layer that cannot be easily differentiated as typical neural network layers. Due to space limitations, we defer to the appendix the the differentiation of the loss function with respect to ML model weights W .

We validate the theoretical analysis by exploring the empirical performance of RCL using a case study of battery management in electric vehicle (EV) charging stations [47]. Our results are presented in the appendix. The results highlight the advantage of RCL in terms of robustness compared to pure ML models, as well as the benefit of using a robustification-aware ML model in terms of the average cost.

5 Experiments

We now explore the performance of RCL using a case study focused on battery management in electric vehicle (EV) charging stations [48]. We first formulate the problem as an instance of SOCO. More details can be found at Appendix B.1. Then, we test RCL on a public dataset provided by ElaadNL, compared with several baseline algorithms including ROBD [26], EC-L2O [12], and HitMin (details in Appendix B.2). Our results highlight the advantage of RCL in terms of robustness guarantees compared to pure ML models, as well as the benefit of training a robustification-aware ML model in terms of the average cost.

5.1 Results

We now present some results for the case in which the hitting cost function (parameterized by y t ) is immediately known without feedback delay. The results for the case with feedback delay are presented in Appendix B.4. Throughout the discussion, the reported values are normalized with

respect to those of the respective OPT. The average cost ( AVG ) and competitive ratio ( CR ) are all empirical results reported on the testing dataset.

By Theorem 4.1, there is a trade-off (governed by λ > 0 ) between exploiting ML predictions for good average performance and following the expert for robustness. Here, we focus on the default setting of λ = 1 and discuss the impact of different choices of λ on both RCL and RCL-O in Appendix B.3.2. As shown in Table 1, with λ = 1 , both RCL and RCL-O have a good average cost, but RCL has a lower average cost than RCL-O and is outperformed only by ML in terms of the average cost. RCL and RCL-O have the same competitive ratio (i.e., (1 + λ ) times the competitive ratio of ROBD). Empirically, RCL has the lowest competitive ratio among all the other algorithms, demonstrating the practical power of RCL for robustifying, potentially untrusted, ML predictions. In this experiment, RCL outperforms ROBD in terms of the empirical competitive ratio because it exploits the good ML predictions for those problem instances that are adversarial to ROBD. This result complements Theorem 4.1, where we show theoretically that RCL can outperform ROBD in terms of the average cost by properly setting λ . By comparison, ML performs well on average by exploiting the historical data, but has the highest competitive ratio due to its expected lack of robustness. The two alternative baselines, EC-L2O and HitMin, are empirically good on average and also in the worst case, but they do not have guaranteed robustness. On the other hand, ROBD is very robust, but its average cost is also the worst among all the algorithms under consideration.

More results, including using HitMin as the expert and large distributional shifts, are available in Appendix B.3.

6 Conclusion

We have considered a general SOCO setting (including multi-step switching costs and delayed hitting costs) and proposed RCL , which ensures a worst-case performance bound by utilizing an expert algorithm to robustify untrusted ML predictions. We prove that RCL is able to guarantee (1 + λ ) -competitive against any given expert for any λ > 0 . Additionally, we provide a sufficient condition that achieves finite robustness and 1-consistency simultaneously. To improve the average performance, we explicitly train the ML model in a robustification-aware manner by differentiating the robustification step, and provide an explicit average cost bound. Finally, we evaluate RCL using a case study of battery management for EV stations, which highlights the improvement in both robustness and average performance compared to existing algorithms.

Limitations and future work. We discuss two limitations of our work. First, we make a few assumptions (e.g., convexity and smoothness) on the hitting costs and switching costs. While they are common in the SOCO literature, these assumptions may not always hold in practice, and relaxing them will be interesting and challenging. Second, for online optimization, we only consider the ML prediction for the current step, whereas predictions for future steps can also be available in practice and may be explored for performance improvement. There are also a number of interesting open questions that follow from this work. For example, it is interesting to study alternative reservation costs and the optimality in terms of the tradeoff between bi-competitiveness. Additionally, it would also be interesting to extend RCL to other related problems such as convex body chasing and metrical task systems.

Acknowledgement. We would like to thank the anonymous reviewers for their helpful comments. Pengfei Li, Jianyi Yang and Shaolei Ren were supported in part by the U.S. NSF under the grant CNS-1910208. Adam Wierman was supported in part by the U.S. NSF under grants CNS-2146814, CPS-2136197, CNS-2106403, NGSDI-2105648.

References

Antonios Antoniadis, Christian Coester, Marek Elias, Adam Polak, and Bertrand Simon. Online metric algorithms with untrusted predictions. In ICML , 2020.

M. Lin, A. Wierman, L. L. H. Andrew, and E. Thereska. Dynamic right-sizing for powerproportional data centers. In INFOCOM , 2011.

Daan Rutten and Debankur Mukherjee. Capacity scaling augmented with unreliable machine learning predictions. SIGMETRICS Perform. Eval. Rev. , 49(2):24-26, jan 2022.

Yingying Li, Xin Chen, and Na Li. Online optimal control with linear dynamics and predictions: Algorithms and regret analysis. In NeurIPS , Red Hook, NY, USA, 2019. Curran Associates Inc.

Lijun Zhang, Wei Jiang, Shiyin Lu, and Tianbao Yang. Revisiting smoothed online learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Guanya Shi, Yiheng Lin, Soon-Jo Chung, Yisong Yue, and Adam Wierman. Online optimization with memory and competitive control. In NeurIPS , volume 33. Curran Associates, Inc., 2020.

Nicolas Christianson, Tinashe Handina, and Adam Wierman. Chasing convex bodies and functions with black-box advice. In COLT , 2022.

Yiheng Lin, Gautam Goel, and Adam Wierman. Online optimization with predictions and non-convex losses. Proc. ACM Meas. Anal. Comput. Syst. , 4(1), May 2020.

Niangjun Chen, Gautam Goel, and Adam Wierman. Smoothed online convex optimization in high dimensions via online balanced descent. In COLT , 2018.

Joshua Comden, Sijie Yao, Niangjun Chen, Haipeng Xing, and Zhenhua Liu. Online optimization in cloud resource provisioning: Predictions, regrets, and algorithms. Proc. ACM Meas. Anal. Comput. Syst. , 3(1), March 2019.

Ming Shi, Xiaojun Lin, and Lei Jiao. Combining regularization with look-ahead for competitive online convex optimization. In INFOCOM , 2021.

Pengfei Li, Jianyi Yang, and Shaolei Ren. Expert-calibrated learning for online optimization with switching costs. Proc. ACM Meas. Anal. Comput. Syst. , 6(2), Jun 2022.

Gautam Goel, Yiheng Lin, Haoyuan Sun, and Adam Wierman. Beyond online balanced descent: An optimal algorithm for smoothed online optimization. In NeurIPS , volume 32, 2019.

Daan Rutten, Nicolas Christianson, Debankur Mukherjee, and Adam Wierman. Smoothed online optimization with unreliable predictions. Proc. ACM Meas. Anal. Comput. Syst. , 7(1), mar 2023.

Nicolas Christianson, Junxuan Shen, and Adam Wierman. Optimal robustness-consistency tradeoffs for learning-augmented metrical task systems. In AI STATS , 2023.

Goran Zuzic, Di Wang, Aranyak Mehta, and D. Sivakumar. Learning robust algorithms for online allocation problems using adversarial training. In https://arxiv.org/abs/2010.08418 , 2020.

Bingqian Du, Zhiyi Huang, and Chuan Wu. Adversarial deep learning for online resource allocation. ACM Trans. Model. Perform. Eval. Comput. Syst. , 6(4), feb 2022.

Brandon Amos. Tutorial on amortized optimization for learning to optimize over continuous domains. CoRR , abs/2202.00665, 2022.

Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang, and Wotao Yin. Learning to optimize: A primer and a benchmark, 2021.

Jakub Chł˛ edowski, Adam Polak, Bartosz Szabucki, and Konrad Tomasz ˙ Zołna. Robust learningaugmented caching: An experimental study. In ICML , 2021.

Elbert Du, Franklyn Wang, and Michael Mitzenmacher. Putting the 'learning" into learningaugmented algorithms for frequency estimation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 2860-2869. PMLR, 18-24 Jul 2021.

Alexander Wei and Fred Zhang. Optimal robustness-consistency trade-offs for learningaugmented online algorithms. In NeurIPS , 2020.

Rishi Gupta and Tim Roughgarden. Data-driven algorithm design. Commun. ACM , 63(6):87-94, May 2020.

Joan Boyar, Lene M. Favrholdt, Christian Kudahl, Kim S. Larsen, and Jesper W. Mikkelsen. Online algorithms with advice: A survey. SIGACT News , 47(3):93-129, August 2016.

Pengfei Li, Jianyi Yang, and Shaolei Ren. Robustified learning for online optimization with memory costs. In INFOCOM , 2023.

Weici Pan, Guanya Shi, Yiheng Lin, and Adam Wierman. Online optimization with feedback delay and nonlinear switching cost. Proc. ACM Meas. Anal. Comput. Syst. , 6(1), Feb 2022.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference on machine learning (icml-03) , pages 928-936, 2003.

Niangjun Chen, Joshua Comden, Zhenhua Liu, Anshul Gandhi, and Adam Wierman. Using predictions in online optimization: Looking forward with an eye on the past. SIGMETRICS Perform. Eval. Rev. , 44(1):193-206, June 2016.

Yingying Li, Guannan Qu, and Na Li. Online optimization with predictions and switching costs: Fast algorithms and the fundamental limit. IEEE Transactions on Automatic Control , 2020.

Weiwei Kong, Christopher Liaw, Aranyak Mehta, and D. Sivakumar. A new dog learns old tricks: RL finds classic optimization algorithms. In ICLR , 2019.

Mohammad Ali Alomrani, Reza Moravej, and Elias Boutros Khalil. Deep policies for online bipartite matching: A reinforcement learning approach. Transactions on Machine Learning Research , 2022.

Bingqian Du, Chuan Wu, and Zhiyi Huang. Learning resource allocation and pricing for cloud profit maximization. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence , AAAI'19/IAAI'19/EAAI'19. AAAI Press, 2019.

Thodoris Lykouris and Sergei Vassilvitskii. Competitive caching with machine learned advice. J. ACM , 68(4), July 2021.

Keerti Anand, Rong Ge, Amit Kumar, and Debmalya Panigrahi. A regression approach to learning-augmented online algorithms. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Pengfei Li, Jianyi Yang, and Shaolei Ren. Expert-calibrated learning for online optimization with switching costs. In SIGMETRICS , 2022.

Sascha Geulen, Berthold Vöcking, and Melanie Winkler. Regret minimization for online buffering problems using the weighted majority algorithm. In COLT , pages 132-143. Citeseer, 2010.

Amir Sani, Gergely Neu, and Alessandro Lazaric. Exploiting easy data in online optimization. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014.

Vaidehi Srinivas, David P. Woodruff, Ziyu Xu, and Samson Zhou. Memory bounds for the experts problem. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , STOC 2022, page 1158-1171, New York, NY, USA, 2022. Association for Computing Machinery.

Yunchang Yang, Tianhao Wu, Han Zhong, Evrard Garcelon, Matteo Pirotta, Alessandro Lazaric, Liwei Wang, and Simon Shaolei Du. A reduction-based framework for conservative bandits and reinforcement learning. In International Conference on Learning Representations , 2022.

Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J. Ramadge. Projection-based constrained policy optimization. In International Conference on Learning Representations , 2020.

Yecheng Jason Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforcement learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Guanya Shi, Xichen Shi, Michael O'Connell, Rose Yu, Kamyar Azizzadenesheli, Animashree Anandkumar, Yisong Yue, and Soon-Jo Chung. Neural lander: Stable drone landing control using learned dynamics. In International Conference on Robotics and Automation (ICRA) , pages 9784-9790. IEEE, 2019.

Guanya Shi. Competitive control via online optimization with memory, delayed feedback, and inexact predictions. In 2021 55th Annual Conference on Information Sciences and Systems (CISS) , pages 1-1. IEEE, 2021.

Ohad Shamir and Liran Szlak. Online learning with local permutations and delayed feedback. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 , ICML'17, page 3086-3094. JMLR.org, 2017.

Daan Rutten, Nico Christianson, Debankur Mukherjee, and Adam Wierman. Online optimization with untrusted predictions. CoRR , abs/2202.03519, 2022.

Tongxin Li, Ruixiao Yang, Guannan Qu, Guanya Shi, Chenkai Yu, Adam Wierman, and Steven Low. Robustness and consistency in linear quadratic control with untrusted predictions. Proc. ACM Meas. Anal. Comput. Syst. , 6(1), feb 2022.

W. Tang, S. Bi, and Y. Zhang. Online coordinated charging decision algorithm for electric vehicles without future information. IEEE Trans. Smart Grid , 5:2810-2824, May 2014.

A Illustration of RCL

We illustrate the online optimization process of RCL in Fig. 1.

xo

MLModel

美

x

ML Model

XT-1

ML Model

美

x

Robustification

x

Robustification

xT

Robustification

Ox

x1

XT-1

TotalCost....

TotalCost.....

x

Expert

Expert

XT-1

Expert

y

B Case Study: Battery Management for EV Charging Stations

B.1 Problem Formulation

Batteries are routinely used in EV charging stations to handle the rapidly fluctuating charging demands and protect the connected grid. Thus, properly managing battery charging/discharging decisions is crucial for reliability, lifespan, and safety of batteries and grids.

We consider the management of N batteries. At each time step t , suppose that x t ∈ R N + represents the State of Charge (SoC) and u t ∈ R N represents the battery charging/discharging schedule, depending on the sign of u t (i.e., positive means charging, and vice versa). The canonical form of the battery dynamics can be written as x t +1 = Ax t + Bu t -w t , where A is a N × N matrix which models the self-degradation of the N -battery system, B is a N × N matrix which represents the charging efficiency of each battery unit, w t is a N × 1 vector which denotes the current demand in terms of the charging rate (kW) of all the EVs connected to the charging stations. Assuming that the initial SoC as x 0 , the goal is to control the batteries to minimize the difference between the current SoC of all batteries and a nominal value ¯ x , plus a charging/discharging cost to account for battery usage [49, 50], which can be expressed mathematically as min u 1 ,u 2 , ··· ,u T +1 ∑ T +1 t =1 ∥ x t -¯ x ∥ 2 + b ∥ u t ∥ 2 .

This problem falls into SOCO based on the reduction framework described in [49]. Specifically, at time step t + 1 , we can expand x t +1 based on the battery dynamics as x t +1 = A t x 1 + ∑ t j =1 A t -j Bu j -∑ t j =1 A t -j w j , We define the context parameter as y t = ¯ x -A t x 1 + ∑ t i =1 A t -i w i and the action as a t = ∑ t i =1 A t -i Bu i . Then, assuming an identity matrix B (ignoring charging loss), the optimization problem becomes min a 1 , ··· ,a T ∥ x 1 -¯ x ∥ 2 + b ∥ u T ∥ 2 + ∑ T t =1 ∥ a t -y t ∥ 2 + b ∥ a t -Aa t -1 ∥ 2 . Given an initial value of x 1 , this problem can be further simplified and reformulated as

which is in a standard SOCO form by considering y t as the context and a t as the action at time t .

To validate the effectiveness of RCL , we use a public dataset [51] provided by ElaadNL, a Dutch EV charging infrastructure company. We collect a dataset containing transaction records from ElaadNL charging stations in the Netherlands from January to June of 2019. Each transaction record contains the energy demand, transaction start time and charging time. As the data does not specify the details of battery units, we consider the battery units as a single combine battery by summing up the energy demand within each hour to obtain the hourly energy demand.

We use the January to February data as the training dataset, March to April data as the validation dataset for tuning the hyperparameters such as learning rate, and May to June as the testing dataset. We consider each problem instance as one day ( T = 24 hours, plus an initial action). Thus, a sliding

window of 25 is applied, moving one hour ahead each time, on the raw data to generate 1416 problem instances, where the first demand of each instance is used as the initial action of all the algorithms. We set b = 10 and A = I for the cost function in Eqn. (7).

All the algorithms use the same ML architecture, when applicable, with the same initialized weights in our experiments for fair comparison. To be consistent with the literature [52, 53], all the ML models are trained offline. Specifically, we use a recurrent neural network (RNN) model that contains 2 hidden layers, each with 8 neurons, and implement the model using PyTorch. We train the RNN for 140 epochs with a batch size of 50. When the RNN model is trained as a standalone optimizer in a robustification-oblivious manner, the training process takes around 1 minute on a 2020 MacBook Air with 8GB memory and a M1 chipset. When RNN is trained in a robustification-aware manner, it takes around 2 minutes. The testing process is almost instant and takes less than 1 second.

B.2 Baseline Algorithms

By default, RCL uses a robustification-aware ML model due to the advantage of average cost performance compared to a robustification-oblivious model. We compare RCL with several representative baseline algorithms as summarized below.

Offline Optimal Oracle ( OPT ): This is the optimal offline algorithm that has all the contextual information and optimally solves the problem.

Regularized Online Balanced Descent ( ROBD ): ROBD is the state-of-the-art order-optimal online algorithm with the best-known competitive ratio for our SOCO setting [54, 49]. The parameters of ROBD are all optimally set according to [49]. By default, RCL uses ROBD as its expert for robustness.

Hitting Cost Minimizer ( HitMin ): HitMin is a special instance of ROBD by setting the parameters such that it greedily minimizing the hitting cost at each time. This minimizer can be empirically effective and hence also used in ROBD as a regularizer.

Machine Learning Only ( ML ): ML is trained as a standalone optimizer in a robustification-oblivious manner. It does not use robustification during online optimization.

Expert-Calibrated Learning ( EC-L2O ): It is an ML-augmented algorithm that applies to our SOCO setting by using an ML model to regularize online actions without robustness guarantees [55]. We set its parameters based on the validation dataset to have the optimal average performance with an empirical competitive ratio less than (1 + λ ) CR π .

RCL with a robustification-oblivious ML model ( RCL-O ): To differentiate the two forms of RCL , we use RCL to refer to RCL with a robustification-aware ML model and RCL-O for the robustificationoblivious ML model, where ' -O ' represents robustification-obliviousness.

To highlight our key contribution to the SOCO literature, the baseline algorithms we choose are representative of the state-of-the-art expert algorithms, effective heuristics, and ML-augmented algorithms for the SOCO setting we consider. While there are a few other ML-augmented algorithms for SOCO [56, 57, 58], they do not apply to our problem as they consider unsquared switching costs in a metric space and exploit the natural triangular inequality. Adapting them to the squared switching costs is non-trivial.

B.3 Additional Empirical Results

In Section 5, we have evaluated RCL using ROBD as the expert online algorithm with λ = 1 . Here we provide additional experiment results to further evaluate the effectiveness of RCL , and the results are organized as follows. First, we change the expert online algorithm from ROBD to HitMin to show the flexibility of RCL . Second, we quantitatively present the effect of parameter λ in RCL . Third, we introduce additional out-of-distribution samples to test the robustness of RCL and baselines, in terms of competitive ratio. Finally, we experiment under the delayed-feedback setting, which is a more challenging problem setup.

B.3.1 Utilizing HitMin as the expert

RCL is flexible and can work with any expert online algorithm, even an expert that does not have good or bounded competitive ratios. Thus, it is interesting to see how RCL performs given an alternative

1.0

1.5

2.0

2.5

3.0

Cost Ratio

RCL

RCL-O

ML

EC-L2O

HitMin

1.0

1.5

2.0

2.5

3.0

Cost Ratio

RCL

RCL-O

ML

EC-L2O

HitMin

1.0

1.5

2.0

2.5

3.0

Cost Ratio

ML

RCL-O

=5.0

RCL-O

=3.0

RCL-O

=1.0

RCL-O

=0.6

1.0

1.5

2.0

2.5

3.0

Cost Ratio

ML

RCL

=5.0

RCL

=3.0

RCL

=1.0

RCL

=0.6

(a) ROBD as the expert

(b) HitMin as the expert

(c) RCL-O w/ different λ

expert. For example, in Table 1, HitMin empirically outperforms ROBD in terms of the average, although it is not as robust as ROBD. Thus, using λ = 1 , we leverage HitMin as the expert for RCL and RCL-O , and show the cost ratio distributions in Fig. 2(b). Comparing Fig. 2(b) with Fig. 2(a), we see that RCL and RCL-O both have many low cost ratios by using HitMin as the expert, but the worst case for RCL is not as good as when using ROBD as the expert. For example, the average cost and competitive ratio are 1.0515 and 1.6035, respectively, for RCL . This result is not surprising, as the new expert HitMin has a better average performance but worse competitive ratio than the default expert ROBD.

B.3.2 Impact of λ

Theorem 4.1 shows the point that we need to set a large enough λ in order to provide enough flexibility for RCL to exploit good ML predictions. With a small λ > 0 , despite the stronger competitiveness against the expert, it is possible that RCL may even empirically perform worse than both the ML model and the expert. Thus, we now investigate the impact of λ .

We see from Table 1 that the empirical average cost and competitive ratio of RCL are both worse with λ = 0 . 6 than with the default λ = 1 . More interestingly, by setting λ = 5 , the average cost of RCL is even lower than that of ML. This is because ML in our experiment performs fairly well on average. Thus, by setting a large λ = 5 , RCL is able to exploit the benefits of good ML predictions for many typical cases, while using the expert ROBD as a safeguard to handle a few bad problem instances for which ML cannot perform well. Also, the empirical competitive ratio of RCL is better with λ = 5 than with λ = 3 , supporting Theorem 4.1 that a larger λ may not necessarily increase the competitive ratio as RCL can exploit good ML predictions. In addition, given each λ , RCL outperforms RCL-O , which highlights the importance of training the ML model in a robustification-aware manner to avoid the mismatch between training and testing objectives.

We also show in Fig. 2(c) an Fig. 2(d) the cost ratio distributions for RCL-O and RCL , respectively, under different λ . The results reaffirm our main Theorem 4.1 as well as the importance of training the ML model in a robustification-aware manner.

Next, we show the bi-competitive cost ratios of RCL-O against both the expert ROBD and the ML predictions. We focus on RCL-O as its ML model is trained as a standalone optimizer, whereas RCL uses a robustification-aware ML model that is not specifically trained to produce good pre-robustification predictions. According to Theorem 4.1, RCL-O obtains a potentially better competitiveness against MLbut a worse competitive against the expert ROBD when λ increases, and vice versa. To further validate the theoretical analysis, we test RCL-O with different λ and obtain the 2D histogram of its bi-competitive cost ratios against ROBD and ML, respectively. The results are shown in Fig. 3. In agreement with our analysis, the cost ratio of RCL-O against ROBD never exceeds (1 + λ ) for any λ > 0 . Also, with a small λ = 0 . 6 , the cost ratio of RCL-O against ROBD concentrates around 1, while it does not exploit the benefits of ML predictions very well. On the other hand, with a large λ = 5 , the cost ratio of RCL-O against ROBD can be quite high, although it follows (good) ML predictions more closely for better average performance. Most importantly, by increasing λ > 0 , we can see the general trend that RCL-O follows the ML predictions more closely while still being able to guarantee competitiveness against ROBD. Again, this confirms the key point of our main insights in Theorem 4.1.

0.4

0.7

1.0

1.3

1.6

1.9

2.2

RCL-O / ROBD Cost Ratio

0.4

0.7

1.0

1.3

1.6

1.9

2.2

RCL-O / ML Cost Ratio

1.0

0.5

0.0

0.5

1.0

1.5

2.0

2.5

0.4

0.7

1.0

1.3

1.6

1.9

2.2

RCL-O / ROBD Cost Ratio

0.4

0.7

1.0

1.3

1.6

1.9

2.2

RCL-O / ML Cost Ratio

1.0

0.5

0.0

0.5

1.0

1.5

2.0

2.5

0.4

0.7

1.0

1.3

1.6

1.9

2.2

RCL-O / ROBD Cost Ratio

0.4

0.7

1.0

1.3

1.6

1.9

2.2

RCL-O / ML Cost Ratio

1.0

0.5

0.0

0.5

1.0

1.5

2.0

2.5

0.4

0.7

1.0

1.3

1.6

1.9

2.2

RCL-O / ROBD Cost Ratio

0.4

0.7

1.0

1.3

1.6

1.9

2.2

RCL-O / ML Cost Ratio

1.0

0.5

0.0

0.5

1.0

1.5

2.0

2.5

B.3.3 Larger distributional shifts

In our dataset, ML performs very well on average as the testing distribution matches well with its training distribution. To consider more challenging cases as a stress test, we manually increase the testing distributional shifts by adding random noise following N (0 , σ ) to a certain faction p c of the testing samples. Note that, as we intentionally stress test RCL and RCL-O under a larger distributional shift, their ML models remain unchanged as in the default setting and are not re-trained by adding noisy data to the training dataset.

With the default λ = 1 , we show the average cost and competitive ratio results in Table 2. We see that ROBD is very robust and little affected by the distributional shifts. In terms of the competitive ratio, ML, HitMin and EC-L2O are not robust, resulting in a large competitive ratio when we add more noisy samples. The average cost performance of RCL is empirically better than that of RCL-O in almost all cases, except for a slight increase in the practically very rare case where 20% samples are contaminated with large noise. On the other hand, as expected, the competitive ratios of RCL and RCL-O both increase as we add more noise. While RCL has a higher competitive ratio than RCL-O empirically in the experiment, they both have the same guaranteed (1 + λ ) competitiveness against ROBD regardless of how their ML models are trained. Also, their competitive ratios are both better than other algorithms, showing the effectiveness of our novel robustification process.

B.4 Results with Feedback Delay

We now turn to the case when there is a one-step feedback delay, i.e., the context parameter y t is not known to the agent until time t +1 . For this setting, we consider the best-known online algorithm iROBD [49] as the expert that handles the feedback delay with a guaranteed competitive ratio with respect to OPT. The other baseline online algorithms - ROBD, EC-L2O, and HitMin- presented in Section B.2 require the immediate revelation of y t without feedback delay and hence do not directly apply to this case. Thus, for comparison, we use the predicted context, denoted by ˆ y t , with up to 15% prediction errors in the baseline online algorithms, and reuse the algorithm names (e.g., EC-L2O uses predicted ˆ y t as if it were the true context for decision making). We train ML using the same

1

2

3

4

5

6

7

8

9

10

Cost Ratio

RCL

RCL-O

ML

EC-L2O

HitMin

(a) iROBD as the expert

1

2

3

4

5

6

7

8

Cost Ratio

ML

RCL

=5.0

RCL

=3.0

RCL

=1.0

RCL

=0.6

(b)

RCL

w/ different

λ

1

2

3

4

5

6

7

8

Cost Ratio

ML

RCL-O

=5.0

RCL-O

=3.0

RCL-O

=1.0

RCL-O

=0.6

(c)

RCL-O

w/ different

λ

architecture as in Section B.3, with the exception that only delayed context is provided as input for both training and testing. The reported values are normalized with respect to those of the respective offline optimal algorithm OPT. The average cost ( AVG ) and competitive ratio ( CR ) are all empirical results reported on the testing dataset.

We show the results in Table 3 and Fig. 4. We see that with the default λ = 1 , both RCL and RCL-O have a good average cost, but RCL has a lower average cost than RCL-O and is outperformed only by ML in terms of the average cost. RCL and RCL-O have the same competitive ratio guarantee (i.e., (1 + λ ) times the competitive ratio of iROBD). Nonetheless, RCL has the lowest competitive ratio than all the other algorithms, demonstrating the power of RCL to leverage both ML prediction and the robust expert. In this experiment, both RCL and RCL-O outperform iROBD in terms of the empirical competitive ratio because they are able to exploit the good ML predictions for those problem instances that are difficult for iROBD.

By comparison, ML performs well on average by exploiting the historical data, but has a high competitive ratio. The alternative baselines - ROBD, EC-L2O and HitMin- use predicted context ˆ y t as the true context. Except for the good empirical competitive ratio of ROBD, they do not have good average performance or guaranteed robustness due to their naively trusting the predicted context (that can potentially have large prediction errors). Note that the empirical competitive ratio of ROBD with predicted context is still much higher than that with the true context in Table 1. These results reinforce the point that blindly using ML predictions (i.e., predicted context in this example) without additional robustification can lead to poor performance in terms of both average cost and worst-case cost ratio.

We further show in Fig. 4 the box plots for cost ratios of different algorithms, providing a detailed view of the algorithms' performance. The key message is that RCL obtains the best of both worlds a good average cost and a good competitive ratio. Moreover, we see that by setting λ = 1 , we provide enough freedom to RCL to exploit the benefits of ML predictions while also ensuring worst-case robustness. Thus, like in the no-delay case in Table 1 and Fig. 2, the empirical competitive ratio of RCL with λ = 1 is even lower than that with λ = 0 . 6 .

C Proof of Theorems and Corollaries in Section 4

C.1 Proof of Theorem 4.1 (Cost Ratio)

To prove Theorem 4.1, we first give some technical lemmas about the smoothness of cost functions from Lemma C.1 to Lemma C.3.

Lemma C.1 (Lemma 4 in [59]) . Assume f ( x ) is β smooth, for any λ > 0 , we have

Lemma C.2. Assume f ( x ) is β 1 smooth and d ( x ) is β 2 smooth, then f ( x ) + d ( x ) is β 1 + β 2 smooth.

Lemma C.3. Suppose that the hitting cost f ( x, y t ) is β h -smooth with respect to x , The switching cost is d ( x t , x t -1 ) = 1 2 ∥ x t -δ ( x t -p : t -1 ) ∥ 2 , where δ ( · ) is L i -Lipschitz with respect to x t -i . Then for any two action sequences x 1: T and x ′ 1: T , we must have

Proof. The objective to be bounded can be decomposed as

Since hitting cost is β h -smooth, then

Besides, based on the Lipschitz assumption of function δ ( · ) , we have

Summing up the switching costs of all time steps together, we have

Substituting Eqn. (12) and Eqn. (10) into Eqn. (9), we finish the proof.

Nowwepropose Lemma C.4 based on these above lemmas, which ensures the feasibility of robustness constraint in Eqn. (1)

Lemma C.4. Let π be any expert algorithm for the SOCO problem with multi-step feedback delays and multi-step switching costs, for any λ ≥ 0 and λ ≥ λ 0 ≥ 0 , the total cost by the projected actions x t must satisfy cost ( x 1: T ) ≤ (1 + λ ) cost ( x π 1: T )

Proof. We prove by induction that the constraints in Eqn. (1) are satisfied for each t . For t = 1 , since we assume the initial actions are the same ( x -p +1:0 = x π -p +1:0 ), it is obvious that x = x π 1 satisfies the robustness constraints Eqn. (1).

Then for any time step t ≥ 2 , suppose it holds at t -1 that

Now the robustness constraints Eqn. (1) is satisfied if we prove x t = x π t satisfies the constraints in Eqn. (1) at time step t . Since for the sets A and B , we have

so it holds that

Denote α = 1 + ∑ p k =1 L k . For the reservation cost, we have

Continuing with Eqn. (17), we have

where the second inequality holds by Jensen's inequality. Therefore, combining with (16), we have d ( x π t , x t -p : t -1 )+ G ( x π t , x t -p : t -1 , x π t -p : t ) ≤ G ( x t -1 , x t -p -1: t -2 , x π t -p -1: t -1 )+(1+ λ ) d ( x π t , x π t -p : t -1 ) (19)

By Eqn. (19), we have

Now we define a new set D t = A t \A t -1 , which denotes the timestep set for the newly received context parameters at t .

Case 1 : If t ∈ D t , then B t -1 \ B t = D t \{ t } , then we have

Since hitting cost f ( · , y t ) is β h -smooth, we have

Substituting Eqn. (22) back to Eqn. (21), we have

Case 2 : If t / ∈ D t , then ( B t -1 ∪ { t } ) \ B t = D t and we have

Since hitting cost f ( · , y t ) is β h -smooth, we have

Since λ ≥ 0 , we substitute Eqn. (25) back to Eqn. (24), we have the same conclusion as Eqn (23).

Adding Eqn. (13), Eqn. (20) and Eqn. (23) together, we can prove x = x π t satisfies the constraints in Eqn. (1). At time step T , we have

In the next lemma, we bound the difference between the projected action and the ML predictions. Lemma C.5. Suppose hitting cost is β h -smooth, given the expert policy π , ML predictions ˜ x 1: T , for any λ > 0 and λ 1 > 0 , the total distance between actual actions x 1: T and ML predictions ˜ x 1: T are bounded,

Remember that D t = A t \A t -1 is the set of the time steps for the newly received context parameters at t . The robustness constraint in Eqn. (1) is satisfied if x t satisfies the following inequality.

For the switching cost, we have

The first inequality comes from Lemma C.1, the second inequality comes from the L i -Lipschitz assumption, and the third inequality is because α ≥ 1 . Besides, from Eqn (17), we have

Thus we have

Combining with inequality (31), we have

Substituting Eqn. (34) back to Eqn. (30), we have

Case 1 : If t ∈ D t , then B t -1 \ B t = D t \{ t } , then Eqn.(35) becomes

Since hitting cost is β h -smooth, the sufficient condition for Eqn. (35) becomes

Since the hitting cost is non-negative, the sufficient condition can be further simplified, which is

Since hitting cost is β h -smooth, the sufficient condition for Eqn. (39) becomes

Now we define

At time step t , if x ′ t is the solution to this alternative optimization problem

The solution to this problem can be calculated asd

Then ∥ x ′ t -˜ x t ∥ = [ ∥ ˜ x t -x π t ∥ -√ K ( d ( x π t , x π t -p : t -1 ) + ∑ τ ∈D t f ( x π τ , y τ ) ) ] + . Since x ′ t also satisfies the original robustness constraint, we have ∥ x t -˜ x t ∥ ≤ ∥ x ′ t -˜ x t ∥ and we finish the proof.

Proof of Theorem 4.1

Now summing up the distance through 1 to T, we have

Based on Lemma C.3 we have ∀ λ 2 > 0 ,

Suppose the offline optimal action sequence is x ∗ 1: T , the optimal cost is cost ( x ∗ 1: T ) . Then we divide both sides of Eqn. (44) by cost ( x ∗ 1: T ) , and get ∀ λ 2 > 0 ,

By substituting K = 2( λ -λ 0 ) ( β h + α 2 )(1+ 1 λ 0 ) back to Eqn (46), we have

By defining single step cost of the expert π as cost π t = d ( x π t , x π t -p : t -1 ) + ∑ τ ∈D t f ( x π τ , y τ ) and the auxiliary cost as ∆( λ ) = ∑ T i =1 [ ∥ ˜ x t -x π t ∥ 2 -2( λ -λ 0 ) ( β h + α 2 )(1+ 1 λ 0 ) cost π t ] +

Combined with Lemma C.4, we obtain the following bound, which finished this proof.

C.2 Proof of Theorem 4.2

Proof. We first give the formal definition of Rademacher complexity of the ML model space with robustification.

Definition 5 (Rademacher Complexity) . Let Rob λ ( W ) = { Rob λ ( h W ) , W ∈ W} be the ML model space with robustification constrained by (2) . Given the dataset S , the Rademacher complexity with respect to Rob λ ( W ) is

where y i is the i -th sample in S , and ν 1 , · · · , ν n are independently drawn from Rademacher distribution.

Since the cost functions are smooth, they are locally Lipschitz continuous for the bounded action space, and we can apply the generalization bound based on Rademacher complexity [60] for the space of robustified ML model Rob λ ( h W ) . Given any ML model h W trained on dataset S , with probability at least 1 -δ, δ ∈ (0 , 1) ,

where Γ x = √ T |X| [ β h + 1 2 (1 + ∑ p i =1 L i )(1 + ∑ p i =1 L i ) ] with |X| being the size of the action space X and β h , L i , and p as the smoothness constant, Lipschitz constant of the nonlinear term in the switching cost, and the memory length as defined in Assumptions 1 and 2, and ¯ c is the upper bound of the total cost for an episode. We can get the average cost bound in Proposition 4.2.

Next, we prove that the Rademacher complexity of the ML model space with robustification is no larger than the Rademacher complexity of the ML model space without robustification expressed as { h W , W ∈ W} , i.e. we need to prove Rad S ( Rob λ ( W )) ≤ Rad S ( W ) . The Rademacher complexity can be expressed by Dudley's entropy integral [61] as

where N ( ϵ, Rob λ ( W ) , L 2 ( S )) is the covering number [61] with respect to radius ϵ and the function distance metric ∥ h 1 -h 2 ∥ L 2 ( S ) = 1 |S| ∑ i ∈S ∥ h 1 ( x i ) -h 2 ( x i ) ∥ 2 where h 1 and h 2 are two functions defined on the space including dataset S . We can find that for any two different weights W 1 and W 2 , their corresponding post-robustification distance ∥ Rob λ ( h W 1 ) -Rob λ ( h W 2 ) ∥ L 2 ( S ) is no larger than their pre-robustification distance ∥ h W 1 -h W 2 ∥ L 2 ( S ) . To see this, we discuss three cases given any input sample y . If both h W 1 ( y ) and h W 2 ( y ) lie in the projection set, then Rob λ ( h W 1 )( y ) = h W 1 ( y ) and Rob λ ( h W 2 )( y ) = h W 2 ( y ) . If h W 1 ( y ) lies in the projection set while h W 2 ( y ) is out of the projection set, the projection operation based on the closed convex projection set will make ∥ Rob λ ( h W 1 )( y ) -Rob λ ( h W 2 )( y ) ∥ to be less than ∥ h W 1 ( y ) -h W 2 ( y ) ∥ . If both h W 1 ( y ) and h W 2 ( y ) lie out of the projection set, we still have ∥ Rob λ ( h W 1 )( y ) -Rob λ ( h W 2 )( y ) ∥ ≤ ∥ h W 1 ( y ) -h W 2 ( y ) ∥

since the projection set at each round is a closed convex set [62]. Therefore, after robustification, the distance between two models with different weights will not become larger, i.e. ∥ Rob λ ( h W 1 ) -Rob λ ( h W 2 ) ∥ L 2 ( S ) ≤ ∥ h W 1 -h W 2 ∥ L 2 ( S ) , which means RCL has a covering number N ( ϵ, Rob λ ( W ) , L 2 ( S )) no larger than that of the individual ML model N ( ϵ, W , L 2 ( S )) for any ϵ . Thus the Rademacher complexity with the robustification procedure does not increase.

By [63], the upper bound of Rademacher complexity with respect to the space of ML model Rad S ( Rob λ ( W )) is in the order of O ( 1 √ |S| ) . Since the Rademacher complexity with the robustification procedure satisfies Rad S ( Rob λ ( W )) ≤ Rad S ( W ) , it also decreases with the dataset size in the order of O ( 1 √ |S| ) .

D Robustification-aware Training

Theorem 4.2 also shows the benefits of training the ML model in a robustification-aware manner. Specifically, by comparing the losses in (5) and (6), we see that using (6) as the robustification-aware loss for training W can reduce the term cost S ( ROB ( h W )) in the average cost bound, which matches exactly with the training objective in (6). The robustification-aware approach is only beginning to be explored in the ML-augmented algorithm literature and non-trivial (e.g., unconstrained downstream optimization in [55]), especially considering that (1) is a constrained optimization problem with no explicit gradients.

Gradient-based optimizers such as Adam [64] are the de facto state-of-the-art algorithms for training MLmodels, offering better optimization results, convergence, and stability compared to those nongradient-based alternatives [65]. Thus, it is crucial to derive the gradients of the loss function with respect to the ML model weight W given the added robustification step.

Next, we derive the gradients of x t with respect to ˜ x t . For the convenience of presentation, we use the basic SOCO setting with a single-step switching cost and no hitting cost delay as an example, while noting that the same technique can be extended to derive gradients in more general settings. Specifically, for this setting, the pre-robustification prediction is given by ˜ x t = h W (˜ x t -1 , y t ) , where W denotes the ML model weight. Then, the actual post-robustification action x t is obtained by projection in (1) by setting q = 0 and p = 1 , given the ML prediction ˜ x t , the expert's action x π t and cumulative cost( x π 1: t ) up to t , and the actual cumulative cost ( x 1: t -1 ) up to t -1 .

The gradient of the loss function cost( x 1: T ) = ∑ T t =1 ( f ( x t , y t ) + d ( x t , x t -1 )) with respect to the MLmodel weight W is given by ∑ T t =1 ∇ W ( f ( x t , y t ) + d ( x t , x t -1 ) ) . Next, we write the gradient of per-step cost with with respect to W as follows:

where the gradients ∇ x t ( f ( x t , y t ) + d ( x t , x t -1 ) ) and ∇ x t -1 d ( x t , x t -1 ) are trivial given the hitting and switching cost functions, and the gradient ∇ W x t -1 is obtained at time t -1 in the same way as ∇ W x t . To derive ∇ W x t , by the chain rule, we have:

where ∇ W ˜ x t is the gradient of the ML output (following a recurrent architecture illustrated in Fig. 1 in the appendix) with respect to the weight W and can be obtained recursively by using off-the-shelf BPTT optimizers [64], and ∇ W cost ( x 1: t -1 ) = ∑ t -1 τ =1 ∇ W ( f ( x τ , y τ ) + d ( x τ , x τ -1 ) ) can also be recursively calculated once we have the gradient in Eqn. (51). Nonetheless, it is non-trivial to calculate the two gradient terms in Eqn. (52), i.e., ∇ ˜ x t x t and ∇ cost ( x 1: t -1 ) x t , where x t itself is the solution to the constrained optimization problem (1) unlike in the simpler unconstrained case [55]. As we cannot explicitly write x t in a closed form in terms of ˜ x t and cost ( x 1: t -1 ) , we leverage the KKT conditions [66, 67, 68] to implicitly derive ∇ ˜ x t x t and ∇ cost ( x 1: t -1 ) x t in the next proposition.

Proposition D.1 (Gradients by KKT conditions) . Let x t ∈ X and µ ≥ 0 be the primal and dual solutions to the problem (1) , respectively. The gradients of x t with respect to ˜ x t and cost ( x 1: t -1 ) are

If the ML prediction ˜ x t happens to lie on the boundary such that the inequality in (1) becomes an equality for x = ˜ x t , then the gradient does not exist in this case and Sc (∆ , ∆ 11 ) may not be fullrank. Nonetheless, we can still calculate the pseudo-inverse of Sc (∆ , ∆ 11 ) and use Proposition D.1 to calculate the subgradient. Such approximation is actually a common practice to address nondifferentiable points for training ML models, e.g., using 0 as the subgradient of ReLu ( · ) at the zero point [64].

Reference

W. Tang, S. Bi, and Y. Zhang. Online coordinated charging decision algorithm for electric vehicles without future information. IEEE Trans. Smart Grid , 5:2810-2824, May 2014.

Weici Pan, Guanya Shi, Yiheng Lin, and Adam Wierman. Online optimization with feedback delay and nonlinear switching cost. Proc. ACM Meas. Anal. Comput. Syst. , 6(1), feb 2022.

Tongxin Li, Ruixiao Yang, Guannan Qu, Yiheng Lin, Steven Low, and Adam Wierman. Equipping black-box policies with model-based advice for stable nonlinear control. In https://arxiv.org/abs/2206.01341 , 2022.

Chris Develder, Nasrin Sadeghianpourhamami, Matthias Strobbe, and Nazir Refa. Quantifying flexibility in ev charging as dr potential: Analysis of two real-world data sets. In 2016 IEEE International Conference on Smart Grid Communications (SmartGridComm) , pages 600-605. IEEE, 2016.

Mohammad Ali Alomrani, Reza Moravej, and Elias B. Khalil. Deep policies for online bipartite matching: A reinforcement learning approach. CoRR , abs/2109.10380, 2021.

Bingqian Du, Zhiyi Huang, and Chuan Wu. Adversarial deep learning for online resource allocation. ACM Trans. Model. Perform. Eval. Comput. Syst. , 6(4), feb 2022.

Gautam Goel, Yiheng Lin, Haoyuan Sun, and Adam Wierman. Beyond online balanced descent: An optimal algorithm for smoothed online optimization. In NeurIPS , volume 32, 2019.

Pengfei Li, Jianyi Yang, and Shaolei Ren. Expert-calibrated learning for online optimization with switching costs. Proc. ACM Meas. Anal. Comput. Syst. , 6(2), Jun 2022.

Antonios Antoniadis, Christian Coester, Marek Elias, Adam Polak, and Bertrand Simon. Online metric algorithms with untrusted predictions. In ICML , 2020.

Nicolas Christianson, Tinashe Handina, and Adam Wierman. Chasing convex bodies and functions with black-box advice. In COLT , 2022.

Daan Rutten, Nico Christianson, Debankur Mukherjee, and Adam Wierman. Online optimization with untrusted predictions. CoRR , abs/2202.03519, 2022.

Guanya Shi, Yiheng Lin, Soon-Jo Chung, Yisong Yue, and Adam Wierman. Online optimization with memory and competitive control. In NeurIPS , volume 33. Curran Associates, Inc., 2020.

Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3(Nov):463-482, 2002.

Martin Wainwright. Rademacher and empirical covering. https://people.eecs.berkeley. edu/~wainwrig/stat241b/lec20.pdf , 2009.

Constantine Caramanis and Sujay Sanghavi. Projection onto a convex set. https://users. ece.utexas.edu/~cmcaram/EE381V_2012F/Lecture_3_Scribe_Notes.final.pdf , 2012.

Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems , 30, 2017.

Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning . MIT Press, 2016.

Hyung Ju Suh, Max Simchowitz, Kaiqing Zhang, and Russ Tedrake. Do differentiable simulators give better policy gradients? In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 20668-20696. PMLR, 17-23 Jul 2022.

S. Boyd and L. Vandenberghe. Convex Optimization . Cambridge University Press, 2004.

Brandon Amos. Tutorial on amortized optimization for learning to optimize over continuous domains. CoRR , abs/2202.00665, 2022.

Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and J. Zico Kolter. Differentiable convex optimization layers. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies Smoothed Online Convex Optimization (SOCO) problems where the target is to minimize the sum of a per-round hitting cost and a switching cost that penalizes temporal changes in actions.\n\nThe most significant difference from previous studies is that they assume a delayed hitting cost and a nonlinear multi-step switching cost, which is quite more general than before and thus brings technical difficulties to provide theoretically guaranteed methods.\nFurthermore, they also want to use ML-based online optimizers to enhance performance.\n\nTo that end,  they developed a meta-algorithm that robustifies a given online optimizer by projecting its suggested action onto a robustified action set computed from a given expert method and historical observations. \nThey provide a worst-case analysis for the cost of the resulting algorithm, based on which, they further establish a robustness guarantee and consistency result in the language of $CR$-competitive (see Definition 4) once a condition is satisfied.\nIn the appendix, they explore the empirical performance of the proposed algorithm using a case study of battery management and validate its robustness.\n\nSTRENGTHS:\n1. The studied setting is general enough and can incorporate popular neural networks as its part. \n2. The worst-case theoretical analysis seems new and the proof seems correct.\n3. Though put in the appendix, the experiments are well conducted with detailed descriptions of the experiment setup and chosen parameters, and a careful analysis of the experiment results. I appreciate the experiments a lot.\n4. The paper is well-written mostly and the analysis is always to the point.\n\nWEAKNESSES:\nHowever, there are some parts not clear enough.\n\n1. The term ``multi-step nonlinear memory’’ appears abruptly in line 38. I think several additional explanations would help readers to digest this concept better for the first time.\n\n2. In line 200, the author didn’t explain the meaning of $\\text{cost}(x_{1:T})$.\n\n3. Line 353, it is unclear to me what the notation $\\text{Rob}_{\\lambda} \\left(\\tilde{x}_{1: T} \\right)$ means.\n\nQUESTIONS:\n1. Intuitively, once establishing the worst-case bound uniformly in Theorem 4.1, one could easily get an average version (because the average case is always no worse than the worst case). Hence, it is unclear to me why the author bothered to introduce another result to bound the average cost in Theorem 4.2. \n\n2. In Appendix D, the author spent much space explaining how to compute $\\nabla_W \\mathrm{cost}(x_{1:T})$. The reason is that the operator of projecting ML predictions into a robust set cannot be easily differentiated as typical neural network layers. So why bother to use this projection method? Why not use a primal-dual method that provides a projection-free alternative?\n\nLIMITATIONS:\nSee the weakness part and questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the SOCO problem with multi-step nonlinear switching costs and feedback delay. The authors propose the RCL algorithm, which projects the ML predictions on to a robust action set determined by the expert’s predictions. They show that RCL maintain the $(1+\\lambda)$-competitiveness against the expert while exploiting the potential of ML predictions. Moreover, they identify a sufficient condition under which the RCL achieves finite robustness and 1-consistency simultaneously. They also give an upper bound of the average cost of RCL, which reveals the advantage of training the ML model in a robustification-aware manner. Finally, they validate the theoretical results using a case study of battery management.\n\nSTRENGTHS:\n- Challenging setup. This paper considers SOCO with hitting cost feedback delay and multi-step nonlinear memory in the switching cost, which is very hard to deal with.\n- New algorithm design and new proof approach. The intuition behind their algorithm is clearly explained in line 232-240. Theorem 4.1 is proven by considering a new reservation cost, which decouples the dependency of the online action on the history.\n- Good theorems with adequate explanation. The paragraphs following Theorem 4.1, Corollary 4.1.1 and Theorem 4.2 give comprehensive analysis of the results.\n\nWEAKNESSES:\n- Assume known switching cost and smoothness constant. Convexity and smoothness assumptions on the hitting costs and switching costs.\n- The robustification-aware ML model is hard to be trained using standard back-propagation.\n\nQUESTIONS:\n- Few lines explaining why the cost difference and the switching cost difference can be upper bounded by $H$ and $G$ respectively would help understanding.\n- How to efficiently compute the robustified action set? If it cannot be efficiently computed, how you learn that set in your experiment?\n\nLIMITATIONS:\nThe authors have already mentioned in the paper (line 408-417).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper considers the smoothed online convex optimization problem, in which, in addition to the standard hitting cost incurred per step, the algorithm also incurs a switching cost which relates to changes in its chosen action. The paper considers switching costs which are multi-step (relate to the previous p time steps). 

The paper presents an ML-augmented algorithm for the problem, which given a possibly-inaccurate predictor and a robust expert, is able to utilize the predictions from the predictor to the degree to which they are accurate, while not exceeding the cost of the expert times some small factor. This is done by taking the predictions and projecting them onto a space of possible actions dictated by the expert. 

The paper also discusses how to adjust the training of an ML model to take into account the fact that the model's predictions are then projected onto the actions allowed by the expert.

STRENGTHS:
The model, and the concept underlying the algorithm, are natural. 
I liked the figure in Appendix A, perhaps it should be part of the main body of the paper.

WEAKNESSES:
In my opinion, the writing of the paper makes it hard to gain an intuitive understanding of the contributions of the paper within the scope of reasonable reviewing. For example, consider Theorem 4.1; excluding the robustness term, it is very hard to understand the significance of this theorem. Perhaps this bound seems simple and intuitive to experts more familiar with this problem.

QUESTIONS:
none

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-02

comments: SUMMARY:
This paper studies the  Smoothed Online Convex Optimization with  multi-step nonlinear switching costs and feedback delay. In this setting, they propose Robustness-Constrained Learning (RCL), which  combines  existing online algorithm with a novel reservation cost to robustify untrusted ML predictions. Theoretically, they provide both  worst-case and average-case guarantees  for RCL. Finally, they provide some experiments to evaluate RCL.

STRENGTHS:
1. The paper is well written and organized. The paper thoroughly explains the challenges of incorporating ML predictions in their setting and emphasizes the importance of developing novel algorithmic techniques to overcome these difficulties.

2. The theoretical results are sound and look rigorous. The worst-case bound of RCL is novel.

WEAKNESSES:
1. The weakness of the paper lies in the design of reservation costs, which may seem somewhat technical for analysis purposes. Additionally, the discussion on the optimization complexity for solving the constrained convex problem is missing, leaving a gap in understanding the practical implications of the proposed approach.

2. The paper's experimental results should be included in the main part to enhance confidence in the efficiency of RCL. Presenting empirical comparisons of RCL with other methods would demonstrate its advantages and substantiate the claims made in the paper. This would provide valuable evidence of the proposed algorithm's performance and practical benefits.

QUESTIONS:
See Weakness

LIMITATIONS:
There are no limitations with regards to negative societal impacts.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a problem, namely, Smoothed Online Convex Optimization (SOCO) with feedback delay and multi-step nonlinear memory, proposing a machine learning augmented online algorithm (RCL) with certain theoretical guarantees. This algorithm considers to implicitly hedge the decisions made by online and offline predictors, using a new constraint action set onto which the algorithm projects its decisions. This constraint set requires the decisions within it to achieve a trade-off between the hitting cost and the switching cost. By this design, this work achieves the first worst-case type cost bound for machine learning augmented algorithm under the given setting, and proposes a sufficient condition for the proposed algorithm to attain finite robustness and 1-consistency simultaneously. Moreover, the paper demonstrates that the algorithm RCL benefits from robustification-aware training and offers an average cost guarantee. Comprehensive experiments are executed to support these results.\n\nSTRENGTHS:\nThe writing of this paper is good with detailed explanations of the achieved results and the motivation is well substantiated by the experiments. The proposed algorithm is clear and easy to understand. The inherent nature of delayed feedback in the considered problem might pose an obstacle for algorithm design and analysis, and this paper proposes a solution for it.\n\nWEAKNESSES:\nI harbor some reservations regarding the applicability and scalability of the theoretical findings presented in this paper. Please refer to the \"Questions\" section below for further details.\n\nQUESTIONS:\n1.\tThis paper claims that it provides a sufficient condition to achieve the finite robustness and 1-consistency simultaneously (line 63). But Corollary 4.1.1 suggests that this condition might be applicable for the proposed specific algorithm only. Moreover, this condition requires a per round lower bound for the switching cost of the expert algorithm. Does this condition hold for a broad class of expert algorithms? Could the sufficient condition proposed in Corollary 4.1.1 be suitable for a more general ML-augmented algorithm to realize the best-of-both-world guarantee? \n2.\tI am confused about the term \"worst-case\" used in the discussion about Theorem 4.1. Based on my understanding, the worst-case bound should account for the upper bound of the minimax ratio. It would be beneficial if the authors could discuss about the relationship between the minimax bound and the bound in Theorem 4.1.\n3.\tThere are concerns regarding the existence and convexity of $\\mathcal{X}_t$. Does the solution of Eq. (1) below line 223 always exist? It appears from Corollary 4.1.1 that the action set $\\mathcal{X}$ is finite. If that is the case, then the \"robustified\" action set defined at line 223 may not exhibit convexity due to its discrete nature. Can the authors answer the above two questions?\n4.\tWould the authors be able to provide further discussion concerning the technical contributions made in addressing the issues related to feedback delay, especially in comparison with previous works?\n\nLIMITATIONS:\nNot much.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-24"}

comments: SUMMARY:
This paper studies Smoothed Online Convex Optimization with multi-step nonlinear switching costs and feedback delay. They propose an ML-augmented online algorithm named RCL that combines ML predictions with an expert online algorithm. The authors show that RCL can guarantee $(1 + \lambda)$ competitive ratio against any expert while improving the average-case performance. They show the effectiveness of RCL using battery management as a case study.

STRENGTHS:
1. Learning augmented algorithms is an important paradigm for designing more practical algorithms. Incorporating ML-augmented algorithms with multi-step nonlinear switching costs and feedback delay is a new and challenging problem.
2. The algorithm is simple and intuitive to understand. The authors provide a detailed analysis of the algorithm and the proof is rigorous.
3. This paper is well-written and easy to follow. The authors provide a detailed literature review and a clear introduction to the problem setting. The authors also provide a detailed discussion of the experimental results.

WEAKNESSES:
1. The algorithmic contributions of the paper are a bit weak. The idea of constructing reservation costs to hedge against any possible uncertainties and then solving a constrained convex problem to project the ML prediction into a robust action set is quite standard in ML-augmented algorithms. 
2. There lacks a lower bound on the trade-offs between robustness and average performance in ML-augmented algorithms.

QUESTIONS:
Is it possible for the reservation cost design to be overly conservative, resulting in the constrained optimization problem yielding an empty solution set?

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-25