SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Limitations and future work. We discuss two limitations of our work. First, we make a few assumptions (e.g., convexity and smoothness) on the hitting costs and switching costs. While they are common in the SOCO literature, these assumptions may not always hold in practice, and relaxing them will be interesting and challenging. Second, for online optimization, we only consider the ML prediction for the current step, whereas predictions for future steps can also be available in practice and may be explored for performance improvement. There are also a number of interesting open questions that follow from this work. For example, it is interesting to study alternative reservation costs and the optimality in terms of the tradeoff between bi-competitiveness. Additionally, it would also be interesting to extend RCL to other related problems such as convex body chasing and metrical task systems.
Acknowledgement. We would like to thank the anonymous reviewers for their helpful comments. Pengfei Li, Jianyi Yang and Shaolei Ren were supported in part by the U.S. NSF under the grant CNS-1910208. Adam Wierman was supported in part by the U.S. NSF under grants CNS-2146814, CPS-2136197, CNS-2106403, NGSDI-2105648.
[42] Guanya Shi, Xichen Shi, Michael O'Connell, Rose Yu, Kamyar Azizzadenesheli, Animashree Anandkumar, Yisong Yue, and Soon-Jo Chung. Neural lander: Stable drone landing control using learned dynamics. In International Conference on Robotics and Automation (ICRA), pages 9784-9790. IEEE, 2019.
[43] Guanya Shi. Competitive control via online optimization with memory, delayed feedback, and inexact predictions. In 2021 55th Annual Conference on Information Sciences and Systems (CISS), pages 1-1. IEEE, 2021.
[44] Ohad Shamir and Liran Szlak. Online learning with local permutations and delayed feedback. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 3086-3094. JMLR.org, 2017.
[45] Daan Rutten, Nico Christianson, Debankur Mukherjee, and Adam Wierman. Online optimization with untrusted predictions. CoRR, abs/2202.03519, 2022.
[46] Tongxin Li, Ruixiao Yang, Guannan Qu, Guanya Shi, Chenkai Yu, Adam Wierman, and Steven Low. Robustness and consistency in linear quadratic control with untrusted predictions. Proc. ACM Meas. Anal. Comput. Syst., 6(1), feb 2022.
[47] W. Tang, S. Bi, and Y. Zhang. Online coordinated charging decision algorithm for electric vehicles without future information. IEEE Trans. Smart Grid, 5:2810-2824, May 2014.
A Illustration of RCL
We illustrate the online optimization process of RCL in Fig. 1.
Figure 1: Robustness-constrained online optimization using RCL. The expert algorithm and ML model run independently. At each time  $t = 1, \dots, T$ , RCL projects the ML prediction  $\tilde{x}_t$  into a robustified action set.
B Case Study: Battery Management for EV Charging Stations
B.1 Problem Formulation
Batteries are routinely used in EV charging stations to handle the rapidly fluctuating charging demands and protect the connected grid. Thus, properly managing battery charging/discharging decisions is crucial for reliability, lifespan, and safety of batteries and grids.
We consider the management of  $N$  batteries. At each time step  $t$ , suppose that  $x_{t} \in \mathbb{R}_{+}^{N}$  represents the State of Charge (SoC) and  $u_{t} \in \mathbb{R}^{N}$  represents the battery charging/discharging schedule, depending on the sign of  $u_{t}$  (i.e., positive means charging, and vice versa). The canonical form of the battery dynamics can be written as  $x_{t + 1} = Ax_{t} + Bu_{t} - w_{t}$ , where  $A$  is a  $N \times N$  matrix which models the self-degradation of the  $N$ -battery system,  $B$  is a  $N \times N$  matrix which represents the charging efficiency of each battery unit,  $w_{t}$  is a  $N \times 1$  vector which denotes the current demand in terms of the charging rate (kW) of all the EVs connected to the charging stations. Assuming that the initial SoC as  $x_{0}$ , the goal is to control the batteries to minimize the difference between the current SoC of all batteries and a nominal value  $\bar{x}$ , plus a charging/discharging cost to account for battery usage [49, 50], which can be expressed mathematically as  $\min_{u_1, u_2, \dots, u_{T + 1}} \sum_{t = 1}^{T + 1} \| x_t - \bar{x} \|^2 + b \| u_t \|^2$ .
This problem falls into SOCO based on the reduction framework described in [49]. Specifically, at time step  $t + 1$ , we can expand  $x_{t + 1}$  based on the battery dynamics as  $x_{t + 1} = A^{t}x_{1} + \sum_{j = 1}^{t}A^{t - j}Bu_{j} - \sum_{j = 1}^{t}A^{t - j}w_{j}$ . We define the context parameter as  $y_{t} = \bar{x} - A^{t}x_{1} + \sum_{i = 1}^{t}A^{t - i}w_{i}$  and the action as  $a_{t} = \sum_{i = 1}^{t}A^{t - i}Bu_{i}$ . Then, assuming an identity matrix  $B$  (ignoring charging loss), the optimization problem becomes  $\min_{a_1,\dots ,a_T}\| x_1 - \bar{x}\|^2 +b\| u_T\|^2 +\sum_{t = 1}^T\| a_t - y_t\|^2 + b\| a_t - A a_{t - 1}\|^2$ . Given an initial value of  $x_{1}$ , this problem can be further simplified and reformulated as
$$
\min  _ {a _ {1}, a _ {2}, \dots , a _ {T}} \sum_ {t = 1} ^ {T} \frac {1}{b} \| a _ {t} - y _ {t} \| ^ {2} + \| a _ {t} - A a _ {t - 1} \| ^ {2}, \tag {7}
$$
which is in a standard SOCO form by considering  $y_{t}$  as the context and  $a_{t}$  as the action at time  $t$ .
To validate the effectiveness of RCL, we use a public dataset [51] provided by ElaadNL, a Dutch EV charging infrastructure company. We collect a dataset containing transaction records from ElaadNL charging stations in the Netherlands from January to June of 2019. Each transaction record contains the energy demand, transaction start time and charging time. As the data does not specify the details of battery units, we consider the battery units as a single combine battery by summing up the energy demand within each hour to obtain the hourly energy demand.
We use the January to February data as the training dataset, March to April data as the validation dataset for tuning the hyperparameters such as learning rate, and May to June as the testing dataset. We consider each problem instance as one day ( $T = 24$  hours, plus an initial action). Thus, a sliding
window of 25 is applied, moving one hour ahead each time, on the raw data to generate 1416 problem instances, where the first demand of each instance is used as the initial action of all the algorithms. We set  $b = 10$  and  $A = I$  for the cost function in Eqn. (7).
All the algorithms use the same ML architecture, when applicable, with the same initialized weights in our experiments for fair comparison. To be consistent with the literature [52, 53], all the ML models are trained offline. Specifically, we use a recurrent neural network (RNN) model that contains 2 hidden layers, each with 8 neurons, and implement the model using PyTorch. We train the RNN for 140 epochs with a batch size of 50. When the RNN model is trained as a standalone optimizer in a robustification-oblivious manner, the training process takes around 1 minute on a 2020 MacBook Air with 8GB memory and a M1 chipset. When RNN is trained in a robustification-aware manner, it takes around 2 minutes. The testing process is almost instant and takes less than 1 second.
By default, RCL uses a robustification-aware ML model due to the advantage of average cost performance compared to a robustification-oblivious model. We compare RCL with several representative baseline algorithms as summarized below.
- Offline Optimal Oracle (OPT): This is the optimal offline algorithm that has all the contextual information and optimally solves the problem.
- Regularized Online Balanced Descent (ROBD): ROBD is the state-of-the-art order-optimal online algorithm with the best-known competitive ratio for our SOCO setting [54, 49]. The parameters of ROBD are all optimally set according to [49]. By default, RCL uses ROBD as its expert for robustness.
- Hitting Cost Minimizer (HitMin): HitMin is a special instance of ROBD by setting the parameters such that it greedily minimizing the hitting cost at each time. This minimizer can be empirically effective and hence also used in ROBD as a regularizer.
- Machine Learning Only (ML): ML is trained as a standalone optimizer in a robustification-oblivious manner. It does not use robustification during online optimization.
- Expert-Calibrated Learning (EC-L20): It is an ML-augmented algorithm that applies to our SOCO setting by using an ML model to regularize online actions without robustness guarantees [55]. We set its parameters based on the validation dataset to have the optimal average performance with an empirical competitive ratio less than  $(1 + \lambda)CR^{\pi}$ .
- RCL with a robustification-oblivious ML model (RCL-0): To differentiate the two forms of RCL, we use RCL to refer to RCL with a robustification-aware ML model and RCL-0 for the robustification-oblivious ML model, where “-0” represents robustification-obliviousness.
To highlight our key contribution to the SOCO literature, the baseline algorithms we choose are representative of the state-of-the-art expert algorithms, effective heuristics, and ML-augmented algorithms for the SOCO setting we consider. While there are a few other ML-augmented algorithms for SOCO [56, 57, 58], they do not apply to our problem as they consider unsquared switching costs in a metric space and exploit the natural triangular inequality. Adapting them to the squared switching costs is non-trivial.
B.3 Additional Empirical Results
In Section 5, we have evaluated RCL using ROBD as the expert online algorithm with  $\lambda = 1$ . Here we provide additional experiment results to further evaluate the effectiveness of RCL, and the results are organized as follows. First, we change the expert online algorithm from ROBD to HitMin to show the flexibility of RCL. Second, we quantitatively present the effect of parameter  $\lambda$  in RCL. Third, we introduce additional out-of-distribution samples to test the robustness of RCL and baselines, in terms of competitive ratio. Finally, we experiment under the delayed-feedback setting, which is a more challenging problem setup.
B.3.1 Utilizing HitMin as the expert
RCL is flexible and can work with any expert online algorithm, even an expert that does not have good or bounded competitive ratios. Thus, it is interesting to see how RCL performs given an alternative
(a) ROBD as the expert
(b) HitMin as the expert
Figure 2: Cost ratio distributions  $(\lambda = 1$  by default).
(c) RCL-0 w/ different  $\lambda$
(d) RCL w/ different  $\lambda$
expert. For example, in Table 1, HitMin empirically outperforms ROBD in terms of the average, although it is not as robust as ROBD. Thus, using  $\lambda = 1$ , we leverage HitMin as the expert for RCL and RCL-0, and show the cost ratio distributions in Fig. 2(b). Comparing Fig. 2(b) with Fig. 2(a), we see that RCL and RCL-0 both have many low cost ratios by using HitMin as the expert, but the worst case for RCL is not as good as when using ROBD as the expert. For example, the average cost and competitive ratio are 1.0515 and 1.6035, respectively, for RCL. This result is not surprising, as the new expert HitMin has a better average performance but worse competitive ratio than the default expert ROBD.
B.3.2 Impact of  $\lambda$
Theorem 4.1 shows the point that we need to set a large enough  $\lambda$  in order to provide enough flexibility for RCL to exploit good ML predictions. With a small  $\lambda >0$ , despite the stronger competitiveness against the expert, it is possible that RCL may even empirically perform worse than both the ML model and the expert. Thus, we now investigate the impact of  $\lambda$ .
We see from Table 1 that the empirical average cost and competitive ratio of RCL are both worse with  $\lambda = 0.6$  than with the default  $\lambda = 1$ . More interestingly, by setting  $\lambda = 5$ , the average cost of RCL is even lower than that of ML. This is because ML in our experiment performs fairly well on average. Thus, by setting a large  $\lambda = 5$ , RCL is able to exploit the benefits of good ML predictions for many typical cases, while using the expert ROBD as a safeguard to handle a few bad problem instances for which ML cannot perform well. Also, the empirical competitive ratio of RCL is better with  $\lambda = 5$  than with  $\lambda = 3$ , supporting Theorem 4.1 that a larger  $\lambda$  may not necessarily increase the competitive ratio as RCL can exploit good ML predictions. In addition, given each  $\lambda$ , RCL outperforms RCL-0, which highlights the importance of training the ML model in a robustification-aware manner to avoid the mismatch between training and testing objectives.
We also show in Fig. 2(c) an Fig. 2(d) the cost ratio distributions for RCL-0 and RCL, respectively, under different  $\lambda$ . The results reaffirm our main Theorem 4.1 as well as the importance of training the ML model in a robustification-aware manner.
Next, we show the bi-competitive cost ratios of RCL-0 against both the expert ROBD and the ML predictions. We focus on RCL-0 as its ML model is trained as a standalone optimizer, whereas RCL uses a robustification-aware ML model that is not specifically trained to produce good pre-robustification predictions. According to Theorem 4.1, RCL-0 obtains a potentially better competitiveness against ML but a worse competitive against the expert ROBD when  $\lambda$  increases, and vice versa. To further validate the theoretical analysis, we test RCL-0 with different  $\lambda$  and obtain the 2D histogram of its bi-competitive cost ratios against ROBD and ML, respectively. The results are shown in Fig. 3. In agreement with our analysis, the cost ratio of RCL-0 against ROBD never exceeds  $(1 + \lambda)$  for any  $\lambda > 0$ . Also, with a small  $\lambda = 0.6$ , the cost ratio of RCL-0 against ROBD concentrates around 1, while it does not exploit the benefits of ML predictions very well. On the other hand, with a large  $\lambda = 5$ , the cost ratio of RCL-0 against ROBD can be quite high, although it follows (good) ML predictions more closely for better average performance. Most importantly, by increasing  $\lambda > 0$ , we can see the general trend that RCL-0 follows the ML predictions more closely while still being able to guarantee competitiveness against ROBD. Again, this confirms the key point of our main insights in Theorem 4.1.
(a)  $\lambda = 0.6$
(b)  $\lambda = 1.0$
Figure 3: Histogram of bi-competitive cost ratios of RCL-0 (against ROBD and ML) under different  $\lambda$ . For better visualization, the color map represents logarithmic values of the cost ratio histogram with a base of 10.
(c)  $\lambda = 5.0$
(d)  $\lambda = \infty$  (i.e., ML)
\(p_c=0.05\)	\(p_c=0.1\)	\(p_c=0.2\)
\(\sigma = 0.06\)	\(\sigma = 0.08\)	\(\sigma = 0.1\)	\(\sigma = 0.06\)	\(\sigma = 0.08\)	\(\sigma = 0.1\)	\(\sigma = 0.06\)	\(\sigma = 0.08\)	\(\sigma = 0.1\)
AVG	RCL	1.1331	1.1444	1.1556	1.1487	1.1693	1.1904	1.1827	1.2254	1.2697
RCL-0	1.2425	1.2436	1.2462	1.2416	1.2434	1.2478	1.2370	1.2394	1.2469
ML	1.0722	1.0778	1.0855	1.0770	1.0874	1.1018	1.0858	1.1053	1.1325
EC-L2O	1.1728	1.1737	1.1754	1.1731	1.1750	1.1784	1.1727	1.1757	1.1815
ROBD	1.6048	1.6048	1.6048	1.6048	1.6049	1.6049	1.6048	1.6048	1.6049
HitMin	1.2112	1.2195	1.2302	1.2202	1.2357	1.2557	1.2410	1.2724	1.3127
CR	RCL	2.5028	2.9697	3.2247	2.6553	3.0283	3.2711	2.5714	3.0123	3.1653
RCL-0	2.4209	2.4209	2.4209	2.4209	2.4209	2.4209	2.4209	2.4209	2.4209
ML	6.5159	8.9245	11.6627	4.4025	6.5090	9.4168	5.5798	7.3956	9.3903
EC-L2O	3.4639	4.6034	5.9666	2.6545	3.6740	5.1129	2.9766	3.7713	4.6983
ROBD	1.7291	1.7291	1.7291	1.7291	1.7291	1.7291	1.7291	1.7296	1.7298
HitMin	4.8573	6.7746	8.8383	3.1492	4.8253	7.0405	5.0632	6.9699	8.9246
Table 2: Average cost and competitive ratio comparison of different algorithms. We study the effect of introducing out-of-distribution (OOD) samples. Within the testing dataset, we randomly select a fraction of  $p_c$  of samples and add some random noise following  $\mathcal{N}(0,\sigma)$  to contaminate these data samples (whose input values are all normalized within [0,1]).
B.3.3 Larger distributional shifts
In our dataset, ML performs very well on average as the testing distribution matches well with its training distribution. To consider more challenging cases as a stress test, we manually increase the testing distributional shifts by adding random noise following  $\mathcal{N}(0,\sigma)$  to a certain facton  $p_c$  of the testing samples. Note that, as we intentionally stress test RCL and RCL-0 under a larger distributional shift, their ML models remain unchanged as in the default setting and are not re-trained by adding noisy data to the training dataset.
With the default  $\lambda = 1$ , we show the average cost and competitive ratio results in Table 2. We see that ROBD is very robust and little affected by the distributional shifts. In terms of the competitive ratio, ML, HitMin and EC-L2O are not robust, resulting in a large competitive ratio when we add more noisy samples. The average cost performance of RCL is empirically better than that of RCL-0 in almost all cases, except for a slight increase in the practically very rare case where  $20\%$  samples are contaminated with large noise. On the other hand, as expected, the competitive ratios of RCL and RCL-0 both increase as we add more noise. While RCL has a higher competitive ratio than RCL-0 empirically in the experiment, they both have the same guaranteed  $(1 + \lambda)$  competitiveness against ROBD regardless of how their ML models are trained. Also, their competitive ratios are both better than other algorithms, showing the effectiveness of our novel robustification process.
B.4 Results with Feedback Delay
We now turn to the case when there is a one-step feedback delay, i.e., the context parameter  $y_{t}$  is not known to the agent until time  $t + 1$ . For this setting, we consider the best-known online algorithm iROBD [49] as the expert that handles the feedback delay with a guaranteed competitive ratio with respect to OPT. The other baseline online algorithms — ROBD, EC-L2O, and HitMin— presented in Section B.2 require the immediate revelation of  $y_{t}$  without feedback delay and hence do not directly apply to this case. Thus, for comparison, we use the predicted context, denoted by  $\hat{y}_t$ , with up to  $15\%$  prediction errors in the baseline online algorithms, and reuse the algorithm names (e.g., EC-L2O uses predicted  $\hat{y}_t$  as if it were the true context for decision making). We train ML using the same
RCL	RCL-D	ML	EC-L2O	iROBD	HitMin	ROBD
λ=0.6	λ=1	λ=3	λ=5	λ=0.6	λ=1	λ=3	λ=5
AVG	1.5011	1.3594	1.2874	1.2899	1.5134	1.3690	1.2949	1.3026	1.2792	1.4112	2.3076	2.6095	2.5974
CR	2.9797	2.4832	3.2049	3.9847	2.9797	2.4832	3.3367	4.3040	8.4200	15.1928	4.7632	26.0264	2.8478
Table 3: Competitive ratio and average cost comparison of different algorithms with feedback delay.
(a) iROBD as the expert
(b) RCL w/ different  $\lambda$
(c) RCL-0 w/ different  $\lambda$
Figure 4: Cost ratio distributions with feedback delay ( $\lambda = 1$  by default)
architecture as in Section B.3, with the exception that only delayed context is provided as input for both training and testing. The reported values are normalized with respect to those of the respective offline optimal algorithm OPT. The average cost (AVG) and competitive ratio (CR) are all empirical results reported on the testing dataset.
We show the results in Table 3 and Fig. 4. We see that with the default  $\lambda = 1$ , both RCL and RCL-0 have a good average cost, but RCL has a lower average cost than RCL-0 and is outperformed only by ML in terms of the average cost. RCL and RCL-0 have the same competitive ratio guarantee (i.e.,  $(1 + \lambda)$  times the competitive ratio of iROBD). Nonetheless, RCL has the lowest competitive ratio than all the other algorithms, demonstrating the power of RCL to leverage both ML prediction and the robust expert. In this experiment, both RCL and RCL-0 outperform iROBD in terms of the empirical competitive ratio because they are able to exploit the good ML predictions for those problem instances that are difficult for iROBD.
By comparison, ML performs well on average by exploiting the historical data, but has a high competitive ratio. The alternative baselines — ROBD, EC-L2O and HitMin— use predicted context  $\hat{y}_t$  as the true context. Except for the good empirical competitive ratio of ROBD, they do not have good average performance or guaranteed robustness due to their naively trusting the predicted context (that can potentially have large prediction errors). Note that the empirical competitive ratio of ROBD with predicted context is still much higher than that with the true context in Table 1. These results reinforce the point that blindly using ML predictions (i.e., predicted context in this example) without additional robustification can lead to poor performance in terms of both average cost and worst-case cost ratio.
We further show in Fig. 4 the box plots for cost ratios of different algorithms, providing a detailed view of the algorithms' performance. The key message is that RCL obtains the best of both worlds — a good average cost and a good competitive ratio. Moreover, we see that by setting  $\lambda = 1$ , we provide enough freedom to RCL to exploit the benefits of ML predictions while also ensuring worst-case robustness. Thus, like in the no-delay case in Table 1 and Fig. 2, the empirical competitive ratio of RCL with  $\lambda = 1$  is even lower than that with  $\lambda = 0.6$ .
C Proof of Theorems and Corollaries in Section 4
C.1 Proof of Theorem 4.1 (Cost Ratio)
To prove Theorem 4.1, we first give some technical lemmas about the smoothness of cost functions from Lemma C.1 to Lemma C.3.
Lemma C.1 (Lemma 4 in [59]). Assume  $f(x)$  is  $\beta$  smooth, for any  $\lambda > 0$ , we have
$$
f (x) \leq (1 + \lambda) f (y) + (1 + \frac {1}{\lambda}) \frac {\beta}{2} \| x - y \| ^ {2} \quad \forall x, y \in \mathcal {X}
$$
Lemma C.2. Assume  $f(x)$  is  $\beta_{1}$  smooth and  $d(x)$  is  $\beta_{2}$  smooth, then  $f(x) + d(x)$  is  $\beta_{1} + \beta_{2}$  smooth.
Lemma C.3. Suppose that the hitting cost  $f(x, y_{t})$  is  $\beta_{h}$ -smooth with respect to  $x$ . The switching cost is  $d(x_{t}, x_{t-1}) = \frac{1}{2} \|x_{t} - \delta(x_{t-p:t-1})\|^{2}$ , where  $\delta(\cdot)$  is  $L_{i}$ -Lipschitz with respect to  $x_{t-i}$ . Then for any two action sequences  $x_{1:T}$  and  $x_{1:T}'$ , we must have
$$
\operatorname {c o s t} \left(x _ {1: T}\right) - (1 + \lambda) \operatorname {c o s t} \left(x _ {1: T} ^ {\prime}\right) \leq \frac {\beta + \left(1 + \sum_ {k = 1} ^ {p} L _ {k}\right) ^ {2}}{2} \left(1 + \frac {1}{\lambda}\right) \| x _ {1: T} - x _ {1: T} ^ {\prime} \| ^ {2}, \quad \forall \lambda > 0 \tag {8}
$$
Proof. The objective to be bounded can be decomposed as
$$
\begin{array}{l} \operatorname {c o s t} \left(x _ {1: T}\right) - (1 + \lambda) \operatorname {c o s t} \left(x _ {1: T} ^ {\prime}\right) \\ = \left(\sum_ {t = 1} ^ {T} f \left(x _ {t}, y _ {t}\right) - (1 + \lambda) f \left(x _ {t} ^ {\prime}, y _ {t}\right)\right) + \frac {1}{2} \left(\sum_ {t = 1} ^ {T} \| x _ {t} - \delta \left(x _ {t - p: t - 1}\right) \| ^ {2} - (1 + \lambda) \| x _ {t} ^ {\prime} - \delta \left(x _ {t - p: t - 1} ^ {\prime}\right) \| ^ {2}\right) \tag {9} \\ \end{array}
$$
Since hitting cost is  $\beta_{h}$ -smooth, then
$$
\sum_ {t = 1} ^ {T} f \left(x _ {t}, y _ {t}\right) - (1 + \lambda) f \left(x _ {t} ^ {\prime}, y _ {t}\right) \leq \frac {\beta_ {h}}{2} \left(1 + \frac {1}{\lambda}\right) \sum_ {t = 1} ^ {T} \| x _ {t} - x _ {t} ^ {\prime} \| ^ {2} \tag {10}
$$
Besides, based on the Lipschitz assumption of function  $\delta (\cdot)$ , we have