SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tongxin Li
CUHK-SZ, China
litongxin@cuhk.edu.cn
Yiheng Lin
Computing + Mathematical Sciences
Caltech, USA
yiheng1@caltech.edu
UC Riverside, USA
shaolei@ucr.edu
adamw@caltech.edu
We study the tradeoff between consistency and robustness in the context of a single-trajectory time-varying Markov Decision Process (MDP) with untrusted machine-learned advice. Our work departs from the typical approach of treating advice as coming from black-box sources by instead considering a setting where additional information about how the advice is generated is available. We prove a first-of-its-kind consistency and robustness tradeoff given Q-value advice under a general MDP model that includes both continuous and discrete state/action spaces. Our results highlight that utilizing Q-value advice enables dynamic pursuit of the better of machine-learned advice and a robust baseline, thus result in near-optimal performance guarantees, which provably improves what can be obtained solely with black-box advice.
Machine-learned predictions and hand-crafted algorithmic advice are both crucial in online decision-making problems, driving a growing interest in learning-augmented algorithms [1,2] that exploit the benefits of predictions to improve the performance for typical problem instances while bounding the worst-case performance [3,4]. To this point, the study of learning-augmented algorithms has primarily viewed machine-learned advice as potentially untrusted information generated by black-box models. Yet, in many real-world problems, additional knowledge of the machine learning models used to produce advice/predictions is often available and can potentially improve the performance of learning-augmented algorithms.
A notable example that motivates our work is the problem of minimizing costs (or maximizing rewards) in a single-trajectory Markov Decision Process (MDP). More concretely, a value-based machine-learned policy  $\widetilde{\pi}$  can be queried to provide suggested actions as advice to the agent at each step [5-7]. Typically, the suggested actions are chosen to minimize (or maximize, in case of rewards) estimated cost-to-go functions (known as Q-value predictions) based on the current state.
Naturally, in addition to suggested actions, the Q-value function itself can also provide additional information (e.g., the long-term impact of choosing a certain action) potentially useful to the design of a learning-augmented algorithm. Thus, this leads to two different designs for learning-augmented algorithms in MDPs: black-box algorithms and grey-box algorithms. A learning-augmented algorithm using  $\widetilde{\pi}$  is black-box if  $\widetilde{\pi}$  provides only the suggested action  $\widetilde{u}$  to the learning-augmented algorithm,
whereas it is value-based (a.k.a., grey-box) if  $\widetilde{\pi}$  provides an estimate of the Q-value function  $\widetilde{Q}$  (that also implicitly includes a suggested action  $\widetilde{u}$  obtained by minimizing  $\widetilde{Q}$ ) to the learning-augmented algorithm.
Value-based policies  $\widetilde{\pi}$  often perform well empirically in stationary environments in practice [5, 6]. However, they may not have performance guarantees in all environments and can perform poorly at times due to a variety of factors, such as non-stationary environments [8-11], policy collapse [12], sample inefficiency [13], and/or when training data is biased [14]. As a consequence, such policies often are referred to as "untrusted advice" in the literature on learning-augmented algorithms, where the notion of "untrusted" highlights the lack of performance guarantees. In contrast, recent studies in competitive online control [15-21] have begun to focus on worst-case analysis and provide control policies  $\overline{\pi}$  with strong performance guarantees even in adversarial settings, referred to as robustness, i.e.,  $\overline{\pi}$  provides "trusted advice." Typically, the goal of a learning-augmented online algorithm [13] is to perform nearly as well as the untrusted advice when the machine learned policy performs well, a.k.a., achieve consistency, while also ensuring worst-case robustness. Combining the advice of an untrusted machine-learned policy  $\widetilde{\pi}$  and a robust policy  $\overline{\pi}$  naturally leads to a tradeoff between consistency and robustness. In this paper, we explore this tradeoff in a time-varying MDP setting and seek to answer the following key question for learning-augmented online algorithms:
Can  $Q$ -value advice from an untrusted machine-learned policy,  $\widetilde{\pi}$ , in a grey-box scenario provide more benefits than the black-box action advice generated by  $\widetilde{\pi}$  in the context of consistency and robustness tradeoffs for MDPs?
We answer the question above in the affirmative by presenting and analyzing a unified projection-based learning-augmented online algorithm (PROjection Pursuit policy, simplified as PROP in Algorithm [1]) that combines action feedback from a trusted, robust policy  $\overline{\pi}$  with an untrusted ML policy  $\widetilde{\pi}$ . In addition to offering a consistency and robustness tradeoff for MDPs with black-box advice, our work moves beyond the black-box setting. Importantly, by considering the grey-box setting, the design of PROP demonstrates that the structural information of the untrusted machine-learned advice can be leveraged to determine the trust parameters dynamically, which would otherwise be challenging (if not impossible) in a black-box setting. To our best knowledge, PROP is the first-of-its-kind learning-augmented algorithm that applies to general MDP models, which allow continuous or discrete state and action spaces.
Our main results characterize the tradeoff between consistency and robustness for both black-box and grey-box settings in terms of the ratio of expectations, RoE, built upon the traditional consistency and robustness metrics in [3, 22, 23, 4] for the competitive ratio. We show in Theorem 5.2 that for the black-box setting, PROP is  $(1 + \mathcal{O}((1 - \lambda)\gamma))$ -consistent and  $(\mathrm{ROB} + \mathcal{O}(\lambda \gamma))$ -robust where  $0 \leq \lambda \leq 1$  is a hyper-parameter. Moreover, for the black-box setting, PROP cannot be both  $(1 + o(\lambda \gamma))$ -consistent and  $(\mathrm{ROB} + o((1 - \lambda)\gamma))$ -robust for any  $0 \leq \lambda \leq 1$  where  $\gamma$  is the diameter of the action space. In sharp contrast, by using a careful design of a robustness budget parameter in PROP with Q-value advice (grey-box setting), PROP is 1-consistent and  $(\mathrm{ROB} + o(1))$ -robust.
Our result highlights the benefits of exploiting the additional information informed by the estimated Q-value functions, showing that the ratio of expectations can approach the better of the two policies  $\widetilde{\pi}$  and  $\overline{\pi}$  for any single-trajectory time-varying, and even possibly adversarial environments — if the value-based policy  $\widetilde{\pi}$  is near-optimal, then the worst-case RoE(PROP) can approach 1 as governed by a consistency parameter; otherwise, RoE(PROP) can be bounded by the ratio of expectations of  $\overline{\pi}$  subject to an additive term  $o(1)$  that decreases when the time horizon  $T$  increases.
A key technical contribution of our work is to provide the first quantitative characterization of the consistency and robustness tradeoff for a learning-augmented algorithm (PROP) in a general MDP model, under both standard black-box and novel grey-box settings. Importantly, PROP is able to leverage a broad class of robust policies, called Wasserstein robust policies, which generalize the well-known contraction principles that are satisfied by various robust policies [24] and have been used to derive regrets for online control [19, 25]. A few concrete examples of Wasserstein robust policies applicable for PROP are provided in Table I (Section 3.1).
Learning-Augmented Algorithms with Black-Box Advice. The concept of integrating black-box machine-learning guidance into online algorithms was initially introduced by [26]. [3] coined terms "robustness" and "consistency" with formal mathematical definitions based on the competitive ratio. Over the past few years, the consistency and robustness approach has gained widespread popularity and has been utilized to design online algorithms with black-box advice for various applications, including ski rental [3, 22, 23], caching [27-29], bipartite matching [30], online covering [31, 32], convex body chasing [4], nonlinear quadratic control [33]. The prior studies on learning-enhanced algorithms have mainly focused on creating meta-strategies that combine online algorithms with black-box predictions, and typically require manual setting of a trust hyper-parameter to balance consistency and robustness. A more recent learning-augmented algorithm in [33] investigated the balance between competitiveness and stability in nonlinear control in a black-box setting. However, this work limits the robust policy to a linear quadratic regulator and does not provide a theoretical basis for the selection of the trust parameters. [34] generalized the black-box advice setting by considering distributional advice.
Online Control and Optimization with Structural Information. Despite the lack of a systematic analysis, recent studies have explored the usage of structural information in online control and optimization problems. Closest to our work, [7] considered a related setting where the Q-value function is available as advice, and shows that such information can be utilized to reduce regret in a tabular MDP model. In contrast, our analysis applies to more general models that allow continuous state/action spaces. In [17], the dynamical model and the predictions of disturbances in a linear control system are shown to be useful in achieving a near-optimal consistency and robustness tradeoff. The predictive optimization problem solved by MPC [35, 36, 16, 37] can be regarded as a special realization of grey-box advice, where an approximated cost-to-go function is constructed from structural information that includes the (predicted) dynamical model, costs, and disturbances.
MDP with External Feedback. Feedback from external sources such as control baselines [38, 39], visual explanations [40], and human experts [41-43] is often available in MDP. This external feedback can be beneficial for various purposes, such as ensuring safety [44], reducing variance [38], training human-like chatbots [41], and enhancing overall trustworthiness [45], among others. The use of control priors has been proposed by [38] as a way to guarantee the Lyapunov stability of the training process in reinforcement learning. They used the Temporal-Difference method to tune a coefficient that combines a RL policy and a control prior, but without providing a theoretical foundation. Another related area is transfer learning in RL, where external Q-value advice from previous tasks can be adapted and utilized in new tasks. Previous research has shown that this approach can outperform an agnostic initialization of Q, but these results are solely based on empirical observations and lack theoretical support [46-48].
We consider a finite-horizon, single-trajectory, time-varying MDP with  $T$  discrete time steps. The state space  $\mathcal{X}$  is a subset of a normed vector space embedded with a norm  $\| \cdot \|_{\mathcal{X}}$ . The actions are chosen from a convex and compact set  $\mathcal{U}$  in a normed vector space characterized by some norm  $\| \cdot \|_{\mathcal{U}}$ . Notably,  $\mathcal{U}$  can represent either continuous actions or the probability distributions used when choosing actions from a finite set. The diameter of the action space  $\mathcal{U}$  is denoted by  $\gamma := \max_{u \in \mathcal{U}} \| u \|_{\mathcal{U}}$ . Denote  $[T] := \{0, \dots, T - 1\}$ . For each time step  $t \in [T]$ , let  $P_t: \mathcal{X} \times \mathcal{U} \to \mathcal{P}_{\mathcal{X}}$  be the transition probability, where  $\mathcal{P}_{\mathcal{X}}$  is a set of probability measures on  $\mathcal{X}$ . We consider time-varying costs  $c_t: \mathcal{X} \times \mathcal{U} \to \mathbb{R}_+$ , while rewards can be treated similarly by adding a negative sign. An initial state  $x_0 \in \mathcal{X}$  is fixed. This MDP model is compactly represented by  $\mathrm{MDP}(\mathcal{X}, \mathcal{U}, T, P, c)$ .
The goal of a policy in this MDP setting is to minimize the total cost over all  $T$  steps. The policy agent has no access to the full MDP. At each time step  $t \in [T]$ , only the incurred cost value  $c_t(x_t, u_t)$  and the next state  $x_{t+1} \sim P_t(\cdot | x_t, u_t)$  are revealed to the agent after playing an action  $u_t \in \mathcal{U}$ . We denote a policy by  $\pi = (\pi_t : t \in [T])$  where each  $\pi_t : \mathcal{X} \to \mathcal{U}$  chooses an action  $u_t$  when observing
<sup>1</sup>The action space  $\mathcal{U}$  is assumed to be a continuous, convex, and compact set for more generality. When the actions are discrete,  $\mathcal{U}$  can be defined as the set of all probability distributions on a finite action space. We relegate the detailed discussions in Appendix A.2 and F
$x_{t}$  at step  $t\in [T]$ . Note that our results can be generalized to the setting when  $\pi_t$  is stochastic and outputs a probability distribution on  $\mathcal{U}$ . Given  $\mathrm{MDP}(\mathcal{X},\mathcal{U},T,P,c)$ , we consider an optimization with time-varying costs and transition dynamics. Thus, our goal is to find a policy  $\pi$  that minimizes the following expected total cost:
$$
J (\pi) := \mathbb {E} _ {P, \pi} \left[ \sum_ {t \in [ T ]} c _ {t} \left(x _ {t}, \pi_ {t} \left(x _ {t}\right)\right) \right] \tag {1}
$$
where the randomness in  $\mathbb{E}_{P,\pi}$  is from the transition dynamics  $P = (P_{t}:t\in [T])$  and the policy  $\pi = (\pi_t:t\in [T])$ . We focus our analysis on the expected dynamic regret and the ratio of expectations, defined below, as the performance metrics for our policy design.
Definition 1 (Expected dynamic regret). Given  $\mathrm{MDP}(\mathcal{X},\mathcal{U},T,P,c)$ , the (expected) dynamic regret of a policy  $\pi = (\pi_t:t\in [T])$  is defined as the difference between the expected cost induced by the policy  $\pi$ ,  $J(\pi)$  in (1), and the optimal expected cost  $J^{\star}\coloneqq \inf_{\pi}J(\pi)$ , i.e.,  $\mathrm{DR}(\pi)\coloneqq J(\pi) - J^{\star}$ .
Dynamic regret is a more general (and often more challenging to analyze) measure than classical static regret, which has been mostly used for stationary environments [49, 50]. The following definition of the ratio of expectations [51, 52] will be used as an alternative performance metric in our main results.
Definition 2 (Ratio of expectations). Given MDP  $(\mathcal{X},\mathcal{U},T,P,c)$ , the ratio of expectations of a policy  $\pi = (\pi_t:t\in [T])$  is defined as  $\mathsf{RoE}(\pi)\coloneqq J(\pi) / J^{\star}$  where  $J(\pi)$  and  $J^{\star}$  are the same as in Definition
Dynamic regret and the ratio of expectations defined above also depend on the error of the untrusted ML advice; we make this more explicit in Section 3.2. Next, we state the following continuity assumption, which is standard in MDPs with continuous action and state spaces [53-55]. Note that our analysis can be readily adapted to general Hölder continuous costs with minimal modifications.
Assumption 1 (Lipschitz costs). For any time step  $t \in [T]$ , the cost function  $c_t: \mathcal{X} \times \mathcal{U} \to \mathbb{R}_+$  is Lipschitz continuous with a Lipschitz constant  $L_{\mathrm{C}} < \infty$ , i.e., for any  $t \in [T]$ ,  $|c_t(x, u) - c_t(x', u')| \leq L_{\mathcal{C}}$  ( $\| x - x' \|_{\mathcal{X}} + \| u - u' \|_{\mathcal{U}}$ ). Moreover,  $0 < c_t(x, u) < \infty$  for all  $t \in [T]$ ,  $x \in \mathcal{X}$ , and  $u \in \mathcal{U}$ .
3 Consistency and Robustness in MDPs
Figure 1: Left: Overview of settings in our problem. Right: consistency and robustness tradeoff, with RoE and  $\varepsilon$  defined in Definition 2 and Equation (4).
Our objective is to achieve a balance between the worst-case guarantees on cost minimization in terms of dynamic regret provided by a robust policy,  $\overline{\pi}$ , and the average-case performance of a valued-based policy,  $\widetilde{\pi}$ , in the context of MDP  $(\mathcal{X},\mathcal{U},T,P,c)$ . In particular, we denote by ROB  $\geq 1$  a ratio of expectation bound of the robust policy  $\overline{\pi}$  such that the worst case RoE  $(\overline{\pi})\leq$  ROB. In the learning-augmented algorithms literature, these two goals are referred to as consistency and robustness [3, 1]. Informally, robustness refers to the goal of ensuring worst-case guarantees on cost minimization comparable to those provided by  $\overline{\pi}$  and consistency refers to ensuring performance nearly as good as  $\widetilde{\pi}$  when  $\widetilde{\pi}$  performs well (e.g., when the instance is not adversarial). Learning-augmented algorithms seek to achieve consistency and robustness by combining  $\overline{\pi}$  and  $\widetilde{\pi}$ , as illustrated in Figure 1.
Table 1: Examples of models covered in this paper and the associated control baselines. For the right column, bounds on the ratio of expectations RoE are exemplified, where ROB is defined in Section 3 and  $\mathcal{O}$  omits inessential constants.
Model	Robust Baseline π	RoE
Time-varying MDP (Our General Model)	Wasserstein Robust Policy (Definition 3)	ROB
Discrete MDP (Appendix A.2)	Any Policy that Induced a Regular Markov Chain
Time-Varying LQR (Appendix A.1)	MPC with Robust Predictions (Algorithm 3)	O(1)
Our focus in this work is to design robust and consistent algorithms for two types of advice: black-box advice and grey-box advice. The type of advice that is nearly always the focus in the learning-augmented algorithm literature is black-box advice — only providing a suggested action  $\widetilde{u}_t$  without additional information. In contrast, on top of the action  $\widetilde{u}_t$ , grey-box advice can also reveal the internal state of the learning algorithm, e.g., the Q-value  $\widetilde{Q}_t$  in our setting. This contrast is illustrated in Figure 1.
Compared to black-box advice, grey-box advice has received much less attention in the literature, despite its potential to improve tradeoffs between consistency and robustness as recently shown in [34, 17]. Nonetheless, the extra information on top of the suggested action in a grey-box setting potentially allows the learning-augmented algorithm to make a better-informed decision based on the advice, thus achieving a better tradeoff between consistency and robustness than otherwise possible.
In the remainder of this section, we discuss the robustness properties for the algorithms we consider in our learning-augmented framework (Section 3.1), and introduce the notions of consistency in our grey-box and black-box models in Section 3.2.
3.1 Locally Wasserstein-Robust Policies
We begin with constructing a novel notion of robustness for our learning-augmented framework based on the Wasserstein distance as follows. Denote the robust policy by  $\overline{\pi} \coloneqq (\overline{\pi}_t : t \in [T])$ , where each  $\overline{\pi}_t$  maps a system state to a deterministic action (or a probability of actions in the stochastic setting). Denote by  $\rho_{t_1:t_2}(\rho)$  the joint distribution of the state-action pair  $(x_t, u_t) \in \mathcal{X} \times \mathcal{U}$  at time  $t_2 \in [T]$  when implementing the baselines  $\overline{\pi}_{t_1}, \ldots, \overline{\pi}_{t_2}$  consecutively with an initial state-action distribution  $\rho$ . We use  $\| \cdot \|_{\mathcal{X} \times \mathcal{U}} \coloneqq \| \cdot \|_{\mathcal{X}} + \| \cdot \|_{\mathcal{U}}$  as the included norm for the product space  $\mathcal{X} \times \mathcal{U}$ . Let  $W_p(\mu, \nu)$  denote the Wasserstein  $p$ -distance between distributions  $\mu$  and  $\nu$  whose support set is  $\mathcal{X} \times \mathcal{U}$ :
$$
W _ {p} (\mu , \nu) := \left(\inf  _ {J \in \mathcal {J} (\mu , \nu)} \int \| (x, u) - \left(x ^ {\prime}, u ^ {\prime}\right) \| _ {\mathcal {X} \times \mathcal {U}} ^ {p} \mathrm {d} J \left(\left(x, u\right), \left(x ^ {\prime}, u ^ {\prime}\right)\right)\right) ^ {1 / p}
$$
where  $p \in [1, \infty)$  and  $\mathcal{J}(\mu, \nu)$  denotes a set of all joint distributions  $J$  with a support set  $\mathcal{X} \times \mathcal{U}$  that have marginals  $\mu$  and  $\nu$ . Next, we define a robustness condition for our learning-augmented framework.
Definition 3 (r-locally  $p$ -Wasserstein robustness). A policy  $\overline{\pi} = (\pi_t : t \in [T])$  is  $r$ -locally  $p$ -Wasserstein-robust if for any  $0 \leq t_1 \leq t_2 < T$  and any pair of state-action distributions  $\rho, \rho'$  where the  $p$ -Wasserstein distance between them is bounded by  $W_p(\rho, \rho') \leq r$ , for some radius  $r > 0$ , the following inequality holds:
$$
W _ {p} \left(\rho_ {t _ {1}: t _ {2}} (\rho), \rho_ {t _ {1}: t _ {2}} \left(\rho^ {\prime}\right)\right) \leq s \left(t _ {2} - t _ {1}\right) W _ {p} \left(\rho , \rho^ {\prime}\right) \tag {2}
$$
for some function  $s:[T]\to \mathbb{R}_+$  satisfying  $\sum_{t\in [T]}s(t)\leq C_s$  where  $C_s > 0$  is a constant.
Our robustness definition is naturally more relaxed than the usual contraction property in the control/optimization literature [25, 35] — if any two different state-action distributions converge exponentially with respect to the Wasserstein  $p$ -distance, then a policy  $\overline{\pi}$  is  $r$ -locally  $p$ -Wasserstein-robust. This is illustrated in Figure 2. Note that, although the Wasserstein robustness in Definition 3 well captures a variety of distributional robustness metrics such as the total variation robustness defined on finite state/action spaces, it can also be further generalized to other metrics for probability distributions.
As shown in Appendix A (provided in the supplementary material), by establishing a connection between the Wasserstein distance and the total variation metric, any policy that induces a regular
Figure 2: An illustration of an  $r$ -locally  $p$ -Wasserstein-robust policy.
Markov chain satisfies the fast mixing property and the state-action distribution will converge with respect to the total variation distance to a stationary distribution [56]. A more detailed discussion can be found in Appendix A.2. Moreover, the Wasserstein-robustness in Definition 3 includes a set of contraction properties in control theory as special cases. For example, for a locally Wasserstein-robust policy, if the transition kernel  $P$  and the baseline policy  $\overline{\pi}$  are deterministic, then the state-action distributions become point masses, reducing Definition 3 to a state-action perturbation bound in terms of the  $\ell_2$ -norm when implementing the policy  $\overline{\pi}$  from different starting states [35, 19].
The connections discussed above highlight the existence of several well-known robust policies that satisfy Definition 3. Besides the case of discrete MDPs discussed in Appendix A.2, another prominent example is model predictive control (MPC), for which robustness follows from the results in [19] (see Appendix A.1 for details). The model assumption below will be useful in our main results.
Assumption 2. There exists a  $\gamma$ -locally  $p$ -Wasserstein-robust baseline control policy (Definition 3)  $\overline{\pi}$  for some  $p \geq 1$ , where  $\gamma$  is the diameter of the action space  $\mathcal{U}$ .
3.2 Consistency and Robustness for RoE
In parallel with the notation of "consistency and robustness" in the existing literature on learning-augmented algorithms [31], we define a new metric of consistency and robustness in terms of RoE. To do so, we first introduce an optimal policy  $\pi^{\star}$ . Based on MDP  $(\mathcal{X},\mathcal{U},T,P,c)$ , let  $\pi_t^\star = (\pi_t^\star :t\in [T])$  denote the optimal policy at each time step  $t\in [T]$ , whose optimal Q-value function is
$$
Q _ {t} ^ {\star} (x, u) := \inf  _ {\pi} \mathbb {E} _ {P, \pi} \left[ \sum_ {\tau = t} ^ {T - 1} c _ {\tau} \left(x _ {\tau}, u _ {\tau}\right) \mid x _ {t} = x, u _ {t} = u \right],
$$