Title: Beyond Black-Box Advice: Learning-Augmented Algorithms for MDPs with Q-Value Predictions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Time-varying MDP

Learning-augmented online algorithm

consistency and robustness tradeoff
Abstract: We study the tradeoff between consistency and robustness in the context of a single-trajectory time-varying Markov Decision Process (MDP) with untrusted machine-learned advice. Our work departs from the typical approach of treating advice as coming from black-box sources by instead considering a setting where additional information about  how the advice is generated is available. We prove a first-of-its-kind consistency and robustness tradeoff given Q-value advice under a general MDP model that includes both continuous and discrete state/action spaces. Our results highlight that utilizing Q-value advice enables dynamic pursuit of the better of machine-learned advice and a robust baseline, thus result in near-optimal performance guarantees, which provably improves what can be obtained solely with black-box advice.

Beyond Black-Box Advice: Learning-Augmented Algorithms for MDPs with Q-Value Predictions

Abstract

We study the tradeoff between consistency and robustness in the context of a single-trajectory time-varying Markov Decision Process (MDP) with untrusted machine-learned advice. Our work departs from the typical approach of treating advice as coming from black-box sources by instead considering a setting where additional information about how the advice is generated is available. We prove a first-of-its-kind consistency and robustness tradeoff given Q-value advice under a general MDP model that includes both continuous and discrete state/action spaces. Our results highlight that utilizing Q-value advice enables dynamic pursuit of the better of machine-learned advice and a robust baseline, thus result in near-optimal performance guarantees, which provably improves what can be obtained solely with black-box advice.

1 Introduction

Machine-learned predictions and hand-crafted algorithmic advice are both crucial in online decisionmaking problems, driving a growing interest in learning-augmented algorithms [1, 2] that exploit the benefits of predictions to improve the performance for typical problem instances while bounding the worst-case performance [3, 4]. To this point, the study of learning-augmented algorithms has primarily viewed machine-learned advice as potentially untrusted information generated by black-box models. Yet, in many real-world problems, additional knowledge of the machine learning models used to produce advice/predictions is often available and can potentially improve the performance of learning-augmented algorithms.

A notable example that motivates our work is the problem of minimizing costs (or maximizing rewards) in a single-trajectory Markov Decision Process (MDP). More concretely, a value-based machine-learned policy ˜ π can be queried to provide suggested actions as advice to the agent at each step [5-7]. Typically, the suggested actions are chosen to minimize (or maximize, in case of rewards) estimated cost-to-go functions (known as Q-value predictions) based on the current state.

Naturally, in addition to suggested actions, the Q-value function itself can also provide additional information (e.g., the long-term impact of choosing a certain action) potentially useful to the design of a learning-augmented algorithm. Thus, this leads to two different designs for learning-augmented algorithms in MDPs: black-box algorithms and grey-box algorithms. A learning-augmented algorithm using ˜ π is black-box if ˜ π provides only the suggested action ˜ u to the learning-augmented algorithm,

whereas it is value-based (a.k.a., grey-box) if ˜ π provides an estimate of the Q-value function ˜ Q (that also implicitly includes a suggested action ˜ u obtained by minimizing ˜ Q ) to the learning-augmented algorithm.

Value-based policies ˜ π often perform well empirically in stationary environments in practice [5, 6]. However, they may not have performance guarantees in all environments and can perform poorly at times due to a variety of factors, such as non-stationary environments [8-11], policy collapse [12], sample inefficiency [13], and/or when training data is biased [14]. As a consequence, such policies often are referred to as 'untrusted advice' in the literature on learning-augmented algorithms, where the notion of 'untrusted' highlights the lack of performance guarantees. In contrast, recent studies in competitive online control [15-21] have begun to focus on worst-case analysis and provide control policies π with strong performance guarantees even in adversarial settings, referred to as robustness , i.e., π provides 'trusted advice.' Typically, the goal of a learning-augmented online algorithm [1, 3] is to perform nearly as well as the untrusted advice when the machine learned policy performs well, a.k.a., achieve consistency , while also ensuring worst-case robustness. Combining the advice of an untrusted machine-learned policy ˜ π and a robust policy π naturally leads to a tradeoff between consistency and robustness. In this paper, we explore this tradeoff in a time-varying MDP setting and seek to answer the following key question for learning-augmented online algorithms:

Can Q-value advice from an untrusted machine-learned policy, ˜ π , in a grey-box scenario provide more benefits than the black-box action advice generated by ˜ π in the context of consistency and robustness tradeoffs for MDPs?

1.1 Contributions

We answer the question above in the affirmative by presenting and analyzing a unified projectionbased learning-augmented online algorithm (PROjection Pursuit policy, simplified as PROP in Algorithm 1) that combines action feedback from a trusted, robust policy π with an untrusted ML policy ˜ π . In addition to offering a consistency and robustness tradeoff for MDPs with black-box advice, our work moves beyond the black-box setting. Importantly, by considering the grey-box setting, the design of PROP demonstrates that the structural information of the untrusted machinelearned advice can be leveraged to determine the trust parameters dynamically, which would otherwise be challenging (if not impossible) in a black-box setting. To our best knowledge, PROP is the first-ofits-kind learning-augmented algorithm that applies to general MDP models, which allow continuous or discrete state and action spaces.

Our main results characterize the tradeoff between consistency and robustness for both black-box and grey-box settings in terms of the ratio of expectations, RoE , built upon the traditional consistency and robustness metrics in [3, 22, 23, 4] for the competitive ratio. We show in Theorem 5.2 that for the black-box setting, PROP is (1 + O ((1 -λ ) γ )) -consistent and ( ROB + O ( λγ )) -robust where 0 ≤ λ ≤ 1 is a hyper-parameter. Moreover, for the black-box setting, PROP cannot be both (1 + o ( λγ )) -consistent and ( ROB + o ((1 -λ ) γ )) -robust for any 0 ≤ λ ≤ 1 where γ is the diameter of the action space. In sharp contrast, by using a careful design of a robustness budget parameter in PROP with Q-value advice (grey-box setting), PROP is 1 -consistent and ( ROB + o (1)) -robust.

Our result highlights the benefits of exploiting the additional information informed by the estimated Q-value functions, showing that the ratio of expectations can approach the better of the two policies ˜ π and π for any single-trajectory time-varying, and even possibly adversarial environments - if the value-based policy ˜ π is near-optimal, then the worst-case RoE ( PROP ) can approach 1 as governed by a consistency parameter; otherwise, RoE ( PROP ) can be bounded by the ratio of expectations of π subject to an additive term o (1) that decreases when the time horizon T increases.

A key technical contribution of our work is to provide the first quantitative characterization of the consistency and robustness tradeoff for a learning-augmented algorithm (PROP) in a general MDP model, under both standard black-box and novel grey-box settings. Importantly, PROP is able to leverage a broad class of robust policies, called Wasserstein robust policies, which generalize the well-known contraction principles that are satisfied by various robust policies [24] and have been used to derive regrets for online control [19, 25]. A few concrete examples of Wasserstein robust policies applicable for PROP are provided in Table 1(Section 3.1).

1.2 Related Work

Learning-Augmented Algorithms with Black-Box Advice. The concept of integrating black-box machine-learned guidance into online algorithms was initially introduced by [26]. [3] coined terms 'robustness" and 'consistency" with formal mathematical definitions based on the competitive ratio. Over the past few years, the consistency and robustness approach has gained widespread popularity and has been utilized to design online algorithms with black-box advice for various applications, including ski rental [3, 22, 23], caching [27-29], bipartite matching [30], online covering [31, 32], convex body chasing [4], nonlinear quadratic control [33]. The prior studies on learning-enhanced algorithms have mainly focused on creating meta-strategies that combine online algorithms with black-box predictions, and typically require manual setting of a trust hyper-parameter to balance consistency and robustness. A more recent learning-augmented algorithm in [33] investigated the balance between competitiveness and stability in nonlinear control in a black-box setting. However, this work limits the robust policy to a linear quadratic regulator and does not provide a theoretical basis for the selection of the trust parameters. [34] generalized the black-box advice setting by considering distributional advice.

Online Control and Optimization with Structural Information. Despite the lack of a systematic analysis, recent studies have explored the usage of structural information in online control and optimization problems. Closest to our work, [7] considered a related setting where the Q-value function is available as advice, and shows that such information can be utilized to reduce regret in a tabular MDP model. In contrast, our analysis applies to more general models that allow continuous state/action spaces. In [17], the dynamical model and the predictions of disturbances in a linear control system are shown to be useful in achieving a near-optimal consistency and robustness tradeoff. The predictive optimization problem solved by MPC [35, 36, 16, 37] can be regarded as a special realization of grey-box advice, where an approximated cost-to-go function is constructed from structural information that includes the (predicted) dynamical model, costs, and disturbances.

MDPwith External Feedback. Feedback from external sources such as control baselines [38, 39], visual explanations [40], and human experts [41-43] is often available in MDP. This external feedback can be beneficial for various purposes, such as ensuring safety [44], reducing variance [38], training human-like chatbots [41], and enhancing overall trustworthiness [45], among others. The use of control priors has been proposed by [38] as a way to guarantee the Lyapunov stability of the training process in reinforcement learning. They used the Temporal-Difference method to tune a coefficient that combines a RL policy and a control prior, but without providing a theoretical foundation. Another related area is transfer learning in RL, where external Q-value advice from previous tasks can be adapted and utilized in new tasks. Previous research has shown that this approach can outperform an agnostic initialization of Q, but these results are solely based on empirical observations and lack theoretical support [46-48].

2 Problem Setting

We consider a finite-horizon, single-trajectory, time-varying MDP with T discrete time steps. The state space X is a subset of a normed vector space embedded with a norm ‖ · ‖ X . The actions are chosen from a convex and compact set U in a normed vector space characterized by some norm ‖ · ‖ U . Notably, U can represent either continuous actions or the probability distributions used when choosing actions from a finite set. 1 The diameter of the action space U is denoted by γ := max u ∈ U ‖ u ‖ U . Denote [ T ] := { 0 , . . . , T -1 } . For each time step t ∈ [ T ] , let P t : X × U → P X be the transition probability, where P X is a set of probability measures on X . We consider time-varying costs c t : X × U → R + , while rewards can be treated similarly by adding a negative sign. An initial state x 0 ∈ X is fixed. This MDP model is compactly represented by MDP ( X , U , T, P, c ) .

The goal of a policy in this MDP setting is to minimize the total cost over all T steps. The policy agent has no access to the full MDP. At each time step t ∈ [ T ] , only the incurred cost value c t ( x t , u t ) and the next state x t +1 ∼ P t ( ·| x t , u t ) are revealed to the agent after playing an action u t ∈ U . We denote a policy by π = ( π t : t ∈ [ T ]) where each π t : X → U chooses an action u t when observing

x t at step t ∈ [ T ] . Note that our results can be generalized to the setting when π t is stochastic and outputs a probability distribution on U . Given MDP ( X , U , T, P, c ) , we consider an optimization with time-varying costs and transition dynamics. Thus, our goal is to find a policy π that minimizes the following expected total cost:

where the randomness in E P, π is from the transition dynamics P = ( P t : t ∈ [ T ]) and the policy π = ( π t : t ∈ [ T ]) . Wefocus our analysis on the expected dynamic regret and the ratio of expectations, defined below, as the performance metrics for our policy design.

Definition 1 (Expected dynamic regret) . Given MDP ( X , U , T, P, c ) , the (expected) dynamic regret of a policy π = ( π t : t ∈ [ T ]) is defined as the difference between the expected cost induced by the policy π , J ( π ) in (1) , and the optimal expected cost J /star := inf π J ( π ) , i.e., DR ( π ) := J ( π ) -J /star .

Dynamic regret is a more general (and often more challenging to analyze) measure than classical static regret, which has been mostly used for stationary environments [49, 50]. The following definition of the ratio of expectations [51, 52] will be used as an alternative performance metric in our main results.

Definition 2 (Ratio of expectations) . Given MDP ( X , U , T, P, c ) , the ratio of expectations of a policy π = ( π t : t ∈ [ T ]) is defined as RoE ( π ) := J ( π ) /J /star where J ( π ) and J /star are the same as in Definition 1.

Dynamic regret and the ratio of expectations defined above also depend on the error of the untrusted ML advice; we make this more explicit in Section 3.2. Next, we state the following continuity assumption, which is standard in MDPs with continuous action and state spaces [53-55]. Note that our analysis can be readily adapted to general Hölder continuous costs with minimal modifications.

Assumption 1 (Lipschitz costs) . For any time step t ∈ [ T ] , the cost function c t : X × U → R + is Lipschitz continuous with a Lipschitz constant L C < ∞ , i.e., for any t ∈ [ T ] , | c t ( x, u ) -c t ( x ′ , u ′ ) | ≤ L C ( ‖ x -x ′ ‖ X + ‖ u -u ′ ‖ U ) . Moreover, 0 < c t ( x, u ) < ∞ for all t ∈ [ T ] , x ∈ X , and u ∈ U .

3 Consistency and Robustness in MDPs

Expected

元

RobustPolicy

Q-ValueAdvice

Q

Suggested Action u

Black/Grey-Box

Grey-Box

Black-Box

RoE

π

元

ROB

π

ut

ut

Environment

Algorithm

RobustPolicy

0

Our objective is to achieve a balance between the worst-case guarantees on cost minimization in terms of dynamic regret provided by a robust policy, π , and the average-case performance of a valued-based policy, ˜ π , in the context of MDP ( X , U , T, P, c ) . In particular, we denote by ROB ≥ 1 a ratio of expectation bound of the robust policy π such that the worst case RoE ( π ) ≤ ROB . In the learningaugmented algorithms literature, these two goals are referred to as consistency and robustness [3, 1]. Informally, robustness refers to the goal of ensuring worst-case guarantees on cost minimization comparable to those provided by π and consistency refers to ensuring performance nearly as good as ˜ π when ˜ π performs well (e.g., when the instance is not adversarial). Learning-augmented algorithms seek to achieve consistency and robustness by combining π and ˜ π , as illustrated in Figure 1.

Our focus in this work is to design robust and consistent algorithms for two types of advice: black-box advice and grey-box advice. The type of advice that is nearly always the focus in the learningaugmented algorithm literature is black-box advice - only providing a suggested action ˜ u t without additional information. In contrast, on top of the action ˜ u t , grey-box advice can also reveal the internal state of the learning algorithm, e.g., the Q-value ˜ Q t in our setting. This contrast is illustrated in Figure 1.

Compared to black-box advice, grey-box advice has received much less attention in the literature, despite its potential to improve tradeoffs between consistency and robustness as recently shown in [34, 17]. Nonetheless, the extra information on top of the suggested action in a grey-box setting potentially allows the learning-augmented algorithm to make a better-informed decision based on the advice, thus achieving a better tradeoff between consistency and robustness than otherwise possible.

In the remainder of this section, we discuss the robustness properties for the algorithms we consider in our learning-augmented framework (Section 3.1), and introduce the notions of consistency in our grey-box and black-box models in Section 3.2.

3.1 Locally Wasserstein-Robust Policies

We begin with constructing a novel notion of robustness for our learning-augmented framework based on the Wasserstein distance as follows. Denote the robust policy by π := ( π t : t ∈ [ T ]) , where each π t maps a system state to a deterministic action (or a probability of actions in the stochastic setting). Denote by ρ t 1 : t 2 ( ρ ) the joint distribution of the state-action pair ( x t , u t ) ∈ X × U at time t 2 ∈ [ T ] when implementing the baselines π t 1 , . . . , π t 2 consecutively with an initial state-action distribution ρ . We use ‖ · ‖ X × U := ‖ · ‖ X + ‖ · ‖ U as the included norm for the product space X × U . Let W p ( µ, ν ) denote the Wasserstein p -distance between distributions µ and ν whose support set is X × U :

where p ∈ [1 , ∞ ) and J ( µ, ν ) denotes a set of all joint distributions J with a support set X × U that have marginals µ and ν . Next, we define a robustness condition for our learning-augmented framework.

Definition 3 ( r -locally p -Wasserstein robustness) . A policy π = ( π t : t ∈ [ T ]) is r -locally p -Wasserstein-robust if for any 0 ≤ t 1 ≤ t 2 < T and any pair of state-action distributions ρ , ρ ′ where the the p -Wasserstein distance between them is bounded by W p ( ρ , ρ ′ ) ≤ r , for some radius r > 0 , the following inequality holds:

Our robustness definition is naturally more relaxed than the usual contraction property in the control/optimization literature [25, 35] - if any two different state-action distributions converge exponentially with respect to the Wasserstein p -distance, then a policy π is r -locally p -Wasserstein-robust . This is illustrated in Figure 2. Note that, although the Wasserstein robustness in Definition 3 well captures a variety of distributional robustness metrics such as the total variation robustness defined on finite state/action spaces, it can also be further generalized to other metrics for probability distributions.

As shown in Appendix A (provided in the supplementary material), by establishing a connection between the Wasserstein distance and the total variation metric, any policy that induces a regular

p

r

Wp

(Puiu(0),Pt:u(p))

Wp(p,p)

p

元

t1

Markov chain satisfies the fast mixing property and the state-action distribution will converge with respect to the total variation distance to a stationary distribution [56]. A more detailed discussion can be found in Appendix A.2. Moreover, the Wasserstein-robustmess in Definition 3 includes a set of contraction properties in control theory as special cases. For example, for a locally Wasserstein-robust policy, if the transition kernel P and the baseline policy π are deterministic, then the state-action distributions become point masses, reducing Definition 3 to a state-action perturbation bound in terms of the /lscript 2 -norm when implementing the policy π from different starting states [35, 19].

The connections discussed above highlight the existence of several well-known robust policies that satisfy Definition 3. Besides the case of discrete MDPs discussed in Appendix A.2, another prominent example is model predictive control (MPC), for which robustness follows from the results in [19] (see Appendix A.1 for details). The model assumption below will be useful in our main results.

Assumption 2. There exists a γ -locally p -Wasserstein-robust baseline control policy (Definition 3) π for some p ≥ 1 , where γ is the diameter of the action space U .

3.2 Consistency and Robustness for RoE

In parallel with the notation of 'consistency and robustness' in the existing literature on learningaugmented algorithms [3, 1], we define a new metric of consistency and robustness in terms of RoE . To do so, we first introduce an optimal policy π /star . Based on MDP ( X , U , T, P, c ) , let π /star t = ( π /star t : t ∈ [ T ]) denote the optimal policy at each time step t ∈ [ T ] , whose optimal Q-value function is

where E P, π denotes an expectation with respect to the randomness of the trajectory { ( x t , u t ) : t ∈ [ T ] } obtained by following a policy π and the transition probability P at each step t ∈ [ T ] . The Bellman optimality equations can then be expressed as

We can now define a consistency measure for Q-value advice ˜ Q t , which measures the error of the estimates of the Q-value functions due to approximation error and time-varying environments, etc. Let p ∈ (0 , ∞ ] . Fix a sequence of distributions ρ = ( ρ t : t ∈ [ T ]) whose support set is X × U and let φ t be the marginal distribution of ρ t on X . We define a quantity representing the error of the Q-value for all ( x, u ) ∈ X × U , t ∈ [ T ] and t ∈ [ T ] , where we write ( P t V /star ) ( x, u ) := E x ′ ∼ P t ( ·| x,u ) [ V /star ( x ′ )] . This indicates that for each time step t ∈ [ T ] , π /star t is the greedy policy with respect to its optimal Q-value functions ( Q /star t : t ∈ [ T ]) . Note that for any t ∈ [ T ] , Q /star t ( x, u ) = 0 . Given this setup, the value-based policies ˜ π := ( ˜ π t : t ∈ [ T ]) take the following form. For any t ∈ [ T ] , a value-based policy ˜ π t : X → U produces an action ˜ u t ∈ arg min v ∈ U ˜ Q t ( x t , v ) by minimizing an estimate of the optimal Q-value function ˜ Q t . We make the following assumption on the machine-learned untrusted policy ˜ π and the Q-value advice. Assumption 3. The machine-learned untrusted policy ˜ π is value-based. The Q-value advice ˜ Q t : X × U → R is Lipschitz continuous with respect to u ∈ U for any x ∈ X , with a Lipschitz constant L Q for all t ∈ [ T ] . Moreover, ˜ Q t ( x, u ) -Q /star t ( x, u ) = o ( T ) for all ( x, u ) ∈ X × U and t ∈ [ T ] .

advice where ‖ · ‖ p, ρ := (∫ |·| p d ρ ) 1 /p denotes the L p, ρ -norm. A policy with Q-value functions { Q t : t ∈ [ T ] } is said to be ( ε , p, ρ ) -consistent if there exists an ε satisfying (4). In addition, a policy is (0 , ∞ ) -consistent if ˜ Q t is a Lebesgue-measurable function for all t ∈ [ T ] and ( ∞ , ε ) -consistent if the L ∞ -norm satisfies ∑ t ∈ [ T ] ‖ ˜ Q t -Q /star t ‖ ∞ ≤ ε . The consistency error of a policy in (4) quantifies how the Q-value advice is close to optimal Q-value functions. It depends on various factors such the function approximation error or training error due to the distribution shift, and has a close connection to a rich literature on value function approximation [57-61]. The results in [59] generalized the worstcase L ∞ guarantees to arbitrary L p, ρ -norms under some mixing assumptions via policy iteration for a stationary Markov decision process (MDP) with a continuous state space and a discrete action space. Recently, approximation guarantees for the average case for parametric policy classes (such as a neural network) of value functions have started to appear [57, 58, 60]. These bounds are useful in lots of supervised machine learning methods such as classification and regression, whose bounds are typically given on the expected error under some distribution. These results exemplify richer instances of the consistency definition (see (4)) and a summary of these bounds can be found in [61].

Now, we are ready to introduce our definition of consistency and robustness with respect to the ratio of expectations, similar to the growing literature on learning-augmented algorithms [3, 22, 23, 4]. We write the ratio of expectations RoE ( ε ) of a policy π as a function of the Q-value advice error ε in terms of the L ∞ norm, defined in (4).

Definition 4 (Consistency and Robustness) . An algorithm π is said to be k -consistent if its worst-case (with respect to the MDP model MDP ( X , U , T, P, c ) ) ratio of expectations satisfies RoE ( ε ) ≤ k for ε = 0 . On the other hand, it is l -robust if RoE ( ε ) ≤ l for any ε > 0 .

4 The Projection Pursuit Policy (PROP)

In this section we introduce our proposed algorithm (Algorithm 1), which achieves near-optimal consistency while bounding the robustness by leveraging a robust baseline (Section 3.1) in combination with value-based advice (Section 3.2). A key challenge in the design is how to exploit the benefits of good value-based advice while avoiding following it too closely when it performs poorly. To address this challenge, we propose to judiciously project the value-based advice into a neighborhood of the robust baseline. By doing so, the actions we choose can follow the value-based advice for consistency while staying close to the robust baseline for robustness. More specifically, at each step t ∈ [ T ] , we choose u t = Proj U t ( u t ) where a projection operator Proj U t ( · ) : U → U is defined as

corresponding to the projection of u onto a ball U t := { u ∈ U : ‖ u -π t ( x t ) ‖ U ≤ R t } . Note that when the optimal solution of (5) is not unique, we choose the one on the same line with π t ( x t ) -u .

The PROjection Pursuit policy, abbreviated as PROP, can be described as follows. For a time step t ∈ [ T ] , let ˜ π t : X → U denote a policy that chooses an action ˜ u t (arbitrarily choose one if there are multiple minimizers of ˜ Q t ), given the current system state x t at time t ∈ [ T ] and step t ∈ [ T ] . An action u t = Proj U t ( ˜ u t ( x t )) is selected by projecting the machine-learned action ˜ u t ( x t ) onto a norm ball U t defined by the robust policy π given a radius R t ≥ 0 . Finally, PROP applies to both black-box and grey-box settings (which differ from each other in terms of how the radius R t is decided). The results under both settings are provided in Section 5, revealing a tradeoff between consistency and robustness.

The radii ( R t : t ∈ [ T ]) can be interpreted as robustness budgets and are key design parameters that determine the consistency and robustness tradeoff. Intuitively, the robustness budgets reflect the trustworthiness on the value-based policy ˜ π -the larger budgets, the more trustworthiness and hence the more freedom for PROP to follow ˜ π . How the robustness budget is chosen differentiates the grey-box setting from the black-box one.

Algorithm 1 PRO jection P ursuit Policy ( PROP )

4.1 Black-Box Setting

In the black-box setting, the only information provided by ˜ π is a suggested action ˜ u for the learningaugmented algorithm. Meanwhile, the robust policy π can also be queried to provide advice u . Thus, without additional information, a natural way to utilize both ˜ π and π is to decide a projection radius at each time based on the how the obtained ˜ u and u . More concretely, at each time t ∈ [ T ] , the robustness budget R t is chosen by the following BLACK-BOX Procedure, where we set R t = λη t with η t := ‖ ˜ u t -u t ‖ U representing the difference between the two advice measured in terms of the norm ‖ · ‖ U and 0 ≤ λ ≤ 1 being a tradeoff hyper-parameter that measures the trustworthiness on the machine-learned advice. The choice of R t = λη t can be explained as follows. The value of η t indicates the intrinsic discrepancy between the robust advice and the machine-learned untrusted advice - the larger discrepancy, the more difficult to achieve good consistency and robustness simultaneously. Given a robust policy and an untrusted policy, by setting a larger λ , we allow the actual action to deviate more from the robust advice and to follow the untrusted advice more closely, and vice versa. λ is a crucial hyper-parameter that can be pre-determined to yield a desired consistency and robustness tradeoff. The computation of R t is summarized in Procedure 1 below.

4.2 Grey-Box Setting

In the grey-box setting, along with the suggested action ˜ u , the value-based untrusted policy ˜ π also provides an estimate of the Q-value function ˜ Q that indicates the long-term cost impact of an action. To utilize such additional information informed by ˜ Q t at each time t ∈ [ T ] , we propose a novel algorithm that dynamically adjusts the budget R t to further improve the consistency and robustness tradeoff. More concretely, let us consider the Temporal-Difference (TD) error TD t = c t -1 + P t -1 ˜ V t -˜ Q t -1 . Intuitively, if a non-zero TD-error is observed, the budget R t needs to be decreased so as to minimize the impact of the learning error. However, the exact TD-error is difficult to compute in practice, since it requires complete knowledge of the transition kernels ( P t : t ∈ [ T ]) . To address this challenge, we use the following estimated TD-error based on previous trajectories:

Denote by β > 0 a hyper-parameter. Based on the estimated TD-error in (6), the robustness budget in Algorithm 1 is set as

which constitutes two terms. The first term η t := ‖ ˜ π t ( x t ) -π t ( x t ) ‖ measures the decision discrepancy between the untrusted policy ˜ π and the baseline policy π , which normalizes the total budget, similar to the one used in the black-box setting in Procedure 1. The second term is the approximate TD-error, which is normalized by the Lipschitz constant L Q of Q-value functions. With these terms defined, the GREY-BOX Procedure below first chooses a suggested action ˜ u t by minimizing ˜ Q t and then decides a robustness budget R t using (7).

Procedure 2 GREY-BOX Procedure at t ∈ [ T ] (Input: state x t and hyper-parameter 0 ≤ β ≤ 1 ) Obtain advice ˜ Q t and ˜ u t where ˜ u t ∈ arg inf v ∈ U ˜ Q t ( x t , v ) Implement π t and obtain u t Set robustness budget R t as (7); Return ( ˜ u t , R t )

5 Main Results

We now formally present the main results for both the black-box and grey-box settings. Our results not only quantify the tradeoffs between consistency and robustness formally stated in Definition 4 with respect to the ratio of expectations, but also emphasize a crucial role that additional information about the estimated Q-values plays toward improving the consistency and robustness tradeoff.

5.1 Black-Box Setting

In the existing learning-augmented algorithms, the untrusted machine-learned policy ˜ π is often treated as a black-box that generates action advice ˜ u t at each time t ∈ [ T ] . Our first result is the following general dynamic regret bound for the black-box setting (Section 4.1). We utilize the Big-O notation, denoted as O ( · ) and o ( · ) to disregard inessential constants.

Theorem 5.1. Suppose the machine-learned policy ˜ π is ( ∞ , ε ) -consistent. For any MDP model satisfying Assumption 1,2, and 3, the expected dynamic regret of PROP with the BLACK-BOX Procedure is bounded by DR ( PROP ) ≤ min {O ( ε ) + O ((1 -λ ) γ T ) , O (( ROB + λγ -1) T ) } where ε is defined in (4) , γ is the diameter of the action space U , T is the length of the time horizon, ROB is the ratio of expectations of the robust baseline π , and 0 ≤ λ ≤ 1 is a hyper-parameter.

When λ increases, the actual action can deviate more from the robust policy, making the dynamic regret potentially closer to that of the value-based policy. While the regret bound in Theorem 5.1 clearly shows the role of λ in terms of controlling how closely we follow the robust policy, the dynamic regret given a fixed λ ∈ [0 , 1] grows linearly in O ( T ) . In fact, the linear growth of dynamic regret holds even if the black-box policy ˜ π is consistent, i.e., ε is small. This can be explained by noting the lack of dynamically tuning λ to follow the better of the two policies - even when one policy is nearly perfect, the actual action still always deviates from it due to the fixed choice of λ .

Consider any MDP model satisfying Assumptions 1,2, and 3. Following the classic definitions of consistency and robustness (see Definition 4), we summarize the following characterization of PROP, together with a negative result in Theorem 5.3. Proofs of Theorem 5.1, 5.2, and 5.3 are detailed in Appendix C.

Theorem 5.2 (BLACK-BOX Consistency and Robustness) . PROP with the BLACK-BOX Procedure is (1 + O ((1 -λ ) γ )) -consistent and ( ROB + O ( λγ )) -robust where 0 ≤ λ ≤ 1 is a hyper-parameter. Theorem 5.3 (BLACK-BOX Impossibility) . PROP with the BLACK-BOX Procedure cannot be both (1 + o ((1 λ ) γ )) -consistent and ( ROB + o ( λγ )) -robust for any 0 λ 1 .

5.2 Grey-Box Setting

To overcome the impossibility result in the black-box setting, we dynamically tune the robustness budgets by tapping into additional information informed by the estimated Q-value functions using the GREY-BOX Procedure (Section 4.2). By setting the robustness budgets in (7), an analogous result of Theorem 5.1 is given in Appendix D, which leads to a dynamic regret bound of PROP in the grey-box setting (Theorem D.1 in Appendix D). Consider any MDP model satisfying Assumptions 1,2, and 3. Our main result below indicates that knowing more structural information about a black-box policy can indeed bring additional benefits in terms of the consistency and robustness tradeoff, even if the black-box policy is untrusted.

Theorem 5.4 (GREY-BOX Consistency and Robustness) . PROP with the GREY-BOX Procedure is 1 -consistent and ( ROB + o (1)) -robust for some β > 0 .

Theorem 5.3 implies that using the BLACK-BOX Procedure, PROP cannot be 1 -consistent and ( ROB + o (1)) -robust, while this can be achieved using the GREY-BOX Procedure. On one hand, this theorem validates the effectiveness of the PROP policy with value-based machine-learned advice that

may not be fully trusted. On the other hand, this sharp contrast between the black-box and grey-box settings reveals that having access to information of value function can improve the tradeoff between consistency and robustness (see Definition 4) non-trivially. A proof of Theorem 5.4 can be found in Appendix D. Applications of our main results are discussed in Appendix A.

6 Concluding Remarks

Our results contribute to the growing body of literature on learning-augmented algorithms for MDPs and highlight the importance of considering consistency and robustness in this context. In particular, we have shown that by utilizing the structural information of machine learning methods, it is possible to achieve improved performance over a black-box approach. The results demonstrate the potential benefits of utilizing value-based policies as advice; however, there remains room for future work in exploring other forms of structural information.

Limitations and Future Work. One limitation of our current work is the lack of analysis of more general forms of black-box procedures. Understanding and quantifying the available structural information in a more systematic way is another future direction that could lead to advances in the design of learning-augmented online algorithms and their applications in various domains.

References

Michael Mitzenmacher and Sergei Vassilvitskii. Algorithms with predictions. Communications of the ACM , 65(7):33-35, 2022.

Tongxin Li. Learning-Augmented Control and Decision-Making: Theory and Applications in Smart Grids . PhD thesis, California Institute of Technology, 2023.

Manish Purohit, Zoya Svitkina, and Ravi Kumar. Improving online algorithms via ml predictions. Advances in Neural Information Processing Systems , 31, 2018.

Nicolas Christianson, Tinashe Handina, and Adam Wierman. Chasing convex bodies and functions with black-box advice. In Conference on Learning Theory , pages 867-908. PMLR, 2022.

Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Advances in neural information processing systems , 30, 2017.

Zhuoran Yang, Chi Jin, Zhaoran Wang, Mengdi Wang, and Michael I Jordan. On function approximation in reinforcement learning: optimism in the face of large state spaces. In Proceedings of the 34th International Conference on Neural Information Processing Systems , pages 13903-13916, 2020.

Noah Golowich and Ankur Moitra. Can q-learning be improved with advice? In Conference on Learning Theory , pages 4548-4619. PMLR, 2022.

Chen-Yu Wei and Haipeng Luo. Non-stationary reinforcement learning without prior knowledge: An optimal black-box approach. In Conference on Learning Theory , pages 4300-4354. PMLR, 2021.

Weichao Mao, Kaiqing Zhang, Ruihao Zhu, David Simchi-Levi, and Tamer Basar. Near-optimal model-free reinforcement learning in non-stationary episodic mdps. In International Conference on Machine Learning , pages 7447-7458. PMLR, 2021.

Yuwei Luo, Varun Gupta, and Mladen Kolar. Dynamic regret minimization for control of nonstationary linear dynamical systems. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 6(1):1-72, 2022.

Peng Zhao, Long-Fei Li, and Zhi-Hua Zhou. Dynamic regret of online markov decision processes. In International Conference on Machine Learning , pages 26865-26894. PMLR, 2022.

Christian Scheller, Yanick Schraner, and Manfred Vogel. Sample efficient reinforcement learning through learning from demonstrations in minecraft. In NeurIPS 2019 Competition and Demonstration Track , pages 67-76. PMLR, 2020.

Matthew Botvinick, Sam Ritter, Jane X Wang, Zeb Kurth-Nelson, Charles Blundell, and Demis Hassabis. Reinforcement learning, fast and slow. Trends in cognitive sciences , 23(5):408-422, 2019.

Xueying Bai, Jian Guan, and Hongning Wang. A model-based reinforcement learning with adversarial training for online recommendation. Advances in Neural Information Processing Systems , 32, 2019.

Guanya Shi, Yiheng Lin, Soon-Jo Chung, Yisong Yue, and Adam Wierman. Online optimization with memory and competitive control. Advances in Neural Information Processing Systems , 33:20636-20647, 2020.

Gautam Goel and Babak Hassibi. Competitive control. IEEE Transactions on Automatic Control , 2022.

Tongxin Li, Ruixiao Yang, Guannan Qu, Guanya Shi, Chenkai Yu, Adam Wierman, and Steven Low. Robustness and consistency in linear quadratic control with untrusted predictions. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 6(1):1-35, 2022.

Oron Sabag, Sahin Lale, and Babak Hassibi. Optimal competitive-ratio control. arXiv preprint arXiv:2206.01782 , 2022.

Yiheng Lin, Yang Hu, Guannan Qu, Tongxin Li, and Adam Wierman. Bounded-regret mpc via perturbation analysis: Prediction error, constraints, and nonlinearity. Advances in Neural Information Processing Systems , 35:36174-36187, 2022.

Tongxin Li, Yue Chen, Bo Sun, Adam Wierman, and Steven H Low. Information aggregation for constrained online control. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 5(2):1-35, 2021.

Tongxin Li, Bo Sun, Yue Chen, Zixin Ye, Steven H Low, and Adam Wierman. Learningbased predictive control via real-time aggregate flexibility. IEEE Transactions on Smart Grid , 12(6):4897-4913, 2021.

Alexander Wei and Fred Zhang. Optimal robustness-consistency trade-offs for learningaugmented online algorithms. Advances in Neural Information Processing Systems , 33:80428053, 2020.

Shom Banerjee. Improving online rent-or-buy algorithms with sequential decision making and ml predictions. Advances in Neural Information Processing Systems , 33:21072-21080, 2020.

Stephen Tu, Alexander Robey, Tingnan Zhang, and Nikolai Matni. On the sample complexity of stability constrained imitation learning. In Learning for Dynamics and Control Conference , pages 180-191. PMLR, 2022.

Hiroyasu Tsukamoto, Soon-Jo Chung, and Jean-Jaques E Slotine. Contraction theory for nonlinear stability analysis and learning-based control: A tutorial overview. Annual Reviews in Control , 52:135-169, 2021.

Mohammad Mahdian, Hamid Nazerzadeh, and Amin Saberi. Online optimization with uncertain information. ACM Transactions on Algorithms (TALG) , 8(1):1-29, 2012.

Dhruv Rohatgi. Near-optimal bounds for online caching with machine learned advice. In Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 1834-1845. SIAM, 2020.

Thodoris Lykouris and Sergei Vassilvitskii. Competitive caching with machine learned advice. Journal of the ACM (JACM) , 68(4):1-25, 2021.

Sungjin Im, Ravi Kumar, Aditya Petety, and Manish Purohit. Parsimonious learning-augmented caching. In International Conference on Machine Learning , pages 9588-9601. PMLR, 2022.

Antonios Antoniadis, Christian Coester, Marek Elias, Adam Polak, and Bertrand Simon. Online metric algorithms with untrusted predictions. In International Conference on Machine Learning , pages 345-355. PMLR, 2020.

Etienne Bamas, Andreas Maggiori, and Ola Svensson. The primal-dual method for learning augmented algorithms. Advances in Neural Information Processing Systems , 33:20083-20094, 2020.

Keerti Anand, Rong Ge, Amit Kumar, and Debmalya Panigrahi. Online algorithms with multiple predictions. In International Conference on Machine Learning , pages 582-598. PMLR, 2022.

Tongxin Li, Ruixiao Yang, Guannan Qu, Yiheng Lin, Adam Wierman, and Steven H Low. Certifying black-box policies with stability for nonlinear control. IEEE Open Journal of Control Systems , 2023.

Ilias Diakonikolas, Vasilis Kontonis, Christos Tzamos, Ali Vakilian, and Nikos Zarifis. Learning online algorithms with distributional advice. In International Conference on Machine Learning , pages 2687-2696. PMLR, 2021.

Yiheng Lin, Yang Hu, Guanya Shi, Haoyuan Sun, Guannan Qu, and Adam Wierman. Perturbation-based regret analysis of predictive control in linear time varying systems. Advances in Neural Information Processing Systems , 34:5174-5185, 2021.

Yiheng Lin, Yang Hu, Guannan Qu, Tongxin Li, and Adam Wierman. Bounded-regret mpc via perturbation analysis: Prediction error, constraints, and nonlinearity. In Advances in Neural Information Processing Systems , 2022.

David Hoeller, Farbod Farshidian, and Marco Hutter. Deep value model predictive control. In Conference on Robot Learning , pages 990-1004. PMLR, 2020.

Richard Cheng, Abhinav Verma, Gabor Orosz, Swarat Chaudhuri, Yisong Yue, and Joel Burdick. Control regularization for reduced variance reinforcement learning. In International Conference on Machine Learning , pages 1141-1150. PMLR, 2019.

Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems , 5:411-444, 2022.

Lin Guan, Mudit Verma, Suna Sihang Guo, Ruohan Zhang, and Subbarao Kambhampati. Widening the pipeline in human-guided reinforcement learning with explanation and contextaware data augmentation. Advances in Neural Information Processing Systems , 34:2188521897, 2021.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L Roberts, Matthew E Taylor, and Michael L Littman. Interactive learning from policy-dependent human feedback. In International Conference on Machine Learning , pages 2285-2294. PMLR, 2017.

Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning , pages 10835-10866. PMLR, 2023.

Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. Advances in neural information processing systems , 30, 2017.

Mengdi Xu, Zuxin Liu, Peide Huang, Wenhao Ding, Zhepeng Cen, Bo Li, and Ding Zhao. Trustworthy reinforcement learning against intrinsic vulnerabilities: Robustness, safety, and generalizability. arXiv preprint arXiv:2209.08025 , 2022.

Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research , 10(7), 2009.

Irina Higgins, Arka Pal, Andrei Rusu, Loic Matthey, Christopher Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot transfer in reinforcement learning. In International Conference on Machine Learning , pages 1480-1490. PMLR, 2017.

Elynn Y Chen, Michael I Jordan, and Sai Li. Transferred q-learning. arXiv preprint arXiv:2202.04709 , 2022.

Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in neural information processing systems , 21, 2008.

Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning , pages 263-272. PMLR, 2017.

Allan Borodin and Ran El-Yaniv. Online computation and competitive analysis . cambridge university press, 2005.

Nikhil R Devanur and Thomas P Hayes. The adwords problem: online keyword matching with budgeted bidders under random permutations. In Proceedings of the 10th ACM conference on Electronic commerce , pages 71-78, 2009.

Naman Agarwal, Brian Bullins, Elad Hazan, Sham Kakade, and Karan Singh. Online control with adversarial disturbances. In International Conference on Machine Learning , pages 111-119. PMLR, 2019.

Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning , pages 2681-2691. PMLR, 2019.

Paula Gradu, John Hallman, and Elad Hazan. Non-stochastic control with bandit feedback. Advances in Neural Information Processing Systems , 33:10764-10774, 2020.

Sheldon M Ross. Stochastic processes . John Wiley & Sons, 1995.

Dimitri P Bertsekas and John N Tsitsiklis. Neuro-dynamic programming: an overview. In Proceedings of 1995 34th IEEE conference on decision and control , volume 1, pages 560-564. IEEE, 1995.

Rémi Munos. Error bounds for approximate policy iteration. In ICML , volume 3, pages 560-567. Citeseer, 2003.

András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning , 71(1):89-129, 2008.

Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. In International Conference on Machine Learning , pages 2160-2169. PMLR, 2019.

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. J. Mach. Learn. Res. , 22(98):1-76, 2021.

Yiheng Lin, James A Preiss, Emile Timothy Anand, Yingying Li, Yisong Yue, and Adam Wierman. Online adaptive policy selection in time-varying systems: No-regret via contractive perturbations. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Runyu Zhang, Yingying Li, and Na Li. On the regret analysis of online LQR control with predictions. In 2021 American Control Conference (ACC) , pages 697-703. IEEE, 2021.

James R Norris. Markov chains . Cambridge university press, 1998.

Steve Lalley. Markov chains: Basic theory. https://galton.uchicago.edu/~lalley/ Courses/312/MarkovChains.pdf , 2016.

Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably efficient? Advances in neural information processing systems , 31, 2018.

Yingying Li and Na Li. Online learning for markov decision processes in nonstationary environments: A dynamic regret analysis. In 2019 American Control Conference (ACC) , pages 1232-1237. IEEE, 2019.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning , pages 387-395. Pmlr, 2014.

Leonid Kantorovitch. On the translocation of masses. Management science , 5(1):1-4, 1958.

Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: no regret and experimental design. In Proceedings of the 27th International Conference on International Conference on Machine Learning , pages 1015-1022, 2010.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers the setting of using advice in Markov Decision Processes (MDPs). They consider the setting of using only action suggestions (termed black-box) or action and Q-values (grey-box) from a potentially erroneous advisor. They propose an algorithm that decides between the advice and a robust policy by projecting the advisor’s suggested action onto a ball centered around the robust policy’s action. Depending on the type of advisor, the radii of the projection ball can be static (black-box) or dynamically-chosen based on the estimated error of the Q-values (grey-box). The algorithm is analyzed along the axes of consistency (performance) and robustness, and it is shown that the grey-box setting can achieve much better consistency and robustness than the black-box setting.\n\nSTRENGTHS:\nThis paper is very well-written and clear. There is a lot of effort in explaining the setting, the definitions, and the approach which helps with readability and understanding. \n\nThe results are more general than prior works, which had similar results in the tabular setting (Golowich, 2021), however, I am also not an expert in this area. \n\nThe Projection Pursuit Policy (PROP) approach generalizes the black-box and grey-box settings nicely, and the results for the grey-box setting are promising for future work.\n\nWEAKNESSES:\nI think the paper overall could benefit from some empirical or practical demonstration of the setting, similar to what is presented in Appendix A, but with more focus on the applicability of the algorithm.  Ideally this would be in the main body but I understand that this is difficult given the page limit.  \n\nMy only concern is in the usefulness of the setting and some of its assumptions, which I explain below and hope the authors can comment on.\n\nAn example of a setting this could be used in: we have learned an optimal or close-to-optimal policy/value function but our environment is non-stationary and may have shifted since we learned this policy. We also have a baseline policy that is known to be robust in this environment. We then use PROP to take an action depending on how much to trust the robust vs the previously-learned policy. My concerns are:\n\n- The grey-box advice is in the form of Q-values: The choice of $(\\infty, \\epsilon)$-consistent policies is quite strong (defined in L243). In most realistic situations, we might only be able to guarantee the Q-function on certain data-dependent distributions. I think a more natural assumption would be to consider slowly-varying dynamics, for example where the change in $\\mathbb{P}_t$ is bounded under some norm. Could you comment on the limitations of this assumption?\n\n- Access to a robust policy: This was discussed for the case of discrete MDPs and LQR, however for the case of general MDPs, how can this policy be obtained? If the environment is time-varying, it might also affect the robustness guarantees of the policy.\n\n- This may be out-of-scope but a central assumption is that advice is provided at each step: advice might generally only be provided in some steps and not in others but the agent should still have the ability to learn the best policy in those situations.\n\nQUESTIONS:\nMinor/Typos:\n\n- Algorithm 1 box line 5 and 6 should be switched, since we don’t know all $x_t$ before taking the selected actions.\n\n- L302, instead of $\\tilde{\\pi}_t$ should be $\\tilde{u}_t$?\n\nLIMITATIONS:\nThey have addressed some limitations, but some questions remain (See Weaknesses)", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the tradeoff between consistency and robustness in the context of a single-trajectory time-varying Markov Decision Process (MDP) with untrusted machine-learned advice. Time-varying MDPs are relevant when dealing with environments where costs, rewards, and transition probabilities might change over time. In such settings, it is essential to design algorithms that can adapt to the changing dynamics of the environment. The proposed Projection Pursuit Policy (PROP) aims to balance the performance of an untrusted machine-learned policy with a trusted robust policy, taking into account the time-varying nature of the problem. The algorithm is designed to make better use of Q-value advice from the untrusted policy while maintaining worst-case performance guarantees close to those provided by the robust policy. The authors provide theoretical analysis for the consistency and robustness of the proposed algorithm. In the appendix, the authors also provide examples to demonstrate the versatility and effectiveness of their findings to various real-world problems.\n\nSTRENGTHS:\nOriginality:\n\nThe paper presents a novel learning-augmented algorithm (PROP) for MDPs with Q-value predictions.\n\nThe work considers both black-box and grey-box settings, demonstrating the benefits of leveraging additional structural information in achieving improved performance guarantees.\n\nQuality:\n\nThe submission is technically sound, and the claims are well supported by theoretical analysis.\n\nThe authors provide a thorough analysis of the consistency and robustness tradeoffs in the context of MDPs.\n\n\nClarity:\n\nThe submission is clearly written and well-organized.\n\nSignificance:\n\nThe work advances the state of the art in learning-augmented algorithms for MDPs and demonstrates the potential benefits of utilizing value-based policies as advice.\n\nThe findings are relevant to researchers and practitioners interested in developing algorithms that can adapt to changing environments while maintaining performance guarantees.\n\nWEAKNESSES:\nInsufficient comparison to existing approaches: The paper does not provide a detailed comparison of the proposed PROP algorithm with existing learning-augmented algorithms, particularly in the context of discrete action spaces. A thorough comparison, both in terms of theoretical analysis and empirical evaluations, would help demonstrate the advantages and potential improvements offered by the proposed approach over existing methods. This would also provide a clearer understanding of the practical implications and the extent to which the proposed method advances the state of the art.\n\nQUESTIONS:\nThe authors mentioned in the abstract that their method can be applied in the discrete action space setting. However, the Projection Pursuit Policy may require adjustments to ensure that the resulting action remains feasible. When combining actions from different policies or projecting them onto a certain space, the resulting action may not lie exactly in the discrete action space. These adjustments may affect the theoretical guarantees of the algorithm, and further analysis might be needed to determine the impact of such modifications on the consistency and robustness tradeoffs of the Projection Pursuit Policy in the discrete action space setting.\n\nThe $\\epsilon$ parameter in Figure 1 is not introduced. It seems that it should be the same as in Equation 4.\n\nLIMITATIONS:\nThe dependence of the theoretical results on the parameter $\\lambda$ in the black box setting may affect the practical implementation of the algorithm, as the choice of $\\lambda$ could influence the trade-off between consistency and robustness. Potential strategies for selecting an appropriate value are needed.\n\nIn the grey box setting, the theoretical results are only valid for some $\\beta$, which should satisfy the condition $R_t = 0$. It may be challenging to determine a suitable value of $\\beta$ in practice. Further analysis of the algorithm's performance and robustness under different values of $\\beta$ would be valuable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel algorithm that combines a robust policy and a potentially untrusted machine learning based policy. The idea is to be able to achieve a trade-off between robustness and consistency or high performance entailed by the ML policy. The paper suggests a novel projection algorithm along with theoretical guarantees on the proposed algorithm.\n\nSTRENGTHS:\nThe paper seems to have solid theoretical grounding on the proposed algorithm, the background problem is also relatively well motivated and can be useful for general contexts where a ML policy is present alongside a default robust policy. This can be of interest to the general decision making practitioners for real applications.\n\nWEAKNESSES:\nThe paper can be potentially improved by being more concrete about the problems and applications it aims to address, as well as showcasing some empirical evidence that the suggested algorithm achieves nice guarantees predicted by the theory.\n\nQUESTIONS:\n#### === *Motivation* ===\n\nChecking my understanding of the problem at hand, we have a default robust policy $\\bar{\\pi}$ as well as a ML based policy $\\tilde{\\pi}$ whose performance guarantee is potentially not trusted. The aim is to derive a new policy that combines the two and be able to achieve consistency vs. robustness trade-off in real-time decision making, does this sound right?\n\nI'd suggest giving more concrete examples for this setup, e.g. education, medical and so on, such that it is better to contextualize the problem at hand and better understand various algorithmic components.\n\n#### === *Fig 1* ===\n\nRightmost plot of Fig 1 -- what's on the x-axis and y-axis? There should be more explanations in the caption.\n\n#### === *Projection pursuit* ===\n\nThe projection pursuit policy derived in Sec 4 seems to be the core contribution of the paper and proposes to combine the two policies using a projection step. A main criticism might be that the algorithm seems specialized to continuous action, where the projection is more well defined? What if the action space is discrete, do we carry out projection in the logits / probability space or in the discrete action space?\n\n#### === *Empirical assessment* ===\n\nThere is no empirical assessment of the approach at all in the paper, as the narrative is purely theoretical. Though the theoretical contribution of the paper seems solid, readers from the NeurIPS community would certainly benefit from certain empirical validations of the theoretical insights in the paper. One should at least test the e.g. projection pursuit algorithm in some even potentially artificial domains, or even real domains (simulated envs) and showcase how the bound prediction is reflected in practice. One would also benefit from a discussion on how those hyper-parameters (e.g. regret budget $R_t$) would actually impact algorithm performance in practice.\n\nLIMITATIONS:\nDiscussed above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates ways to combine untrusted learned policy and Q value predictions with a trusted baseline fixed policy such that the resulting combination achieves a higher performance bound and lower regret than established methods for combining the two. In particular, this paper studies how the use of Q-value predictions in addition to policy action predictions allows for stronger robust performance.\n\nSTRENGTHS:\nI'll first state that that my expertise is in the field of experimentally-driven deep reinforcement learning, so as this paper seems to come from the control-theory literature my ability to evaluate some aspects of it and to properly value its contribution to that field is limited. As such I will focus my comments on the elements where I can speak with some confidence and give the benefit of the doubt regarding other aspects (particularly the overall significance and originality of the problem and set of assumptions being made to the control theory community).\n\nAll that said, from my perspective, this paper makes a lot of sense- the core argument, that using information about TD error (or more generally the error estimates of the RL model- I can imagine there's ways to extend this work to other flavors of RL algorithms without Q-functions) allows for a better tradeoff between consistency and robustness, seems like a sensible one. \n\nIn deep RL it's common to use various measures of model error/confidence to constrain the actions and/or gradient updates of the model in a less rigorous way (Conservative Q Learning is a recent notable example of this), so doing the same thing to bound robustness guarantees makes sense to me.\n\nDespite the difference in field, this paper communicated well and got the core arguments and algorithms across, and seems well written in general, though I can't comment on the perspective of a control theorist on this point.\n\nWEAKNESSES:\nTo err on the side of generosity, I'm going to use this section to mostly speak to the practical feasibility and relevance of this paper (at least my sense of it). It's a theory paper, so I don't consider any of this a critical flaw in the work, but hopefully this is a useful perspective for the authors in their future work. \n\nI'm unsure if assumption 3 is reasonable by default- it might require a very large Lipschitz constant for most neural network architectures (not an expert on this, but my understanding is most common architectures have very poor guarantees on curvature). While theoretically valid I worry that it might require the use of specialized models to get a reasonable constant in practice, especially since per equation 7 it looks like a specific value is needed for normalization, so there's incentive to use as small a number as possible.\n\nThe approximate TD error in equation 6 makes sense to use as an approximate trust measure, but I'll caution that in practice the magnitude of these values can vary by orders of magnitude and is highly MDP dependent, so there may be MDP/model dependent scale/normalization hyperparameters that need to be introduced. In theory and principle it makes sense to me, but my sense is that further work is needed to make this measure practical. \n\nAnother limitation might be applicability in contexts that are very different from those in which the Q-function was trained. Assuming the immediate reward/cost is not available or not highly informative (for example, if we have a lot of deferred and/or sparse rewards) and that the model is not being finetuned, in such cases the TD error between Q_t and Q_t+1 could be small despite the model being highly inaccurate in its Q estimates and thus its actions might be highly untrustworthy. \"Confident but wrong\" is a sadly common phenomenon in deep RL.\n\nOverall though I don't think these issues are immediately relevant to the significance of this work, due to being both out of scope and possible to address through future work. As such (and being deferential regarding significance and novelty since I can't comment on those well) I'm inclined to recommend acceptance for this paper, subject to the opinions of other reviewers who may have perspective I lack.\n\nQUESTIONS:\nThinking out loud, I wonder if relative TD error (compared to some MDP-specific baseline or a running average or similar) might make more sense in practice for equation 6 given the challenges introduced by deep RL models and MDP-specific scale variance. Relative TD error prioritization has been used for both prioritized experience replay and exploration, so there's evidence in deep RL that this sort of ranked/relative error measure is at least semi-reliable.\n\nRegarding using various forms of structural information/error signals for RL, it might be useful for future work to look at some of the literature on exploration and robust/safety-conscious RL (if the authors are not already familiar with this literature) and how novelty/uncertainty metrics are used in those fields. A wide range of metrics have been proposed with differing tradeoffs which might be theoretically useful or at least provide inspiration.\n\nLIMITATIONS:\nWhile I can't speak well to the theoretical limitations of this paper, I have some discussion of possible practical limitations in the above sections (though such concerns are out of scope for this paper).\n\nI don't see any potential negative societal impact from this paper. I will note the authors include a consideration of the potential impacts at the bottom of the paper regardless, which is pleasant to see.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper provides a theoretical study of the tradeoff between consistency and robustness in Markov Decision Process for which machine learned advice is available. With extensive theoretical analyses, the paper leverages Q-value advice for improving the impact of machine-learned advice in these contexts. Several hyperparameters are suggested which control the impact of these learned advices. The paper provides derivations for both black-box and grey-box settings. 


I have read the rebuttal. I am more positive about this submission after the rebuttal and reading the other reviews.The authors have addressed my concerns regarding empirical validation, and the lack of practical examples.

STRENGTHS:
- The paper studies an interesting and salient problem and provides theoretical derivations for their findings. 
- The paper is clearly structured. 
- Limitations and broader impact are adequately studied. 
- Supplementary material is impressively extensive and detailed.

WEAKNESSES:
- I believe the paper could strongly benefit from empirical validation to better communicate the potential impact of their theoretical findings. 
- Some decision choices (see Questions) require further discussion.
- This paper is at times hard to follow and could be written in a more clear way.

QUESTIONS:
- L 308: What is the impact of this approximation? 
- L 283: Could you give a practical example of the impact of this radii?

LIMITATIONS:
Limitations and the broader impact of this method are properly addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-20

comments: SUMMARY:
Authors study algorithms for Marcov Decision Processes receiving advice in the
form of a value-based machine-learned policy. Value-based policies decide
their actions based on minimizing an estimated Q-function, which predicts
the best possible outcome in the rest of the time horizon given a current
action. Such policies are known to perform well in some practical scenarios,
but collapse in others.

They propose algorithms for two regimes. In the first regime, the algorithm
receives only the action taken by the value-based policy while, in the
second regime, it also receives an estimate of the Q-function used by the
value-based policy to determine its action.
In the first regime, they propose a consistency-robustness tradeoff.
In the second regime, they use the additional information (Q-function) to
estimate the quality of the advice provided by the value-based policy
and tune the trade-off parameter online, obtaining consistency 1 and
near-optimal robustness.
They show that the analysis of their algorithm for the first regime is tight.

STRENGTHS:
* work on an important problem
* online tuning of the consistency-robustness trade-off
* tight analysis of their algorithm

WEAKNESSES:
* It is not clear whether consistency-robustness tradeoff with weaker predictions is necessary for any algorithm. They prove a lower bound only for their algorithm.
* requires machine-learned advice in a form of a value-based policies. Other kinds of policies might be also beneficial.

QUESTIONS:
* In algorithm 1, should line 6 be part of the for loop?

LIMITATIONS:
their ML-augmented algorithms are limited to a certain kind of predictions. Authors have mentioned this limitation clearly in the last section of their submission.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-26