Title: Sequential Decision Making with Expert Demonstrations under Unobserved Heterogeneity
Venue: NeurIPS 2024
Accepted: yes
Keywords: meta-reinforcement learning

unobserved heterogeneity

empirical Bayes

maximum entropy

posterior sampling
Abstract: We study the problem of online sequential decision-making given auxiliary demonstrations from _experts_ who made their decisions based on unobserved contextual information. These demonstrations can be viewed as solving related but slightly different tasks than what the learner faces.  This setting arises in many application domains, such as self-driving cars, healthcare, and finance, where expert demonstrations are made using contextual information, which is not recorded in the data available to the learning agent. We model the problem as a zero-shot meta-reinforcement learning setting with an unknown task distribution and a Bayesian regret minimization objective, where the unobserved tasks are encoded as parameters with an unknown prior. We propose the Experts-as-Priors algorithm (ExPerior), an empirical Bayes approach that utilizes expert data to establish an informative prior distribution over the learner's decision-making problem. This prior enables the application of any Bayesian approach for online decision-making, such as posterior sampling. We demonstrate that our strategy surpasses existing behaviour cloning and online algorithms, as well as online-offline baselines for multi-armed bandits, Markov decision processes (MDPs), and partially observable MDPs, showcasing the broad reach and utility of ExPerior in using expert demonstrations across different decision-making setups.

Sequential Decision Making with Expert Demonstrations under Unobserved Heterogeneity

Abstract

We study the problem of online sequential decision-making given auxiliary demonstrations from experts who made their decisions based on unobserved contextual information. These demonstrations can be viewed as solving related but slightly different problems than what the learner faces. This setting arises in many application domains, such as self-driving cars, healthcare, and finance, where expert demonstrations are made using contextual information, which is not recorded in the data available to the learning agent. We model the problem as zero-shot meta-reinforcement learning with an unknown distribution over the unobserved contextual variables and a Bayesian regret minimization objective, where the unobserved variables are encoded as parameters with an unknown prior. We propose the Experts-as-Priors algorithm (ExPerior), an empirical Bayes approach that utilizes expert data to establish an informative prior distribution over the learner's decision-making problem. This prior distribution enables the application of any Bayesian approach for online decision-making, such as posterior sampling. We demonstrate that our strategy surpasses existing behaviour cloning, online, and online-offline baselines for multi-armed bandits, Markov decision processes (MDPs), and partially observable MDPs, showcasing the broad reach and utility of ExPerior in using expert demonstrations across different decision-making setups.

1 Introduction

Reinforcement learning (RL) has found success in complex decision-making tasks, spanning areas such as game playing [1, 2, 3], robotics [4, 5], and aligning with human preferences [6]. However, RL's considerable sample inefficiency, necessitating millions of training frames for convergence, remains a significant challenge. A notable body of work within RL has been dedicated to integrating expert demonstrations to accelerate the learning process, employing strategies like offline pretraining [7] and the use of combined offline-online datasets [8, 9]. While these approaches are theoretically sound and empirically validated [10, 11], they typically presume homogeneity between the offline and online datasets. A vital question arises about the effectiveness of these methods when expert data embody heterogeneous tasks, indistinguishable by the learner.

An important example of such heterogeneity is in situations where experts operate with additional information not available to the learner, a scenario previously explored in imitation learning with unobserved contexts [12, 13, 14, 15]. Existing literature either relies on the availability of experts to query during training [16, 17, 18, 19] or focuses on the assumptions that enable imitation learn-

Goal 1

Goal 2

Goal N

Goal

Expert

Parametric or

Prior

Algorithm

Prior/Posterior

Bayesian

Agent

Environment

History

Update

Goal (Unobserved Contextual Variable)

Step 1: Expert

Demonstrations

Step 2: Learning

the Prior

Step 3: Reinforcement

Learning

Contextual Variables (Goals)

Contextual Variables (Goals)

Contextual Variables (Goals)

D

E

=

{

(

s

,

a

,

s

′

)

, . . .

}

.

.

.

µ

θ

/star

c

∼

µ

ME

(

·|

history

)

a

∼

π

c

(

·|

s

)

(

s

,

a

,

s

′

,

r

)

?

∈

{      ,      , … ,     }

or

µ

ME

or

c

∼

µ

ME

(

·|

history

)

ing with unobserved contexts, sidestepping online reward-based interactions [20, 21]. Recent contributions by Hao et al. [22, 23] suggest using offline expert data for online RL, albeit without accounting for unobserved variations. Our work addresses the more general challenge of online decision-making given auxiliary offline expert data with unobserved heterogeneity. We view such demonstrations as solving related yet distinct problems from those faced by the learner, where differences remain invisible to the learner. For instance, in a personalized education scenario, while a learning agent can observe characteristics like grades or demographics, it might remain oblivious to factors such as learning styles, which are visible to an expert teacher and can significantly influence teaching methods. Na¨ ıve imitation without access to this 'private' information will only learn a single policy for each observed characteristic [24], leading to sub-optimal actions. On the other hand, a purely online approach requires extensive trial and error to result in meaningful decisions.

We integrate offline expert data with online RL, treating the scenario as a zero-shot metareinforcement learning (meta-RL) problem with an unknown distribution over unobserved contextual variables. Unlike typical meta-RL frameworks where the learner is exposed to multiple instances during training (different students in our example) to learn the underlying distribution [25, 26], our approach only leverages offline expert data to infer the distribution of unobserved factors, embodying a zero-shot meta-RL paradigm [27].

Contributions. We define a Bayesian regret minimization objective and consider unobserved variables as parameters under an unknown prior distribution. We use empirical Bayes to derive an informative prior over the unobserved variables from expert data. We use the learned prior distribution to drive exploration in the online RL task, using approaches like posterior sampling [28]. We propose two procedures to learn such a prior: (1) a parametric approach that can utilize any existing knowledge about the parametric form of the prior distribution, and (2) a nonparametric approach that employs the principle of maximum entropy when such prior knowledge does not exist. We call our framework Experts-as-Priors or ExPerior for short. See Figure 1 for a goal-oriented RL example. ExPerior outperforms existing offline, online, and offline-online baselines in multi-armed bandits, Markov decision processes (MDPs), and partially observable MDPs. For multi-armed bandits, we find the Bayesian regret incurred by ExPerior is proportional to the entropy of the optimal action under the prior distribution, aligning with the entropy of expert policy if the experts are optimal. We introduce a frequentist algorithm for multi-armed bandits and prove a Bayesian regret bound proportional to a term that closely resembles the entropy of the optimal action. Our results suggest using the entropy of expert demonstrations to evaluate the impact of unobserved factors.

2 Related Work

Our work is an addition to the recent body of reinforcement learning research that leverages offline demonstrations to speed up online learning [29, 10, 30, 7, 9]. Classic algorithms such as DDPGfD [31] and DQfD [32] achieve this by combining imitation learning and RL. They modify DDPG [5] and DQN [1] by warm-starting the algorithms' replay buffers with expert trajectories and ensuring that the offline data never gets overridden by online trajectories. Closely related to our study is the meta-RL literature, which aims to accelerate learning in a given RL task by using

prior experience from related tasks [33, 34, 35]. These papers present model-agnostic meta-learning training objectives to maximize the expected reward from novel tasks as efficiently as possible.

Two unique features distinguish our problem from the settings considered above. First, our setting assumes heterogeneity within the offline data and with the online RL task that is unobserved to the learner, while the (optimal) experts are privy to that heterogeneity. Second, we assume the learner will only interact with one online task, making our setup similar to zero-shot meta-RL [27, 36, 37]. Most similar to our work is the ExPLORe algorithm [38], which assigns optimistic rewards to the offline data during the online interaction and runs an off-policy algorithm using both online and labelled offline data as buffers. For our setting, the algorithm incentivizes the learner to explore the expert trajectories, leading to faster convergence. We consider this work one of our baselines.

Our methodology utilizes only the state-action trajectory data from expert demonstrations without task-specific information or reward labels. Other similar methods require additional offline information. For example, Nair et al. [30] assume that the offline data contains the reward labels and use that to pre-train a policy, which is then fine-tuned online. Mendonca et al. [39] require task labelling for each trajectory and use the offline data to learn a single meta-learner. Similarly, Zhou et al. [40] and Rakelly et al. [41] require the task label and reward labels. They then infer the task during online interaction and use the task-specific offline data. Lee et al. [42] requires a large amount of noisy expert data with reward labels, in addition to the optimal trajectory data, for good performance. Finally, our methodology builds on posterior sampling [43]. Hao et al. [22, 23] consider a similar problem using posterior sampling to leverage offline expert demonstration data to improve online RL. However, they assume homogeneity between the expert data and online tasks. In contrast, our setting accounts for heterogeneity.

3 Problem Setup

Decision Model for Unobserved Heterogeneity. To account for unobserved heterogeneity, we consider a generalization of finite-horizon Markov Decision Processes (MDPs) with a notion of probabilistic contextual variables [44, 13, 21]. The underlying model for the MDP will additionally depend on an unobserved variable that encapsulates the information hidden from the learner. For example, consider a personalized education setup where teaching a student corresponds to a task, and the learning agent can observe students' characteristics, like their demographic status and grades. Other factors, such as the student's learning style (e.g., visual learners or self-study), may not be readily available, even though they are important in determining the optimal teaching style.

Let C be the set of all unobserved context variables that can describe the unobserved heterogeneity of the decision-making problem (e.g., the set of all possible learning styles). A contextual MDP M = ( S , A , T , R, H, ρ, µ ⋆ ) is parameterized by states S , actions A , transition function T : S × A × C → ∆( S ) , reward function R : S × A × C → ∆( R ) , horizon H > 0 , initial state distribution ρ ∈ ∆( S ) , and context distribution µ ⋆ . We assume the transition/reward functions and µ ⋆ are unknown, and for simplicity, ρ does not depend on the context variable. For each unobserved context c ∼ µ ⋆ , we consider T episodes, where at the beginning of each episode t ∈ [ T ] , an initial state s 1 ∼ ρ is sampled. Then, at each timestep h ∈ [ H ] , the learner chooses an action a h ∈ A , observes a reward r h ∼ R ( s h , a h , c ) and the next state s h +1 ∼ T ( s h , a h , c ) . Without loss of generality, we assume the states are partitioned by [ H ] to make the notation invariant to the timestep. Let Π be the set of all Markovian policies. For a policy function π : S → ∆( A ) ∈ Π and context variable c , we define the value function V c ( π ) = E [ ∑ H h =1 r h ∣ ∣ ∣ π, c ] and the Q-function as Q π c ( s, a ) := E [ ∑ H h ′ = h r h ′ ∣ ∣ ∣ s h = s, a h = a, π, c ] for all s ∈ S , a ∈ A . Moreover, we define the optimal policy for a context variable c ∈ C as π c := arg max π ∈ Π V c ( π ) . Note that since the context variable is unobserved, the learner's policy will not depend on it. The learning agent's goal is to learn history-dependent distributions p 1 , . . . , p T ∈ ∆(Π) over Markovian policies to minimize the expected regret, defined as Reg := E c µ ⋆ [ ∑ T t =1 V c ( π c ) E π t p t [ V c ( π t )] ] .

∼ -∼

In the personalized education example, the above setup assumes a fixed distribution µ ⋆ over the set of learning styles and aims to minimize expected regret over the population of students. Our setup and regret assume the unobserved factors remain fixed during training. This captures scenarios wherein the unobserved variables correspond to less-variant factors (a student's learning style is more likely to remain unchanged). No learning algorithm can control the regret value if we allow the unobserved factors to change arbitrarily throughout T episodes without access to hidden information; consider

a two-armed bandit with a context value drawn with uniform probability from C = { c 1 , c 2 } that can change at each episode. Assume the expected reward of the first arm under c 1 and c 2 is one and zero, respectively, and it is reversed for the other arm. Any algorithm that does not have access to c would result in linear regret since each action is sub-optimal with a probability of 0 . 5 , independent of the algorithm's choice.

Remark. Our setup can be formulated as a Bayesian model parameterized by C , and our regret can be seen as the Bayesian regret of the learner. However, the distribution µ ⋆ is not the learner's prior belief about the true model as it is often formulated in Bayesian learning, but a distribution over potential contexts that the learner can encounter. Our setup can thus be seen as a meta-learning problem. In fact, it is zero-shot meta-learning since we do not assume having access to more than one online RL task during training - we only learn the prior distribution using the offline data.

Expert Demonstrations. In addition to the online setting described above, we assume the learner has access to an offline dataset of expert demonstrations D E, where each demonstration τ E = ( s 1 , a 1 , s 2 , a 2 , . . . , s H , a H , s H +1 ) refers to an interaction of the expert with a decision-making task during a single episode, containing the actions made by the expert and the resulting states. We assume that the unobserved context variables for D E are drawn i.i.d. from distribution µ ⋆ , and the expert had access to such unobserved variables (private information) during their decision-making. Moreover, we assume the expert follows a near-optimal strategy [22, 23].

Assumption 1 (Noisily Rational Expert) . For any c ∈ C , experts select actions based on a distribution defined as p E ( a | s ; c ) ∝ exp { β · Q π c c ( s, a ) } , for all s ∈ S , a ∈ A , and some known competence value of β ∈ [0 , ∞ ] . In particular, the expert follows the optimal policy if β →∞ .

We assume experts do not provide any rationale for their strategy, nor do we have access to rewards in the offline data; this is a combination of imitation and online learning rather than offline RL [22].

4 Experts-as-Priors Framework for Unobserved Heterogeneity

Our goal is to leverage offline data to help guide the learner through its interaction with the decisionmaking task. The key idea is to use expert demonstrations to infer a prior distribution over C and then to use a Bayesian approach such as posterior sampling [28] to utilize the inferred prior for a more informative exploration. If the current context is from the same distribution of contexts in the offline data, we expect that using such priors will lead to faster convergence to optimal trajectories compared to the commonly used non-informative priors. Consider the personalized education example. Suppose we have gathered offline data on an expert's teaching strategies for students with similar observed information like grade, age, location, etc. The teacher can observe more finegrained information about the students that is generally absent from the collected data (e.g., their learning style). Our work relies on the following observation: The space of the optimal strategies for students with similar observed information but different learning styles is often much smaller than the space of all possible strategies. With the inferred prior distribution, the learner needs only to focus on the span of potentially optimal strategies for a new student, allowing for significantly more efficient exploration.

We resort to empirical Bayes and use maximum marginal likelihood estimation [45] to construct a prior distribution from D E . Given a probability distribution (prior) µ on C , the marginal likelihood of an expert demonstration τ E = ( s 1 , a 1 , s 2 , a 2 , . . . , s H , a H , s H +1 ) ∈ D E is given by

Weaimto find a prior distribution to maximize the log-likelihood of D E under the model described in (1). This is equivalent to minimizing the KL divergence between the marginal likelihood P E and the empirical distribution of expert demonstrations, which we denote by ̂ P E . In particular, we form an uncertainty set over the set of plausible priors as P ( ϵ ) := { µ ; D KL ( ̂ P E ∥ ∥ ∥ P E ( · ; µ ) ) ≤ ϵ } , where the value of ϵ can be chosen based on the number of samples so the uncertainty set contains the true prior with high probability [27]. However, the set of plausible priors does not uniquely identify the appropriate prior. In fact, even for ϵ = 0 , P ( ϵ ) can have infinite plausible priors. To solve this ill-posed problem, we propose two approaches, parametric and nonparametric prior learning.

Parametric Experts-as-Priors. For settings where we have existing knowledge about the parametric form of the prior, we can directly apply maximum marginal likelihood estimation to learn it. In

particular, we define the parametric expert prior as the following. Note that we can calculate the gradients of the marginal likelihood using the score function estimator [46].

Definition 1 (Parametric Expert Prior) . Let Θ be a set of plausible prior distribution parameters (e.g., Beta distribution parameters for a Bernoulli bandit). We call µ θ ⋆ a parametric expert prior, iff θ ⋆ ∈ arg min θ ∈ Θ ∑ τ ∈D E -log P E ( τ ; µ θ ) .

Nonparametric Experts-as-Priors. For settings where there is no existing knowledge on the parametric form of the prior, we can employ the principle of maximum entropy to choose the least informative prior that is compatible with expert data:

Definition 2 (Max-Entropy Expert Prior) . Let µ 0 be a non-informative prior on C (e.g., a uniform distribution). Given some ϵ > 0 , we define the maximum entropy expert prior µ ME as the solution to the following optimization problem:

Note that the set of plausible priors P ( ϵ ) is a convex set, and therefore, (2) is a convex optimization problem. We can derive the solution to problem (2) using Fenchel's duality theorem [47, 48]:

Proposition 1 (Max-Entropy Expert Prior) . Let N = |D E | be the number of demonstrations in D E. For each c ∈ C and demonstration τ E = ( s 1 , a 1 , s 2 , a 2 , . . . , s H , a H , s H +1 ) ∈ D E, define m τ E ( c ) as the (partial) likelihood of τ E under c , i.e., m τ E ( c ) = ∏ H h =1 p E ( a h | s h ; c ) T ( s h +1 | s h , a h , c ) .

Denote m ( c ) ∈ R N as the vector with elements m τ E ( c ) for τ E ∈ D E. Moreover, let λ ⋆ ∈ R ≥ 0 be the optimal solution to the Lagrange dual problem of (2) . Then, the solution to optimization (2) is:

where { α n } ∞ n ='1 is a sequence converging to the following supremum:

The proof is provided in Appendix A.3. Instead of solving for λ ⋆ , we set it as a hyperparameter and then solve (3). Even though Proposition 1 requires the correct form of Q-functions for different values of c , we will see in the following sections that we can parameterize the Q-functions and treat those parameters as a proxy for the unobserved factors. Once such a prior is derived, we can employ any Bayesian approach for decision-making. We provide a pseudo-algorithm for ExPerior in Algorithm 1. The following sections will detail the algorithm for bandits and MDPs.

5 Learning in Bandits

K -armed Bandits. For K -armed bandits, note that S = ∅ , H = 1 , and A = { 1 , . . . , K } . Each expert demonstration τ E = a will be the pulled arm by the expert for a particular bandit, and the (partial) likelihood function in Proposition 1 can be simplified as m τ E ( c ) = p E ( a ; c ) . This likelihood function only depends on the context variable c through the expert policy p E , and since p E only depends on c through the mean reward function (Assumption 1), we can consider the set of mean reward functions as a proxy for the unobserved context variables C . e.g. in a Bernoulli K -armed bandit setting, we can define C Ber = { a ↦→⟨ e a , ϑ ⟩ ; ϑ ∈ [0 , 1] K } .

Oracle-TS

ExPerior-Param

Oracle-TS (SGLD)

ExPerior-MaxEnt

Na¨ ıve-TS

Na¨ ıve-UCB

UCB-ExPLORe

BC

0

50

100

150

200

250

300

Bayesian Regret

Low Entropy

Mid Entropy

High Entropy

Posterior Sampling. With the above parameterizations of C , we can use Proposition 1 to derive the maximum entropy prior distribution over the context parameters. However, we cannot sample from the exact posterior since the derived prior is not a conjugate prior for standard likelihood functions. Instead, we resort to approximate posterior sampling via stochastic gradient Langevin dynamics (SGLD) [49]. We call this method ExPerior-MaxEnt in our experiments. We also employ a parametric approach as discussed in section 4, which we call ExPerior-Param . In particular, we use the Beta distribution as our prior model and learn the parametric expert prior in Definition 1.

In the following, we evaluate our approach compared to online methods that do not use expert data and offline behaviour cloning. We provide an empirical regret analysis for ExPerior based on the informativeness of expert data, number of actions, and number of training episodes. We also discuss the robustness of ExPerior to misspecified expert models and the advantage of ExPerior-MaxEnt to ExPerior-Param when the parametric prior model is misspecified. To characterize the effect of expert data on the learner's performance, we propose an alternative for K -armed bandits inspired by the successive elimination and derive a Bayesian regret bound for it.

Experiments. We consider K -armed Bernoulli bandits for our experimental setup. We evaluate the learning algorithms in terms of the Bayesian regret over multiple (prior) distributions µ ⋆ over the unobserved contexts. In particular, we consider up to N µ ⋆ = 64 different beta distributions, where their parameters are chosen to span a different range of heterogeneity, consisting of tasks with various expert data informativeness. To estimate the Bayesian regret, we sample N task = 128 bandit tasks from each prior distribution and calculate the average regret. We use N E = 1000 expert demonstrations for each prior distribution in our experiments. We compare ExPerior to the following baselines: (1) Behaviour cloning ( BC ), which learns a policy by minimizing the cross-entropy loss between the expert demonstrations and the agent's policy solely based on offline data. (2) Na¨ ıve Thompson sampling ( Na¨ ıve-TS ) that chooses the action with the highest sampled mean from a posterior distribution under an uninformative prior. (3) Na¨ ıve upper confidence bound ( Na¨ ıve-UCB ) algorithm that selects the action with the highest upper confidence bound. Both Na¨ ıve-TS and Na¨ ıve-UCB ignore expert demonstrations. (4) UCB-ExPLORe , a variant of the algorithm proposed by Li et al. [38] tailored to bandits. It labels the expert data with optimistic rewards and then uses it alongside online data to compute the upper confidence bounds for exploration, and (5) Oracle-TS , which performs exact Thompson sampling with the true prior distribution µ ⋆ . For a fair comparison, we also consider a variant of Oracle-TS , which uses SGLD for approximate posterior sampling.

Comparison to baselines. Figure 2 demonstrates the average Bayesian regret for various prior distributions over T = 1 , 500 episodes with K = 10 arms. To better understand the effect of expert data, we categorize the prior distributions by the entropy of their optimal actions into low entropy (less than 0.8), high entropy (greater than 1.6), and medium entropy. Oracle-TS and ExPerior-Param outperform other baselines, yet the performance of ExPerior-MaxEnt is comparable to the SGLD variant of Oracle-TS . This indicates that the maximum entropy prior derived from Proposition 1 closely approximates the true prior distribution, µ ⋆ , and the performance difference with Oracle-TS is primarily due to approximate posterior sampling. Moreover, the pure online algorithms Na¨ ıve-TS and Na¨ ıve-UCB , which disregard expert data, display similar performance across different entropy levels, contrasting with other algorithms that show significantly reduced regret in low-entropy contexts. This underlines the impact of expert data in settings where the unobserved confounding has less effect on the optimal actions. Specifically, in the extreme case of no unobserved heterogeneity, BC is anticipated to yield optimal performance. Additionally, Na¨ ıve-UCB surpasses UCB-ExPLORe in medium and high entropy settings, possibly due to the over-optimism of the reward labelling in Li et al. [38], which can hurt the performance when the expert demonstrations are uninformative.

Empirical regret analysis for Experts-as-Priors. We examine how the quality of expert demonstrations affects the Bayesian regret achieved by ExPerior. Settings with highly informative demon-

(a)

(b)

0

.

0

0

.

5

1

.

0

Entropy of Optimal Action

0

25

50

24

20

32

40

Number of Arms,

K

0

50

Bayesian Regret

0

2

Entropy of Optimal Action

0

25

50

0

1000

Episodes,

T

0

20

ExPerior-MaxEnt

ExPerior-Param

Oracle-TS

±

±

±

±

±

±

±

±

±

strations, where unobserved factors minimally affect the optimal action, should exhibit near-zero regret since there is no diversity in the unobserved contexts, and the experts are near-optimal. Conversely, in scenarios where unobserved factors significantly influence the optimal actions, we anticipate the regret to align with standard online regret bounds, similar to the outcomes of Thompson sampling with a non-informative prior. We conduct trials with ExPerior and Oracle-TS across various numbers of arms over T = 1 , 500 episodes, calculating the mean and standard error of Bayesian regret across distinct prior distributions. As depicted in Figure 3 (a), both ExPerior and Oracle-TS yield sub-linear regret relative to K and T , comparable to the established regret bound of O ( √ KT ) for Thompson sampling. However, the middle panel indicates that the regret of ExPerior is proportional to the entropy of the optimal action, having an almost linear relationship. This observation seems to be in contrast with the standard Bayesian regret bounds for Thompson sampling under correct prior that have shown a sublinear relationship of O ( √ Ent ( π c ) ) , where Ent ( π c ) denotes the entropy of the optimal action under µ ⋆ [50]. We analyze this observation in section 5.1.

Ablations. We also run additional experiments to assess the robustness of ExPerior to misspecified experts. We create expert data from different experts with various competence levels, such as optimal, noisily rational, and random-optimal experts, where the latter chooses an action optimally with a fixed probability and randomly otherwise. Table 1 shows ExPerior's robustness to different expert models. Setting β = 10 for training ExPerior-MaxEnt and β = 1 for ExPerior-Param achieves consistent out-performance among different expert types. Moreover, We evaluate the advantage of learning nonparametric max-entropy prior over misspecified parametric priors in Table 2. Even though ExPerior-Param with a Beta prior outperforms ExPerior-MaxEnt , ExPerior-MaxEnt is superior to ExPerior-Param if the prior mismatches the correct form (e.g., Gaussian or Gamma).

5.1 An Alternative Frequentist Approach for K -armed Bandits

To analyze the effect of expert data on the Bayesian regret, we devise an alternative frequentist approach, based on the successive elimination algorithm [51], which follows a similar intuition to Experts-as-Priors. In particular, we prove a bound on its Bayesian regret and show that the derived bound is proportional to a term that closely resembles the entropy of the optimal action, showing that the observation in the middle panel of Figure 3 (a) is consistent within different approaches.

±

±

±

Algorithm 2 Successive Elimination with Expert Sampling

1: Input: Episodes T , Arms A , expert policy ̂ P E , step size p min , unknown c ∼ µ ⋆ , and δ ∈ (0 , 1) .

2: for t = 1 . . . T do

3: try an active arm a with a relative frequency of ⌈ ̂ P E ( a ) p min ⌉ . // all arms are active at t = 0 . // n t ( a ) is the number of times arm a is pulled by episode t and V t c ( a ) is its empirical mean reward.

4: increment n t ( a ) and update V t c ( a ) .

5:

construct UCB

7:

t

a

c

=

V

t

(

a

)+

√

log (4

T

4

K/δ

)

/

2

n

t

(

a

)

t

a

=

V

t

(

6: de-activate all arms a s.t. ∃ a ′ with UCB a ≤ LCB a ′ , and normalize ̂ P E .

end for

The idea of successive elimination is to identify suboptimal arms and deactivate them over time. In particular, it runs a uniform sampling policy among active arms and builds confidence intervals for each. It then deactivates all the arms with an upper confidence bound smaller than at least one arm's lower confidence bound. We modify this algorithm using the policy derived from expert demonstrations instead of a uniform sampling policy. Recall that in K -armed bandits, each expert trajectory τ E represents the pulled arm by the expert. Hence, the empirical distribution of expert demonstrations can be seen as a sampling policy over different arms. To simplify the analysis, we employ a deterministic sampling approach by pulling each arm a fixed number of times based on its probability. To do so, we discretize the expert policy with a step size p min , which leads to a relative frequency of ⌈ ̂ P E ( a ) /p min ⌉ for an arm a . In particular, we can choose p min = min a ; ̂ P E ( a ) =0 ̂ P E ( a ) . We provide the concrete algorithm in Algorithm 2 and prove the following Bayesian regret bound:

̸

K/δ

̸

Theorem 2. Consider a stochastic K -armed bandit and let p be the empirical expert policy. Assume that (i) the mean reward function is bounded in [0 , 1] for all arms, (ii) T ≥ 1 min a ; p ( a ) =0 p ( a ) , (iii) the expert is optimal, i.e., ∀ a ∈ A : p ( a ) = P E ( a ; µ ⋆ ) and β → ∞ , and (iv) the learner follows Algorithm 2. Then, with probability at least 1 -δ ,

̸

See Appendix A.4 for the proof. Two terms in (4) depend on expert data: (1) The relative standard deviation between any two pairs of arms and (2) a scaling factor that depends on the magnitude of probability that the arms are optimal. For homogeneous demonstrations, where the expert data only includes one unique pulled arm, the standard deviation (Term 1) is zero, resulting in zero regret. However, in extreme heterogeneity, where the empirical expert distribution is uniform over the arms, we have Reg ≲ K √ KT log T . 2 Finally, to assess the relationship between the regret bound and the entropy of the expert data, we fix K = 2 , T = 100 , and plot the bound from (4) as a function of the entropy of the optimal action for various prior distributions. Figure 3 (b) demonstrates a linear relationship, similar to the regret incurred by ExPerior in Figure 3 (a). This observation opens up new directions to further analyze the regret for ExPerior and similar approaches in MDPs.

6 Learning in Markov Decision Processes (MDPs)

For MDPs, we need to parameterize both the mean reward and transition functions. However, we assume the transition functions are invariant to the context variables to simplify our methodology and avoid extra modelling assumptions. Under this assumption, it is sufficient to parameterize the optimal Q-functions, e.g., using a deep Q-network (DQN) and treat those parameters as a proxy for the unobserved context variables, i.e., C MDP := { ( s, a ) ↦→ Q ( s, a ; θ ) ; θ ∈ Θ } , where Θ is the set of parameters for a DQN. We can then derive a closed-form log-pdf of the posterior distribution under the maximum entropy prior. See Appendix A.5 for details. The derived posterior log-pdf can

±

±

±

±

and LCB

c

a

)

-

√

log (4

T

4

)

/

2

n

t

(

a

)

.

±

±

±

±

±

±

±

±

then be used as the loss function for DQN Langevin Monte Carlo [52, 53] as the counterpart for Thompson sampling with SGLD. However, running Langevin dynamics can lead to highly unstable policies due to the complexity of the optimization landscape in DQNs. Instead, we use a heuristic that combines the learned prior distribution with bootstrapped DQNs [54].

The original method of Bootstrapped DQNs utilizes an ensemble of L randomly initialized Qnetworks. It samples a Q-network uniformly at each episode and uses it to collect data. Then, each Q-network is trained using the temporal difference loss on parts of or possibly the entire collected data. This method and its subsequent iterations [55, 56, 57] achieve deep exploration by ensuring diversity among the learned Q-networks. To incorporate Bootstrapped DQN into the ExPerior framework and utilize the expert data, we can formulate the ensemble as a discrete prior distribution over the Q-networks. Let θ ens = ( θ 1 ens , . . . , θ L ens ) be the parameter vector for an ensemble of Q-functions. We can define the ensemble prior, parameterized by θ ens , as µ θ ens ( θ ) := 1 L ∑ L i =1 I ( θ i ens = θ ) for any θ ∈ Θ . Based on this prior model, we can learn the parametric expert prior using maximum marginal likelihood estimation, as formulated below.

Proposition 3 (Ensemble Marginal Likelihood) . Consider a contextual MDP M = ( S , A , T , R, H, ρ, µ ⋆ ) . Assume the transition function T does not depend on the context variables and Assumption 1 holds. Then, the negative marginal log-likelihood of expert data D E under the ensemble prior µ θ ens is upper bounded by

where β is the competence level of the expert in Assumption 1.

Proposition 3 is proved in Appendix A.6. We can then initialize the Q-networks in the Bootstrapped DQNmethod using ensemble parameters that minimize the above upper bound. We will refer to this method as ExPerior-Param . As an alternative approach, instead of minimizing the above upper bound, we can match the discrete prior distribution µ θ ens to the max-entropy prior by initializing the Q-functions in the ensemble with parameters sampled from the max-entropy expert prior. In particular, we can apply SGLD on the log-pdf of the max-entropy prior derived in Appendix A.5. We will refer to this approach as ExPerior-MaxEnt .

Experimental Setup. One challenge in RL is the reward sparsity , where the learner needs to explore the environment deeply to observe rewards. Utilizing expert demonstrations can significantly improve the efficiency of exploration. Here, we focus on 'Deep Sea,' a sparse-reward tabular RL environment proposed by Osband et al. [56] to assess deep exploration for different RL methods. The environment is an M × M grid, where the agent starts at the top-left corner of the map, and at each time step, it chooses an action from A = { left , right } to move to the left or right column, while going down by one row. In the original version of Deep Sea, the goal is always on the bottom-right corner of the map. We introduce unobserved contexts by defining a distribution over the goal columns while keeping the goal row the same. We consider four types of goal distributions where the goal is situated at (1) the bottom-right corner of the grid, (2) uniformly at the bottom of any of the right-most M 4 columns, (3) uniformly at the bottom of any of the right-most M 2 columns, and (4) uniformly at the bottom of any of the M columns. We set M = 30 and generate N = 1 , 000

0

1000

2000

0

.

0

0

.

5

1

.

0

Average Reward

0

1000

2000

0

1000

2000

0

1000

2000

Episodes,

T

Na¨ ıve Boot-DQN

ExPerior-MaxEnt (Ours)

ExPerior-Param (Ours)

ExPLORe

samples from the optimal policies as offline expert demonstrations. To further evaluate ExPerior and showcase its applicability to partially-observed MDP, we also consider the 'Frozen Lake' environment, which requires the learner to navigate to a goal while avoiding hazards [17]. The learner cannot observe the hazard location, while the expert has access to the whole map. Taking action, reaching the goal, and hitting the hazard incur rewards of -2, 20, and -100, respectively. The frozen lake map is 5 × 5 , where the hazard (weak ice) is randomly located in the interior squares. We consider different settings with 2, 5, 7, and 9 potential locations for the hazard. At the start of each episode, the hazard will be chosen randomly within the potential locations. We generate N = 1 , 000 samples from noisily rational experts with different competence levels for this environment.

Baselines. We compare ExPerior to the following: (1) ExPLORe , proposed by Li et al. [38] to accelerate off-policy reinforcement learning using unlabeled prior data. In this method, the offline demonstrations are assigned optimistic reward labels generated using the online data with regular updates. This information is then combined with the buffer data to perform off-policy learning. (2) Na¨ ıve Boot-DQN , which is the original Bootstrapped DQN with randomly initialized Q-networks [54].

Deep Sea Results. Figure 4 demonstrates the average reward per episode achieved by the baselines for T = 2 , 000 episodes. For each goal distribution, we run the baselines with 30 different seeds and take the average to estimate the expected reward. ExPerior outperforms the baselines in all instances. However, the gap between ExPerior and the fully online Na¨ ıve Boot-DQN , which measures the effect of using the expert data, decreases as we go from the low-entropy setting (upper left) to the high-entropy distribution over the contexts (bottom right). This is consistent with the empirical and theoretical results discussed in section 5 and confirms our expectation that the expert demonstrations may not be helpful under strong unobserved confounding (strong heterogeneity). The ExPLORe baseline substantially underperforms, even compared to the fully online Na¨ ıve Boot-DQN (except for the first distribution with zero-entropy). We suspect this is because ExPLORe uses actor-critic methods as its backbone model, which are shown to struggle with deep exploration [58].

Frozen Lake Results. We run all the baselines for 90 , 000 steps with 30 different seeds. Table 3 shows the average reward after 500 evaluation steps at the end of the training. ExPerior outperforms the baselines in almost all instances except for the case of β = 0 . 1 , which corresponds to a nearly random expert. On the other hand, ExPLORe achieves near-optimal results for β = 0 . 1 . We hypothesize that ExPLORe 's performance is mainly due to the superiority of their base actor-critic model since it can achieve near-optimal performance even when the expert trajectories are low-quality.

7 Conclusion

We introduce the Experts-as-Priors (ExPerior) framework, a novel empirical Bayes approach to address the problem of sequential decision-making using expert demonstrations with unobserved heterogeneity. We ground our methodology in the maximum entropy principle to infer a prior distribution from expert data that guides the learning process in different setttings, including bandits, Markov decision processes (MDPs), and partially-observed MDPs. Our experimental evaluations demonstrate that we can effectively leverage the expert demonstrations to enhance learning efficiency under unobserved confounding. In multi-armed bandits, we illustrated through empirical analysis that the Bayesian regret incurred by our method is proportional to the entropy of the optimal action, highlighting its capacity to adapt based on the informativeness of the expert data. Our work offers a practical framework readily applied to a broad spectrum of decision-making tasks. One limitation of our work is the limited set of experiments, especially the lack of experiments with human-in-the-loop. Future directions include extending to more complex environments and further investigating the theoretical properties of our RL algorithm.

References

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529 (7587):484-489, 2016.

David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815 , 2017.

David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning , pages 387-395. Pmlr, 2014.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Andrew Wagenmaker and Aldo Pacchiano. Leveraging offline data in online reinforcement learning. In International Conference on Machine Learning , pages 35300-35338. PMLR, 2023.

Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprint arXiv:2210.06718 , 2022.

Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning , pages 15771594. PMLR, 2023.

Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 6292-6299. IEEE, 2018.

Serkan Cabi, Sergio G´ omez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, et al. Scaling data-driven robotics with reward sketching and batch reinforcement learning. arXiv preprint arXiv:1909.12200 , 2019.

Richard A. Briesch, Pradeep K. Chintagunta, and Rosa L. Matzkin. Nonparametric Discrete Choice Models With Unobserved Heterogeneity. Journal of Business & Economic Statistics , 28(2):291-307, 2010. ISSN 0735-0015. Publisher: [American Statistical Association, Taylor &Francis, Ltd.].

Lantao Yu, Tianhe Yu, Chelsea Finn, and Stefano Ermon. Meta-inverse reinforcement learning with probabilistic context variables. Advances in neural information processing systems , 32, 2019.

Nathan Kallus and Angela Zhou. Minimax-Optimal Policy Learning Under Unobserved Confounding. Management Science , 67(5):2870-2890, May 2021. ISSN 0025-1909, 1526-5501. doi: 10.1287/mnsc.2020.3699.

Andrew Bennett, Nathan Kallus, Lihong Li, and Ali Mousavi. Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , pages 1999-2007. PMLR, March 2021. ISSN: 2640-3498.

Sanjiban Choudhury, Mohak Bhardwaj, Sankalp Arora, Ashish Kapoor, Gireeja Ranade, Sebastian Scherer, and Debadeepta Dey. Data-driven planning via imitation learning. The International Journal of Robotics Research , 37(13-14):1632-1672, 2018.

Andrew Warrington, Jonathan W Lavington, Adam Scibior, Mark Schmidt, and Frank Wood. Robust asymmetric learning in pomdps. In International Conference on Machine Learning , pages 11013-11023. PMLR, 2021.

Aaron Walsman, Muru Zhang, Sanjiban Choudhury, Dieter Fox, and Ali Farhadi. Impossibly good experts and how to follow them. In The Eleventh International Conference on Learning Representations , 2022.

Idan Shenfeld, Zhang-Wei Hong, Aviv Tamar, and Pulkit Agrawal. TGRL: An algorithm for teacher guided reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 31077-31093. PMLR, 23-29 Jul 2023.

Junzhe Zhang, Daniel Kumor, and Elias Bareinboim. Causal imitation learning with unobserved confounders. Advances in neural information processing systems , 33:12263-12274, 2020.

Gokul Swamy, Sanjiban Choudhury, J Bagnell, and Steven Z Wu. Sequence model imitation learning with unobserved contexts. Advances in Neural Information Processing Systems , 35: 17665-17676, 2022.

Botao Hao, Rahul Jain, Tor Lattimore, Benjamin Van Roy, and Zheng Wen. Leveraging demonstrations to improve online learning: Quality matters. In International Conference on Machine Learning , pages 12527-12545. PMLR, 2023.

Botao Hao, Rahul Jain, Dengwang Tang, and Zheng Wen. Bridging imitation and online reinforcement learning: An optimistic tale. arXiv preprint arXiv:2303.11369 , 2023.

Luca Weihs, Unnat Jain, Iou-Jen Liu, Jordi Salvador, Svetlana Lazebnik, Aniruddha Kembhavi, and Alex Schwing. Bridging the imitation gap by adaptive insubordination. Advances in Neural Information Processing Systems , 34:19134-19146, 2021.

Leonardo Cella, Alessandro Lazaric, and Massimiliano Pontil. Meta-learning with stochastic linear bandits. In International Conference on Machine Learning , pages 1360-1370. PMLR, 2020.

Leonardo Cella and Massimiliano Pontil. Multi-task and meta-learning with sparse linear bandits. In Uncertainty in Artificial Intelligence , pages 1692-1702. PMLR, 2021.

Jay Mardia, Jiantao Jiao, Ervin T´ anczos, Robert D Nowak, and Tsachy Weissman. Concentration inequalities for the empirical distribution of discrete distributions: beyond the method of types. Information and Inference: A Journal of the IMA , 9(4):813-850, 2020.

Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems , 26, 2013.

Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087 , 2017.

Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 , 2020.

Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Roth¨ orl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817 , 2017.

Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI conference on artificial intelligence , 2018.

Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Metareinforcement learning of structured exploration strategies. Advances in neural information processing systems , 31, 2018.

Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. arXiv preprint arXiv:1803.11347 , 2018.

Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson. A survey of meta-reinforcement learning. arXiv preprint arXiv:2301.08028 , 2023.

Vinay Kumar Verma, Dhanajit Brahma, and Piyush Rai. Meta-learning for generalized zeroshot learning. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 6062-6069, 2020.

Julie Jiang, Kristina Lerman, and Emilio Ferrara. Zero-shot meta-learning for small-scale data from human subjects. In 2023 IEEE 11th International Conference on Healthcare Informatics (ICHI) , pages 311-320. IEEE, 2023.

Qiyang Li, Jason Zhang, Dibya Ghosh, Amy Zhang, and Sergey Levine. Accelerating exploration with unlabeled prior data. Advances in Neural Information Processing Systems , 36, 2024.

Russell Mendonca, Abhishek Gupta, Rosen Kralev, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Guided meta-policy search. Advances in Neural Information Processing Systems , 32, 2019.

Allan Zhou, Eric Jang, Daniel Kappler, Alex Herzog, Mohi Khansari, Paul Wohlhart, Yunfei Bai, Mrinal Kalakrishnan, Sergey Levine, and Chelsea Finn. Watch, try, learn: Meta-learning from demonstrations and reward. arXiv preprint arXiv:1906.03352 , 2019.

Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient offpolicy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning , pages 5331-5340. PMLR, 2019.

Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn in-context reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, Zheng Wen, et al. A tutorial on thompson sampling. Foundations and Trends® in Machine Learning , 11(1):1-96, 2018.

Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint arXiv:1502.02259 , 2015.

Bradley P Carlin and Thomas A Louis. Empirical bayes: Past, present and future. Journal of the American Statistical Association , 95(452):1286-1289, 2000.

Michael C Fu. Chapter 19 gradient estimation. Simulation , 13:575-616, 2006.

R Tyrrell Rockafellar. Convex analysis , volume 11. Princeton university press, 1997.

Miroslav Dud´ ık, Steven J Phillips, and Robert E Schapire. Maximum entropy density estimation with generalized regularization and an application to species distribution modeling. Journal of Machine Learning Research , 2007.

Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11) , pages 681-688, 2011.

Daniel Russo and Benjamin Van Roy. An information-theoretic analysis of thompson sampling. The Journal of Machine Learning Research , 17(1):2442-2471, 2016.

Eyal Even-Dar, Shie Mannor, and Yishay Mansour. Pac bounds for multi-armed bandit and markov decision processes. In Computational Learning Theory: 15th Annual Conference on Computational Learning Theory, COLT 2002 Sydney, Australia, July 8-10, 2002 Proceedings 15 , pages 255-270. Springer, 2002.

Vikranth Dwaracherla and Benjamin Van Roy. Langevin dqn. arXiv preprint arXiv:2002.07282 , 2020.

Haque Ishfaq, Qingfeng Lan, Pan Xu, A. Rupam Mahmood, Doina Precup, Anima Anandkumar, and Kamyar Azizzadenesheli. Provable and practical: Efficient exploration in reinforcement learning via langevin monte carlo. In The Twelfth International Conference on Learning Representations , 2024.

Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems , 29, 2016.

Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. Advances in Neural Information Processing Systems , 31, 2018.

Ian Osband, Benjamin Van Roy, Daniel J Russo, Zheng Wen, et al. Deep exploration via randomized value functions. J. Mach. Learn. Res. , 20(124):1-62, 2019.

Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy. Approximate thompson sampling via epistemic neural networks. In Uncertainty in Artificial Intelligence , pages 1586-1595. PMLR, 2023.

Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, et al. Behaviour suite for reinforcement learning. arXiv preprint arXiv:1908.03568 , 2019.

Jonathan M Borwein and Qiji J Zhu. Techniques of variational analysis, 2004.

A Proofs

A.1 Notation

Weassume C is a measurable set with an appropriate σ -algebra and there exists a probability measure µ 0 on C . We denote L p ( C , µ 0 ) as the space of all measurable functions f : C → R such that ∥ f ∥ p = (∫ C | f | p d µ 0 ) 1 /p < ∞ . Moreover, we define L ∞ ( C , µ 0 ) as the space of all essentially bounded measurable functions from C to R . Unless stated otherwise, we assume the probability measures are absolutely continuous w.r.t. µ 0 , and their density functions are in L 1 ( C , µ 0 ) . We may abuse the notation and use the same symbol for a probability measure and its Radon-Nikodym derivative w.r.t. µ 0 . Finally, we use E [ · ] to denote expectation under the probability measure µ 0 .

A.2 Useful Lemmas

Here, we state and prove a set of results that will be useful for the rest of this section. The first one is Fenchel's duality theorem:

Lemma 4 (Fenchel's Duality [59]) . Let X and Y be Banach spaces, let f : X → R ∪ { + ∞} and g : Y → R ∪ { + ∞} be convex functions and let A : X → Y be a bounded linear map. Define the primal and dual values p, d ∈ [ -∞ , + ∞ ] by the Fenchel problems

̸

where f ∗ and g ∗ are the Fenchel conjugates of f and g defined as f ∗ ( x ∗ ) = sup x ∈ X ⟨ x ∗ , x ⟩ -f ( x ) (similarly for g ), X ∗ is the dual space of X and ⟨· , ·⟩ is its duality pairing, and A ∗ : Y ⋆ → X ⋆ is the adjoint operator of A , i.e., ⟨ A ∗ y ∗ , x ⟩ = ⟨ y ∗ , Ax ⟩ . Suppose A dom ( f ) ∩ cont ( g ) = ∅ , where dom ( f ) := { x ∈ X ; f ( x ) < ∞} and cont ( g ) are the continuous points of g . Then, strong duality holds, i.e., p = d .

Proof. See the proof of Theorem 4.4.3 in Borwein and Zhu [59].

We can use Fenchel's duality to solve generalized maximum entropy problems. In particular, we prove a generalization of Theorem 2 in [48] for density functions in L 1 ( C , µ 0 ) :

Lemma 5. For any function µ ∈ L 1 ( C , µ 0 ) , define the extended KL divergence as

Moreover, assume a set of bounded feature functions m 1 , m 2 , . . . , m N : C → R is given and denote m as the vector of all N features. Consider the linear function A m : L 1 ( C , µ 0 ) → R N defined as

We define the generalized maximum entropy problem as the following:

for an arbitrary closed proper convex function ζ : R N → R . Then the following holds:

The dual optimization of (5) is given by

where ζ ⋆ is the convex conjugate function of ζ .

Denote α 1 , α 2 , . . . as a sequence in R N converging to supremum (6) , and define the following Gibbs density functions

Then,

Proof. Part 1: We first derive the convex conjugate of ψ . Note that ( L 1 ( C , µ 0 ) ) ⋆ = L ∞ ( C , µ 0 ) with the pairing

Hence, by Donsker and Varadhan's variational formula

Moreover, the adjoint operator of A m is given by A ⋆ m : R N → ( C → R ) :

Using (7) and (8) and Lemma 4 concludes the proof.

Part 2: Denote the primal and dual objective functions by

and their optimal values as P ∗ and D ∗ . For any ν ∈ L 1 ( C , µ 0 ) , note that

Using (9), we can re-write the dual objective function as:

Moreover, note that

Combining (10) and (11), we get

Now, fix an arbitrary ϵ > 0 , and consider a sequence of µ 1 , µ 2 , . . . ∈ L 1 ( C , µ 0 ) such that for all j ∈ N :

We can re-write (13) using the fact P ∗ = D ∗ = lim n →∞ D ( α n ) :

In particular, by setting ν = µ j in (12) and combining the result with (14), we get

Hence, lim j ∈∞ lim n →∞ D KL ( µ j ∥ ∥ µ α n Gibbs ) = 0 . From properties of the KL divergence, it follows that lim j →∞ P ( µ j ) = lim n →∞ P ( µ α n Gibbs ) , concluding the proof.

A.3 Max-Entropy Prior

Proposition 1. Let N = |D E | be the number of demonstrations in D E. For each c ∈ C and demonstration τ E = ( s 1 , a 1 , s 2 , a 2 , . . . , s H , a H , s H +1 ) ∈ D E, define m τ E ( c ) as the (partial) likelihood of τ E under c :

Denote m ( c ) ∈ R N as the vector with elements m τ E ( c ) for τ E ∈ D E. Moreover, let λ ⋆ ∈ R ≥ 0 be the optimal solution to the Lagrange dual problem of (2) . Then, the solution to optimization (2) is as follows:

where { α n } ∞ n =1 is a sequence converging to the following supremum:

Proof. We first simplify the KL-divergence between the empirical distribution of the expert trajectories ̂ P E and the marginal likelihood P E ( · ; µ ) :

Using the above equality, we can re-write the definition of uncertainty set P ( ϵ ) as

Therefore, we can re-write the optimization (2) as

where the extended KL divergence ψ ( µ ) is defined as:

Note that P ( ϵ ) is a convex set. To see this, consider µ 1 , µ 2 ∈ P ( ϵ ) . Then, for any 0 ≤ λ ≤ 1 , we have µ = (1 -λ ) µ 1 + λµ 2 ∈ P ( ϵ ) since E [ m τ · µ ] is linear in µ and -log is convex. Moreover, It is easy to see there exists a strictly feasible solution for (16) (e.g., consider the true distribution µ ⋆ over C ). Thus, strong duality holds, and we can form the Lagrangian function as

Given that λ ⋆ ∈ R ≥ 0 is the optimal solution to the Lagrange dual problem, the maximum entropy prior µ ME will be the solution to

Now, for each x ∈ R N , define the convex function ζ ( x ) := λ ⋆ N ( ∑ N i =1 -log x i ) . Moreover, for µ ∈ L 1 ( C , µ 0 ) , define A m ( µ ) := ( E [ m τ (1) · µ ] , E [ m τ (2) · µ ] , . . . , E [ m τ ( N ) · µ ]) . Then,

Combining (17) and (18), the maximum entropy prior µ ME is the solution to

Using Lemma 5 and noting that

concludes the proof.

A.4 Regret Bound for K -armed Bandit

Theorem 2. Consider a stochastic K -armed bandit and let p be the empirical expert policy. Assume that (i) the mean reward function is bounded in [0 , 1] for all arms, (ii) T ≥ 1 min a ; p ( a ) =0 p ( a ) , (iii) the expert is optimal, i.e., ∀ a ∈ A : p ( a ) = P E ( a ; µ ⋆ ) and β → ∞ , and (iv) the learner follows Algorithm 2. Then, with probability at least 1 -δ ,

̸

̸

Proof. Fix δ ∈ (0 , 1) and c ∈ C . Let E be the event that ∣ ∣ ∣ V t c ( a ) -V c ( a ) ∣ ∣ ∣ ≤ √ log(4 T 4 K/δ ) 2 n t ( a ) for all arms a ∈ A , all t ≤ T , and all T ∈ N , where n t ( a ) is the number of times that arm a was pulled by time t . Note that since T ≥ 1 p min , each arm will be pulled at least once and n t ( a ) ≥ 1 .

We first show that P ( E ) ≥ 1 -δ . Fix T , arm a , and t ≤ T . Suppose n t ( a ) = j for 1 ≤ j ≤ T . By Hoeffding's inequality, we have

Now, using the union bound over all episodes and all actions, we get

which concludes that P ( E ) ≥ 1 -δ .

The rest of the proof computes the regret for when E holds. For simplicity and without loss of generality, we assume all expert probabilities are dividable by p min . Recall that we follow a deterministic sampling approach and choose each arm according to its relative frequency p ( · ) p min for multiple batches, where each batch loops over all active actions. Let t a be the episode in which we eliminate an arm a in favour of another arm. Then, it is easy to show that

This lower bound corresponds to the case where no other arm is eliminated before eliminating a . Moreover, we have an upper bound for n t a ( a ) considering the worst-case scenario in which the only remaining arms are a and a c , where a c is the optimal action for unobserved context c :

Now, let Reg c ( a ) be the total regret contributed by the arm a for a given context c ∼ C . We can upper bound the regret as

where ( i ) holds since the confidence intervals of arm a and a c overlap at episode t a (otherwise, a would have been eliminated before t a ), and ( ii ) follows from the fact that t a ≤ T .

̸

Finally, we upper bound the Bayesian regret by taking the expectation of ∑ a = a c Reg c ( a ) over c ∼ C . Note that since the expert is optimal, we have p ( a ) = µ ⋆ ( a c = a ) for all k ∈ A .

̸

̸

̸

where ( i ) follows by partitioning C into { c ; c ∈ C , a c = a ′ } a ′ ∈A and choosing the worst-case context in each partition.

From above, we have

̸

where ( ii ) holds since 4 K/δ > 1 . Replacing p ( a ) p ( a )+ p ( a c ) with 1 -p ( a ′ ) p ( a )+ p ( a c ) concludes the proof.

A.5 Max-Entropy Expert Posterior for MDPs

Proposition 6 (Max-Entropy Expert Posterior for MDPs) . Consider a contextual MDP M = ( S , A , T , R, H, ρ, µ ⋆ ) . Assume the transition function T does not depend on the context variables. Moreover, assume the reward distribution is Gaussian with unit variance and Assumption 1 holds. Then, the log-pdf posterior function under the maximum entropy prior is given as:

where H T = { ( ( s t h , a t h , r t h , s t h +1 ) H h =1 ) T t =1 } is the history of online interactions, D E is the expert demonstration data, β is the competence level of the expert in Assumption 1, and { α ⋆ τ } τ ∈D E are derived from Proposition 1.

Remark. Wenote that, in principle, the ExPerior framework allows for context-dependent transition functions. In this case, the log-pdf in (22) provides an optimistic upper bound on the true posterior log-pdf function. See Hao et al. [23] for a similar analysis. We leave the general case for future work. Note that the second term of (22) is simply the log-pdf of the max-entropy prior.

Proof. Since the transition function is context-independent, the likelihood of an expert trajectory τ E can be simplified as:

The second term in (23) is constant in c . This implies that the likelihood function m τ E ( c ) will depend on c only through the expert policy, which itself is a function of optimal Q-functions by Assumption 1. Note that the second term in the definition of m τ E can be simply removed since we can re-weight the parameters α in the optimization step (3) of Proposition 1. Hence, assuming the deep Q-network is expressive enough, without loss of generality, we can re-define the likelihood function of an expert trajectory τ E = ( s 1 , a 1 , s 2 , a 2 , . . . , s H , a H , s H +1 ) as

̸

̸

̸

We can now write the log-pdf of the posterior distribution of θ given H T :

Now, given the Bellman equations, we can write the mean value of the reward function as

The reward distribution is Gaussian with unit variance. Therefore,

Moreover, by Proposition 1, the log-pdf of the maximum entropy expert prior is given as

Combining (24) to (26), we conclude the proof.

A.6 Ensemble Marginal Likelihood

Proposition 3. Consider a contextual MDP M = ( S , A , T , R, H, ρ, µ ⋆ ) . Assume the transition function T does not depend on the context variables and Assumption 1 holds. Then, the negative marginal log-likelihood of expert data D E under the ensemble prior µ θ ens is upper bounded by

where β is the competence level of the expert in Assumption 1.

Proof. Recalling (1), the log-likelihood of the expert trajectories D E under µ θ ens is given by

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: To the best of our knowledge we theoretically and empirically validate our claims.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our work in the concluding section of the manuscript.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide proofs for the theoretical results in this work.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experiments are reproducible by virtue of the code provided. Our code is available at https://github.com/vdblm/experior

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We release code for this work. Our code is available at https://github. com/vdblm/experior

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We release code to reproduce all the results which provides sufficient detail to answer the above questions. Our code is available at https://github.com/vdblm/ experior

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports error bars and confidence intervals.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have used 110 GPU-hours for all the experiments on Quadro RTX 6000.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our algorithms are evaluated on synthetic and simulated RL-based environments and we do not anticipate any direct or indirect potential positive or negative societal impacts.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not anticipate misuse from the release of our models due to the nature of the evaluations we conduct on synthetic and RL-based datasets.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the relevant papers in cases where we needed to reproduce their code for comparison against our work.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide software code to reproduce our results.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not study our methodology using data from human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not use data that requires IRB approval to study our methodology.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nImitation learning in a setting where experts have contextual information not available to the imitator. Imitator sees state-action (no reward) trajectories for the expert data. The paper proposes to use the expert data as prior and then do meta-RL to learn a policy.\n\nSTRENGTHS:\n- The problem setting considered is important and there isn't too much prior work in it.\n- The proposed method seems sensible.\n- The paper provides theoretical and empirical justification for the proposed method.\n\nWEAKNESSES:\n- The empirical evaluation is limited to simple domains and it is not discussed why that is the case. It would be good to include an explanation on why this was not scaled up to some more complex environments.\n- Comparison to ExPLORe: if there is a way to use the same actor-critic algorithm for the proposed method and the comparison that would strengthen the paper. As the text says, now it is not clear whether the difference comes from the online RL part or from the difference in how they use the expert data.\n- ADVISOR [1] targets the same problem setting. Differences to that algorithm should be discussed. And maybe a pointer should be given on why their method is demonstrated in high-dimensional domains and this isn't. Not saying that this has to be evaluated in high-dimensional domains, but that it is not immediately clear why this isn't.\n- Figure 4\n    - Explain the pane meanings explicitly in the caption.\n    - Explain the axes explicitly in the caption. I.e., are these learning curves?\n    - How come the naive baseline does worse in the low entropy setting than in the high entropy one?\n\n[1] Luca Weihs, Unnat Jain, Iou-Jen Liu, Jordi Salvador, Svetlana Lazebnik, Aniruddha Kembhavi, and Alex Schwing. Bridging the imitation gap by adaptive insubordination. Advances in Neural Information Processing Systems, 34:19134–19146, 2021.\n\nQUESTIONS:\n- Why is human-in-the-loop highlighted as a limitation?\n- Can you give a more fleshed out example of how the problem of unobserved contextual information might appear in a realistic problem?\n- Can you lay out an idea on how to scale the method?\n\nLIMITATIONS:\nLimitations are discussed but some of the limitations highlighted seem a little strange. See questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a 2-stage learning method for sequential decision making, wherein step 1 involves leaning a prior from expert demonstrations and step 2 is online RL where that learnt prior is used. The main problem being addressed is that of heterogeneity of data and/or contexts provided by the expert and those encountered by the online learning agent.\n\nSTRENGTHS:\n- Relevant problem: Finding useful, scalable and efficient ways to leverage offline expert demonstrations for online decision making is of key interest to the community. \n\n- The proposed method combines imitation learning with online model-based RL, paying attention to the issue of heterogeneity between the available contexts that gave rise to the offline data, that might be unavailable during the online RL step.\n\n- The problem is well-motivated and the approach is easy to understand conceptually; contributions are clearly outlined. \n\n- Presentation (writing, figures etc) is mostly clear, with some minor comments included in the below section.\n\nWEAKNESSES:\nClarity\n- Notation: even with the notation paragraph, I find it quite difficult to follow. E.g. I think some `t` superscripts are missing (the definition of the q-function); should $\\mathcal{C}$ be part of the definition of the MDP?\n- Terminology: I'm a little confused with the terminology \"parametric\" vs \"non-parametric\". Unless I'm missing something, the latter is still parametric as you are optimising over a family of distributions?\n- Typo: in figure 1, annotation below \"Step 3\" is repeated, I guess one should be $\\mu_{\\theta}*$.\n- Figure 1: clarity can be improved if you write down what is being learnt and what is being updated.\n- I found the algorithm box quite useful, will be great to have it in the main paper (e.g. next to Figure 1).\n\nMax-entropy prior:\n- Clearly, if $\\mu_0  \\in \\mathcal{P}(\\epsilon)$, then we get a trivial solution, $\\mu_{ME}=\\mu_0$ (usually Uniform), so do you need an assumption on ${P}(\\epsilon)$ to get a non-trivial solution? \n- If $\\mathcal{C}$ is unbounded, then you can't really define a uniform distribution on it, how do you deal with that?\n- Can you handle improper priors?\n\nQUESTIONS:\nIn addition to those raised in the Limitations section:\n\n1. Do you have an intuition on how badly \"misspecified\" the learnt expert prior can get until learning for the online agent becomes very difficult? Is there a way to detect a poorly specified prior, and potentially reverting to an alternative (e.g. uninformative) after a few steps online?\n2. For the \"parametric\" ExPrior - why score gradients? Have you thought about or tried continuous relaxation when the problem is discrete?\n3. On line 112 \"Note that since the task variable is unobservable, the learner's policy will not depend on it\" - does it depend implicitly though, e.g. can the learner infer and integrate over that inferred task distribution?\n\nLIMITATIONS:\nThere are no specific negative societal impact that need to be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper attempt to leverage offline demonstrations to speed up online learning under unobserved heterogeneity, and unlike zero-shot meta reinforcement learning, the proposed ExPerior does not require the task labels (reward labels). ExPerior utilizes expert data to establish an informative prior distribution for online exploration. The experimental results on multi-armed bandits and MDPs showcase the superiority against current baselines.\n\nSTRENGTHS:\nThe setting proposed in this paper is realistic and significant. To solve the problem that the learner faces uncertainty and variability in task parameters, the paper proposes two approaches, parametric and non-parametric, and provides sufficient theoretical basis. Experiments under multi-armed bandits and MDPs also prove its effectiveness.\n\nWEAKNESSES:\n1.\tJust as stated at the end of the paper, the experiments conducted in the paper are limited, and the experiments under MDPs even made the assumption that the transition functions are invariant to the task variables, which makes the experimental environment overly simple. In more difficult experimental benchmarks such as MuJoCo, it is questionable whether the proposed method can show the expected effect. \n2.\tDue to the deficiency mentioned in the first point just now, the paper also cannot further compare with modern offline meta RL algorithms (such as PEARL, FOCAL, CORRO) under complex MDPs, making the baselines seem a bit scarce. \n3.\tThe introduction of related concepts in the paper is slightly insufficient. Besides, there is a clerical error in the upper right part of Fig. 1: the first \\mu_{\\rm{ME}} should be \\mu_{\\theta^*}.\n\nQUESTIONS:\n1.\tWhat does \"Heterogeneity\" mentioned in the paper represent? Does it refer to the differences between the transition functions and reward functions in MDPs? The paper should explain this.  \n2.\tThe paper mentions that the off-policy meta RL method PEARL requires task labels. Does the task label here refer to the reward label? In fact, the trajectories required by PEARL only contain (s, a, r, s') information.\n\nLIMITATIONS:\nA/N", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies online transfer learning in an unknown Markov decision process. The learner has access to demonstration trajectories generated by an expert, who has access to an independent context variable. The expert can observe the values of the latent context to make a decision, but such information is unobserved to the learner. The authors propose a learning strategy that allows the learner to accelerate its online learning process by leveraging the confounded expert's trajectories. More specifically, the expert selects values of the action following a specific parametric family of sofmax policy. Using this parametric information, the learner can extrapolate an informative prior about the underlying system dynamics (i.e., transition function and reward function) from the confounded trajectories and utilize the extrapolated prior to improve future learning. The authors then derive a Bayesian regret bound for the proposed method. Simulation results support the proposed approach.\n\nSTRENGTHS:\n- The paper is well-organized and clearly written. Simulations are comprehensive, supporting the proposed approach.\n- Theoretica regret analysis is provided. I have not read through all the proofs for Theorem 2. But the result seems reasonable.\n\nWEAKNESSES:\n- The proposed method requires the expert to follow a specific form of policy to generate the demonstration data. This parametric restriction is rather strong and may not necessarily hold in many applications.\n- It is unclear to see from Theorem 2 under which condition the expert's demonstration could accelerate the online learning process. It would be appreciated if the authors could further elaborate on this point.\n- Some related references are missing. Extrapolating knowledge from confounded data is one of the main problems in causal inference. There is a growing line of work studying learning from confounded demonstration data in canonical RL tasks, including:\n\n1. Kallus, Nathan, and Angela Zhou. \"Confounding-robust policy improvement.\" Advances in neural information processing systems 31 (2018).\n2. Zhang, Junzhe, and Elias Bareinboim. \"Near-optimal reinforcement learning in dynamic treatment regimes.\" Advances in Neural Information Processing Systems 32 (2019).\n\nQUESTIONS:\n- Could the regret bound in Theorem 2 outperform standard bandit regret when the expert's demonstration is benign? Could the authors elaborate on this?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}