Sequential Decision Making with Expert Demonstrations under Unobserved Heterogeneity
Authors: Vahid Balazadeh
Keertana Chidambaram
Viet Nguyen
Rahul Krishnan
Vasilis Syrgkanis
Venue: NeurIPS 2024 poster (2024)
Keywords: meta-reinforcement learning
unobserved heterogeneity
empirical Bayes
maximum entropy
posterior sampling
Primary area: reinforcement_learning

Abstract: We study the problem of online sequential decision-making given auxiliary demonstrations from _experts_ who made their decisions based on unobserved contextual information. These demonstrations can be viewed as solving related but slightly different tasks than what the learner faces.  This setting arises in many application domains, such as self-driving cars, healthcare, and finance, where expert demonstrations are made using contextual information, which is not recorded in the data available to the learning agent. We model the problem as a zero-shot meta-reinforcement learning setting with an unknown task distribution and a Bayesian regret minimization objective, where the unobserved tasks are encoded as parameters with an unknown prior. We propose the Experts-as-Priors algorithm (ExPerior), an empirical Bayes approach that utilizes expert data to establish an informative prior distribution over the learner's decision-making problem. This prior enables the application of any Bayesian approach for online decision-making, such as posterior sampling. We demonstrate that our strategy surpasses existing behaviour cloning and online algorithms, as well as online-offline baselines for multi-armed bandits, Markov decision processes (MDPs), and partially observable MDPs, showcasing the broad reach and utility of ExPerior in using expert demonstrations across different decision-making setups.

Sequential Decision Making with Expert Demonstrations under Unobserved Heterogeneity
Vahid Balazadeh University of Toronto vahid0es. toronto. edu
Keertana Chidambaram Stanford University vck@stanford edu
Viet Nguyen University of Toronto vietθcs toronto. edu
Rahul G. Krishnan University of Toronto rahulgk@cs. toronto.co edu
Vasilis Syrgkanis" Stanford University veyrgk@stanford. edu
Abstract
We study the problem of online sequential decision-making given auxiliary demonstrations from experts who made their decisions based on unobserved con- textual information. 'hese demonstrations can he viewed as solving related but slightly different problems than what the learner faces. This setting arises many application domains, such as self-driving cars, healthcare, and finance, where ex- pert demonstrations are made using contextual information, which is not recorded in the data available to the learning agent. We model the problem as zero-shot rentrreeinorre learning with an unknown distribution over the unobserved contextual variables and a Bayesian regret minimization objective, where the un- observed variables are encoded as parameters with an unknown prior. We pro- pose the Expertts-as-Proon algorithm (ExPerior), an empirical Bayes approach that utilizes expert data to establish an informative prior distribution over the learner's 's decision-making problem. This prior distribution enables the application any Bayesian approach for online decision-making. such as posterior sampling. We demonstrate that our strategy surpasses existing behaviour cloning, online, and online-offline baselines for multi-armed bandits, Markov decision processes (MDPs), and partially observable MDPs, showcasing the broad reach and utility of ExPerior using expert demonstrations across different decision decision-making maainng setups.
Introduction
Reinforcement learning (RI.) has found success complex decision-making tasks, spanning areas such game playing [1, 2, 3]. robotics [4, 5], and aligning with human preferences [6]. However, RL's considerable sample inefficiency, necessitating millions of training frames for convergence, remains a significant challenge. A notable body of work within RL has been dedicated integrating expert demonstrations to accelerate the learning process, employing strategies like offline pretraining [7] and the of combined offline-online datasets [8, 9]. While these approaches theoretically sound and empirically validated [10, 11], they typically presume homogeneity between the offline and online datasets. A vital question arises about the effectiveness of these methods when expert data embody heterogeneous tasks, indistinguishable by the learner.
An important example of such heterogeneity is in situations where experts operate with additional information not available to the learner, scenario previously explored in imitation learning with unobserved contexts [12, 13, 14, 15]. Existing literature either relies the availability of experts to query during training [16, 17, 18, 19] or focuses on the assumptions that enable imitation learn-
Equal contribution. Our code is accessible at htttps://gitnnn wwwpo///ww////eeii/eeeoooe
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Step 2: Learning Prior
Step Reinfortement
Parametric or Eatrape
Alpertthm PrienPestcrirr
Learning
e- Pare! bistory!
history)
Baymlan Agrer
Contectual Yartatin (Coos))
Step Expert
Update
Demonstrations
Guall
Easiranment
Geal 2
Siowry
[e, (())
Geal
E
Contectual Varlatiin (Gous))
Contectioa Variables (Goal))
Figure 1: Illustration of ExPerior in goal-oriented task. Step (Offline): The experts demonstrate their
policies for different goal types while observing them. Step (Offfine): The expert data Dg only contains the trajectories states/actions - goal types are not collected. We form an informative prior distribution over goal types (unobserved factors) using Dz. Step 3 (Online): The goal type is unknown but drawn from the same distribution of goals in Step 1. The learner uses the learned prior for posterior sampling.
ing with unobserved contexts, sidestepping online reward-based interactions [20, 21J. Recent con- tributions by Hao et al. [22, 23] suggest using offline expert data for online RL, albeit without accounting for unobserved variations. Our work addresses the more general challenge of online decision-making given auxiliary offline expert data with unobserved heterogeneity. We view such demonstrations solving related yet distinct problems from those faced by the learner, where dif- ferences remain invisible to the learner. For instance, a personalized education scenario, while a learning agent observe characteristies like grades or demographics, it might remain oblivious to factors such as learning styles, which are visible to an expert teacher and can significantly influence teaching methods. Naïve imitation without access this "private" information will only learn a sin- gle policy for each observed characteristic [24], leading to sub-optimal actions. On the other hand, purely online approach requires extensive trial and error to result in meaningful decisions.
We integrate offline expert data with online RI, treating the scenario as a zero-shot meta- reinforcement learning (meta-RL) problem with unknown distribution over unobserved con- textual variables. Unlike typical meta-RL frameworks where the learner is exposed to multi- ple instances during training (different students in our example) to learn the underlying distribu- tion [25, 26), our approach only leverages offline expert data to infer the distribution unobserved factors, embodying zero-shot meta-RL paradigm [27].
Contributions. We define Bayesian regret minimization objective and consider unobserved vari- ables as parameters under unknown prior distribution. We use empirical Bayes to derive an informative prior over the unobserved variables from expert data. We use the learned prior distribu- tion to drive exploration in the online RI. task, using approaches like posterior sampling [28]. We propose two procedures to learn such a prior: (1) a parametric approach that can utilize any existing knowledge about the parametric form of the prior distribution, and (2) nonparametric approach that employs the principle of maximum entropy when such prior knowledge does not exist. We call our framework Experts-as-Prioon or ExPerior for short. See Figure for goal-oriented RL example. ExPerior outperforms existing offline, online, and offline-online baselines in multi- Markov decision processes (MDPs), and partially observable MDPs. For multi-armed bandits, find the Bayesian regret incurred by ExPerior is proportional to the entropy of the optimal action under the prior distribution, aligning with the entropy of expert policy if the experts are optimal. We introduce a frequentist algorithm for multi-armed bandits and prove a Bayesian regret bound proportional term that closely resembles the entropy of the optimal action. Our results suggest using the entropy of expert demonstrations to evaluate the impact of unobserved factors.
bandits,
2 Related Work
Our work an addition to the recent body of reinforcement learning research that leverages of- fline demonstrations to speed up online learning [29, 10, 30, 7, 9]. Classic algorithms such as DDPGfD [31] and DQ1D [32] achieve this by combining imitation learning and RL. They modify DDPG [5] and DQN [1] by warm-starting the algorithms" replay buffers with expert trajectories and ensuring that the offline data never gets overridden by online trajectories. Closely related to our study is the meta-RI. RI. literature, which aims to accelerate learning in given RL task by using

---

prior experience from related tasks [33, 34, 35]. These papers present model-agnostee meta-learning training objectives to maximize the expected reward from novel tasks as efficiently as possible.
Two unique features distinguish our problem from the settings considered above. First, our setting assumes heterogeneity within the offline data and with the online RL task that is unobserved the learner, while the (optimal) experts are privy to that heterogeneity. Second, we assume the learner will only interact with one online task, making our setup similar to zero-shot meta-RL [27, 36, 37]. Most similar to our work is the ExPLORe algorithm [38], which assigns optimistic rewards to the offline data during the online interaction and runs an off-policy algorithm using both online and labelled offline data as buffers. For our setting, the algorithm incentivizes the learner to explore the expert trajectories, leading to faster convergence. We consider this work one of our baselines.
Our methodology utilizes only the state-action trajectory data from expert demonstrations without task-specific information reward lahels. Other similar methods require additional offline informa- tion. For example, Nair et al. [30] assume that the offline data contains the reward labels and use that to pre-train policy, which is then fine-tuned online. Mendonca al. [39] require task labelling for each trajectory and use the offline data to learn a single meta-leamner. Similarly, Zhou et al. [40] and Rakelly al. [41] require the task label and reward labels. They then infer the task during online interaction and use the task-specific offline data. Lee et al. [42] requires a large amount of noisy expert data with reward labels, in addition to the optimal trajectory data, for good performance. Finally, our methodology builds on posterior sampling [43]. Hao et al. [22, 23] consider a similar problem using posterior sampling to leverage offline expert demonstration data improve online RL. However, they assume homogeneity between the expert data and online tasks. In contrast, our setting accounts for heterogeneity.
3 Problem Setup
Decision Model for Unobserved Heterogeneity. To account for unobserved heterogeneity, consider a generalization of finite-horizon Markov Decision Processes (MDPs) with a notion of probabilistic contextual variables [44, 13, 21]. The underlying model for the MDP will additionally depend on an unobserved variable that encapsulates the information hidden from the learner. For example, consider a personalized education setup where teaching student corresponds to task, and the learning agent can observe students' characteristics, like their demographic status and grades. Other factors, such the student's learning style (e.g.. visual learners self-study), may not be readily available, even though they are important in determining the optimal teaching style.
Let C be the set of all unobserved context variables that can describe the unobserved heterogene- ity of the decision-making problem (e.g., the set of all possible learning styles). A contextual MDP M
(S. A, T. R. H. p. p,,x") is parameterized by states S, actions AA. transition function T : S x A x + A(S), reward function R x A x C + ARR). horizon H > O, initial state distribution p (SS). and context distribution μ²t We assume the transition/reward functions and '¹ are unknown, and for simplicity, P does not depend on the context variable. For each unob- served context e jt,, we consider T episodes, where at the beginning of each episode & E [T]. an initial state $1 ~ p is sampled. Then, at each timestep h E [A) the learner chooses an action ag E A, observes a reward Th R (sh. ab, c) and the next state Sh++1 T (Sh, ahe c). Without loss of generality, we assume the states are partitioned by [M] to make the notation invariant to the timestep. Let II be the set of all Markovian policies. For a policy function म : S + A(A) Il and
E==171 Th C.C and the Q-function as context variable c, we define the value function Ve (I)) E
Σ
s, an = a, n, e for all s E S,a E A. Moreover, we define the optimal policy for a context variable E Cs ne := arg max Ve (T). Note that since the context variable is unobserved, the learner's 's policy will not depend on it. The learning agent's goal is to leamn desende-de distributions p¹ ..... p" E (III) over Markovian policies to minimize the
Q5 (s. = E Σ A THe 8h
expected regret, defined as Reg == Eewey 1E Ve(me)
Extrps [V((+')]
In the personalized education example, the above setup assumes a fixed distribution μ over the set of learning styles and aims to minimize expected regret over the population of students. Our setup and regret assume the unobserved factors remain fixed during training. This captures scenarios wherein the unobserved variables correspond to less-variant factors (a student's 's learning style is more likely to remain unchanged). No learning algorithm can control the regret value if we allow the unobserved factors to change arbitrarily throughout T episodes without access to hidden information; consider
3

---

a two-armed bandit with context value drawn with uniform probability from C = {ci, C2 that can change at each episode. Assume the expected reward of the first arm under ei and Cg is one and zero, respectively, and it is reversed for the other arm. Any algorithm that does not have access to c would result in linear regret since each action is sub-optimal with a probability of 0.5, independent of the algorithm's 's choice.
Remark. Our setup can be formulated as a Bayesian model parameterized by C, and our regret can be seen as the Bayesian regret of the learner. However, the distribution Je is not the learner's prior belief about the true model as it is often formulated in Bayesian learning, but a distribution over potential contexts that the learner can encounter. Our setup can thus be seen as a meta-learning problem. In fact, is zero-shot meta-learning since we do not assume having access to more than one online RL task during training- - we only learn the prior distribution using the offline data.
Expert Demonstrations. In addition the online setting described above, we assume the learner has access to an offline dataset of expert demonstrations DE. where each demonstration TE = (s), 01 $2, .2:..., SH, ag, 5H+1 refers to an interaction of the expert with decision-making task during single episode, containing the actions made by the expert and the resulting states. We assume that the unobserved context variables for DE are drawn i.i.d. from distribution μ³, and the expert had access to such unobserved variables (private information) during their Docision-making Moreover, we assume the expert follows a near-optimal strategy [22, 23].
Assumption 1 (Noisily Rational Expert). For any c E C. experts select actions based on a dis- tribution defined as pe (a 8; c) x exp {33 QT (s, a)), ), for all E S,a E A, and some known competence value of / E co]. In particular, the expert follows the optimal policy if B co. oo.
We assume experts do not provide any rationale for their strategy, nor do we have access to rewards in the offline data; this is a combination of imitation and online learning rather than offline RL [22].
4 Experts-as-Prien Priors Framework for Unobserved Heterogeneity
Our goal is to leverage offline data to help guide the learner through its interaction with the decision- making task. The key idea is to use expert demonstrations to infer a prior distribution over C and then to use Bayesian approach such as posterior sampling [28] utilize the inferred prior for more informative exploration. If the current context is from the same distribution of contexts in the offline data, we expect that using such priors will lead to faster convergence to optimal trajectories compared to the commonly used non-informative priors. Consider the personalized education ex- ample. Suppose we have gathered offline data on an expert's teaching strategies for students with similar observed information like grade, age, location, etc. The teacher can observe more fine- grained information about the students that is generally absent from the collected data (e.g., their learning style). Our work relies on the following observation: The space of the optimal strategies for students with similar observed information but different learning styles is often much smaller than the space of all possible strategies. With the inferred prior distribution, the learner needs only to focus on the span of potentially optimal strategies for a new student, allowing for significantly more efficient exploration.
We resort to empirical Bayes and use maximum marginal likelihood estimation [45] to construct a prior distribution from DE. Given a probability distribution (prior) on C, the marginal likelihood of an expert demonstration TE= = (s1, 01, 82, 02 ... SH: SH+1) E Dg is given by
H
PE (TH: ()) = EeNp p(s))
II h=1
PE (an sw c) T (81+2 8h, Ch c)
(1)
We aim to find a prior distribution to maximize the log-likelihood of DE under the model described in (1). This is equivalent to minimizing the KL divergence between the marginal likelihood PE and the empirical distribution of expert demonstrations, which we denote by Pg. In particular, we form an uncertainty set over the set of plausible priors as P(e) =- {u; DKT. PF PR (: se). where the value of e can be chosen based on the number of samples so the uncertainty set contains the true prior with high probability [27]. However, the set of plausible priors does not uniquely identify the appropriate prior. In fact, even for = 0, P(c) can have infinite plausible priors. To solve this iol-posed problem, we propose two approaches, parametric and nonparametrie prior learning.
Parametric Experts-as-Prion
For settings where we have existing knowledge about the paramet- ric form of the prior, we can directly apply maximum marginal likelihood estimation to learn it. In

---

particular, we define the parametric expert prior as the following. Note that we can calculate the gradients the marginal likelihood using the score function estimator [46].
Definition 1 (Parametric Expert Prior). Let @ be a set of plausible prior distribution parameters (e.g., Beta distribution parameters for a Bernouli bandit). We call μig a parametric expert prior, iff 0* arg mingge LTED; log Pr. (7; vo))
Nonparametrie Experts-as-Priors. For settings where there is no existing knowledge on the para- metric form of the prior, can employ the principle of maximum entropy choose the least informative prior that is compatible with expert data:
Definition 2 (Max- Max--Etropy Expert Prior). Let do be non-informative prior on C (e.g., uniform distribution). Given some 0, we define the maximum entropy expert prior Mau as the solution to the following optimization problem:
JAME arg min DKL (p ll po) s.t. E P(cc)
Note that the set of plausible priors P(e) is convex set, and therefore, (2) is a convex optimization problem. We can derive the solution to problem (2) using Fenchel's duality theorem [47, 48]: Proposition 1 (Max Etat-Entropy Expert Prior). Let N [Del be the number of demonstrations in Dg. For each E C and demonstration TE = (5), a1, $2, 02. ......,$pp SH: ai $H+1) E Dr. define mz (c) as the (partial) likelihood of TE under c, i.e., (c) = II PE (an 8%;; c) T (sh+1 Sh, az. c).
Denote m(c) E R. as the vector with elements m.rg (c) for E Dr. Moreover, let X E R20 be the optimal solution the Lagrange dual problem of (2). Then, the solution to optimization (2) is:
exp (m(c)] ann
(ME(c)
lim
- 20
Eg
Ep (exp (mi(d'T am))]
where {an) J is sequence converging to the following supremum:
log Eespa [exp
ml()) a
X N
N Σ
-oa N 0z
sup N
log
A
The proof is provided in Appendix A.3. Instead of solving for X*, we set it as hyperparameter and then solve (3). Even though Proposition I requires the correct form of Q-functions for different values of c, we will see in the following sections that we can parameterize the Q-functions and treat those parameters as a proxy for the unobserved factors. Once such a prior is derived, we can employ any Bayesian approach for decision-making. We provide a pseudo-algorithm for ExPerior in Algorithm The following sections will detail the algorithm for bandits and MDPs.
Algorithm Experts-as-Prions Priors (ExPerior- MaxEnt)
1: Input: Expert demonstrations Dr. Reference distribution μoo A 2 0, and unknown μ=" Dz, X")
2: μMи MAXENTROPPYXPPERPPPoo
history + (
4: for episode + 1, 2,..... do
5: sample Ce ~ JIME (- history)
I| posterior sampling
6: for timestep h 1, 1,2 H do
7: take action oj ne. (: 8n)
ohserve rj R(sh: ag, c), s ~ T (shs ahs c), and append (ah, rhe sh+1) to history
9: end for
10:
end for
Learning in Bandits
K K-armed Bandits. For K-armed bandits, note that S = 0, H = 1, and A (1....,KY K}. Each expert demonstration π. a will be the pulled arm by the expert for a particular bandit, and the (partial) likelihood function in Proposition can be simplified as m., (c) = (a; c). This likeli- hood function only depends on the context variable e through the expert policy pr, and since pr only depends on through the mean reward function (Assumption 1), we can consider the set of mean reward functions as a proxy for the unobserved context variables C. egg. Bernoulli K K-ameed bandit setting, can define CBer
{a + (ea. 0) 0 E [0, 1]K].
5

---

Bayesian Regret
300
250-
Entropy
200-
Mid Entropy
Hiigh Enzropy
100-
G0-
Oncile TS
ExPetioe Puram
Omael TS (G4o)
ExPecioe Maxiat
NavveITS
Nalve-UCB
UCB EXPLORs
BC
Figure 2: The Bayesian regret of ExPerior and baselines for K-armed Bernoulli bandits (K = 10). We consider thrce categories of prior distributions based on the entropy of the optimal action.
Posterior Sampling. With the above parameterizations of C, we can use Proposition 1 to derive the maximum entropy prior distribution over the context parameters. However, we cannot sample from the exact posterior since the derived prior is not conjugate prior for standard likelihood functions. Instead, we resort approximate posterior sampling via stochastic gradient Langevin dynamics (SGLD) [49]. We call this method ExPerior-MaxEnt in our experiments. We also employ parametric approach as discussed in section 4, which we call ExPerior-Param, In particular, we use the Beta distribution as our prior model and learn the parametric expert prior in Definition 1.
In the following, we evaluate our approach compared online methods that do not use expert data and offline behaviour cloning. We provide an empirical regret analysis for ExPerior based on the informativeness of expert data, number of actions, and number of training episodes. We also discuss the robustness of ExPerior to misspecified expert models and the advantage of ExPerior-Pee -axEEtt to AePerior-Param when the parametric prior model misspecified. To characterize the effect of expert data on the learner's 's performance, we propose an alternative for K K-armed bandits inspired by the successive elimination and derive a Bayesian regret bound for it.
Experiments. We consider K-armed Bernoulli bandits for experimental setup. We evaluate the learning algorithms in terms of the Bayesian regret over multiple (prior) distributions jı" over the unobserved contexts. In particular, we consider up to N₁. 64 different beta distributions, where their parameters are chosen to span a different range of heterogeneity, consisting of tasks with various expert data informativeness. To estimate the Bayesian regret, we sample Nusk 128 bandit tasks from each prior distribution and calculate the average regret. We use Ng = 1000 expert demonstrations for each prior distribution in our experiments. We compare ExPerior the following baselines; (1) Behaviour cloning (BC), which learns policy by minimizing the cross-entropy loss between the expert demonstrations and the agent's policy solely based on offline data. (2) Naïve Thompson sampling WN) that chooses the action with the highest sampled mean from posterior distribution under an uninformative prior. (3) Naive upper confidence bound (NaIve-UCB) algorithm that selects the action with the highest upper confidence bound. Both Naive-TS and Naive-UCB ignore expert demonstrations. (4) UCB-EXPLORe, a variant of the algorithm proposed by Li et al. [38] tailored to bandits. It labels the expert data with optimistic rewards and then uses it alongside online data to compute the upper confidence bounds for exploration, and (5) Oracle-Ts, which performs exact Thompson sampling with the true prior distribution μ²" For a fair comparison, we also consider a variant of Oracle- -T, which uses SGLD for approximate posterior sampling.
Comparison to baselines. Figure 2 demonstrates the average Bayesian regret for various prior dis- tributions over T = 1,500 episodes with K 10 arms. To better understand the effect of expert data, we categorize the prior distributions by the entropy of their optimal actions into low entropy (less than 0.8), high entropy (greater than 1.6), and medium entropy. Oracle-TS and ExPerior-Paree outperform other baselines, yet the performance of ExPerior-Mazte variant of Dracle-Ts. This indicates that the maximum entropy prior derived from Proposition closely approximates the true prior distribution, μ², and the performance difference with Oracle-TS is primarily due to approximate posterior sampling. Moreover, the pure online algorithms Naive-TS and Naive-UCB, which disregard expert data, display similar performance across different entropy levels, contrasting with other algorithms that show significantly reduced regret in low-entropy con- texts. This underlines the impact of expert data in settings where the unobserved confounding has less effect on the optimal actions. Specifically, in the extreme case of no unobserved heterogeneity, BC is anticipated to yield optimal performance. Additionally, Naive-UCB surpasses NCB-EXPLORe in medium and high entropy settings, possibly due to the over-optimism of the reward labelling in Li al. [38], which can hurt the performance when the expert demonstrations are uninformative.
is comparable to the SGLD
Empirical regret analysis for siperttsas-PPiorss We examine how the quality of expert demon- strations affects the Bayesian regret achieved by ExPerior. Settings with highly informative demon-

---

ExPerioc Maxlin
ExPenee Parm
Orazle TS
Bayesian Regret
20-
25
0.0
0.5
1.0
Number of Arms, K Entropy of Optimal Action
Episodes,
Entropy of Optimal Action
(a)
(b)
Figure 3: (a) Empirical analysis of ExPerior's 's regret in Bernoulli bandits based on the (left) number of arms, (middle) entropy of the optimal action, and (right) number of episodes. (b) The regret bound from Theorem vs. the entropy of the optimal action. The linear relationship is consistent with the middle panel of (a).
Table 1: Ablation experiments to assess the robustness of ExPerior to misspecified expert models. Random- optimal experts choose the optimal action with probability and choose random actions with probability y. ExPerior-MaxEat achieves consistent out perforrrreeene by setting the hyperparameter 10. while ExPerior-Paran get almost similar results for
/ 2.5.
Optimal
Noisily- Rational
-oisily-Rational
Random-Opie Opttinal
B 0.1
8
2.5
8= 10
7-000
0.25
7- 0.5
7
0.75
ExPerior-MaxEat = 0.1) 5,.7
52.3
5.33
5.3
52.0
51.7
5233
52.1
52.0
51.8
ExPerior-Param (3 0.1)
11.1 4.3
33.1
12.6
11.7
338
10.9
4.
29
123
11.4
10.7
4.2
ExPerior-MaxEEnt (3 -
4.
52.2
51.6
50.0
47.
52.
510
49.1
+91
46.0
ExPerior-Param
9.1
2.3
1344
30
94431
228
98
86
89
ExPerior-MaxEot (3 2.5)
19
52.1
510
47.1
383
521
48.
448
40.5
ExPerior-Paraee (9 2.5)
8.5
24.3
29.0-211
+28
922-11
24.6 1.2 159 30 10,9
32
88
88+224
Ex MaxEnt (3 10)
34
52.0
47.6
4,4
29
26.7
52. 533 419
37.7
28
3.
ExPerioe Param (3 10)
11.2 4.8
26.9
1.2
25.0
+ 1.5
21
11.8333
26.8 + 1.1 23.2 - 188
20.1
2.
16.1
OmclleTS
8.5+
8.5
2,7
8.5
+
27
8.5
2.
8.4 8.4
85+
8.5
2.7
8.
Oracle (SGLD)
24.2
24.2
1.9
24.2
24.2
3,9
2.2 139
24.2
24.2
2492
strations, where unobserved factors minimally affect the optimal action, should exhibit near-zero regret since there no diversity in the unobserved contexts, and the experts are near-optimal. Con- versely, in scenarios where unobserved factors significantly influence the optimal actions, we antic- ipate the regret to align with standard online regret bounds, similar to the outcomes of Thompson sampling with a non-informative prior. We conduct trials with ExPerior and Oracle-TS across vari- ous numbers of arms over = 1,500 episodes, calculating the mean and standard error Bayesian regret across distinct prior distributions. As depicted in Figure 3 (a), both ExPerior and Oracle-Ts yield sub-linear regret relative to K and T. comparable to the established regret bound of O (KT) for Thompson sampling. However, the middle panel indicates that the regret of ExPerior is propor- tional to the entropy of the optimal action, having almost linear relationship. This observation seems to be contrast with the standard Bayesian regret bounds for Thompson sampling under correct prior that have shown a sublinear relationship of O (Ven((ze) where Ent(x.) denotes the entropy of the optimal action under j [50]. We analyze this observation in section 5.1.
Ablations. We also run additional experiments assess the robustness of ExPerior to misspecified experts. We create expert data from different experts with various competence levels, such as opti- mal, noisily rational, and random-optimal experts, where the latter chooses an action optimally with fixed probability and randomly otherwise. Table shows ExPerior's robustness to different expert models. Setting 3 10 for training PnPerior-MaxEEn and consistent out- out-performance among different expert types. Moreover, We evaluate the advantage of learning nonparametric max-entropy prior over misspecified parametric priors in Table 2. Even though SePerior-Paraa with Beta prior outperforms ExPerior-Maxdee ExPerior-MazEnt is superior to ExPerior-Param the prior mismatches the correct form (e.g., Gaussian or Gamma).
for ExPerior-Paran achieves
5.1 An Alternative Frequentist Approach for K-armed Bandits
To analyze the effect of expert data on the Bayesian regret, we devise an alternative frequentist approach, based on the successive elimination algorithm [51], which follows similar intuition to Experts-as-Priors. In particular, we prove a bound on its Bayesian regret and show that the derived bound proportional to a term that closely resembles the entropy of the optimal action, showing that the observation in the middle panel of Figure 3 (a) is consistent within different approaches.
7

---

Table 2: Superiority of ExPerior- compared to ExPerior-Paran with misspecified parametric prior.
ExPerior-Parae Paraio ExPerior-MaxEat MaxEat
Gamma Prior Beta-SGLD Prior Noral Prior
Oracle-TS
Oracle-TS (scLn)
Low Entropy
0.7 + 0.3
11.6= 1.3
23.3
60.2 + 6.3
546.5 # 153.4
0.9 0.4
11.0 1.6
Mid-Entropy
0.8
25.7 1.2
36.8 0.9
40.4 = 2.0
492.5 = 185.6
7.3 0.8
21.2
High-Entropy 24.
4.2
618:3.
45.6 = 2.0
46L8 104.8
21.222
39,9
Algorithm 2 Successive Elimination with Expert Sampling
1: Input: Episodes T, Arms A, expert policy PE. step size Pmin, unknown c ~ p², and 6 E (0, 1).
2: for t = 1... T do
3: try an active arm with relative frequency of P
I| all arms active 0.
Piiin
(t((a) is the number of times arm a is pulled by episode t and Va(a) is its empirical mean reward. 4: increment (a) and update (a).
5: construct UCB (a(++ log (4T*K/6)/2m.// and LCB; Vf - Vlog (47"K/8)/2n.(())
6: de-activate all arms a s.1. 3d with UCB. S LCB and normalize Pr.
7: end for
The idea of successive elimination is to identify suboptimal arms and deactivate them over time. In particular, it runs a uniform sampling policy among active arms and builds confidence intervals for each. then deactivates all the arms with an upper confidence bound smaller than at least one arm's lower confidence bound. We modify this algorithm using the policy derived from expert demonstrations instead uniform sampling policy. Recall that in K-armed bandits, each expert trajectory represents the pulled arm by the expert. Hence, the empirical distribution of expert demonstrations can be seen as a sampling policy over different arms. To simplify the analysis, we employ deterministic sampling approach by pulling each arm fixed number of times based on its probability. To do so, we discretize the expert policy with a step size Pmin, which leads to a relative frequency of (PP(a)/Pmin) for an arm a. In particular, we can choose pmin = min a; Pa(a)= PE(a). We provide the concrete algorithm in Algorithm 2 and prove the following Bayesian regret bound: Theorem 2. Consider a stochastic K-an -arredd bandit and let p be the empirical expert policy. Assume that (i) the mean reward function is bounded in [0, fo all arms, (ii) T Z minasptarya plaj' (iti) the expert is optimal, ie., Va E A: pla) = Pe (a; x²) and B + co, and (iv) the learner follows Algorithm 2. Then, with probability at least 1 - 8,
p(a) p(a) + p(a') (1- p(a) p(a) + [Vpla) Volar]. (4)
Reg S /Tlog (TK/8)
Σ a, aa
See Appendix A.4 for the proof. Two terms in (4) depend on expert data: (1) The relative standard deviation between any two pairs of arms and (2) a scaling factor that depends on the magnitude of probability that the arms are optimal. For homogeneous demonstrations, where the expert data only includes one unique pulled arm, the standard deviation (Term 1) is zero, resulting in zero regret. However, in extreme heterogeneity, where the empirical expert distribution uniform over the arms, we have Reg & K KVKTlog T. Finally, to assess the relationship between the regret bound and the entropy of the expert data, we fix K = 2, T 100, and plot the bound from (4) as a function of the entropy of the optimal action for various prior distributions. Figure 3 (b) demonstrates linear relationship, similar to the regret incurred by ExPerior in Figure (a). This observation opens up new directions to further analyze the regret for ExPerior and similar approaches in MDPs.
Learning in Markov Decision Processes (MDPs)
For MDPs, we need to parameterize both the mean reward and transition functions. However, we assume the transition functions are invariant the context variables to simplify our methodology and avoid extra modelling assumptions. Under this assumption, it is sufficient to parameterize the optimal Q-functions, e.g., using a deep Q-network (DQN) and treat those parameters as a proxy for the unobserved context variables, i.e., CMPP == {(s, a) + Q (s, a 0) 0 E @). where θ is the set of parameters for DQN. We can then derive a closed-form log-pdf of the posterior distribution under the maximum entropy prior. See Appendix A.5 for details. The derived posterior log-pdf can
2 Althoug this bound is worse than the standard successive elimination by factor of K, our empirical results show that ExPerior still on par with standard regrets in the non-informative cases.
8

---

Table 3: The average reward per episode in Prozen Lake (PODMP) after 90,000 training steps.
Fixed Hazurd
B
=.5
Hazard
Hazard
Hazard
Hazard
(OOMPP)
ExPerior-Mastee MaxEnt
-22.58 2: 1.17
6.00
1 0.00
3.58 = 0.89
162 1,85
11.47
2: 0.52
5.71 1: 0.67
6.00 2 0.00
6.00 = 0.00
ExPerion Param
0.69
4.31
5.27
0.00
12.00
0.37
1.41
5.42
0.40
4.31 1.80
Naive Boot-DQN
23.322
-23.32
0.69
-23.32
= 0.69
-23.32 = 0.69
-14.36
4: 5.88
-20.57
4 2.91
-20.39
= 1.75
-23.32
: 0.69
ExPLORe
1 0.00
6.00
+: 0.00
6,00
6.00
0.00
-30.68
12.40
-10.64
16.64
-13.00
6,00
Optimal 6.00 4 0.00
6,00
+ 0.00
6.00
0.00
6.00
0.00
12,00
0.37
6.53
+ 0.31
6.00
0.00
6.00 + 0.00
(MDP)
ExPerior-
-23.36
1.26
12.26
0.29
12.68
0.03
12.71 + 0.03
13.02
0.18
12.78
0.11
12.78 + 0.06
12.36 + 0.29
ExPerior-
-25.53:+2.35
12.64
0.08
12.70
0.03
12.68 + 0.03
13.00
0000000. 0.18
12.78
0.12
12.73
4: 0.07
12.64
0.08
Nalve Root-DQQ
-23.32
0.69
-23.32
0.69
-23.32
0.69
-23.32
0.69
-14.39
+ 5.22
-20999
+ 2,86
-20.39
+ 1.25
-23.32
+ 0.69
ExPLORe
11.74 1: 0.41
11.75
0.6
11.96
0.28
12.3
0.22
113.84
1 17:50
-5.89
13.75
100.00
7.60
11.75
0.63
Optimal
12.71 = 0.03
12.71 0.03
12.71
0.03
12.71 + 0.03
13,02
0.18
12.78
0.11
12.76
+ 0.06
12.64
0.03
then be used as the loss function for DQN Langevin Monte Carlo [52, 53] as the counterpart for Thompson sampling with SGLD. However, running Langevin dynamics can lead highly unstable policies due to the complexity of the optimization landscape in DQNs. Instead, we use a heuristic that combines the learned prior distribution with bootstrapped DQNs [54].
The original method of Bootstrapped DQNs utilizes an ensemble of L randomly initialized Q- networks. It samples Q-network uniformly at each episode and uses it to collect data. Then, each Q-network is trained using the temporal difference loss on parts of or possibly the entire collected data. This method and its subsequent iterations [55, 56, 57] achieve deep exploration by ensuring diversity among the learned Q-networks. To incorporate Bootstrapped DQN into the ExPerior framework and utilize the expert data, can formulate the ensemble
discrete prior distribution over the Q-networks. Let eens - - -m......., enss) be the parameter vector for an ensemble of Q-functions. We can define the ensemble prior, parameterized by Oens. as 140m.. (0) == Σt (0 eus = O) for any 0 E O Based on this prior model, we can learn the parametric expert prior using maximum marginal likelihood estimation, as formulated below.
,
Proposition 3 (Ensemble Marginal Likelihood). Consider contextual MDP M A, R, H. p. ).). Assume the transition function T does not depend on the context variables and Assumption holds. Then, the negative marginal log-likelihood of expert data De under the ensemble prior p is upper bounded by
L +E
{8. Q (0, a, (bmm)) -B. Q (s, Bens).
- log Ps: (De3 mm
log
Σ aa'AA
Exp
L
1= TEDS (a.a)Er
where 3 is the competence level of the expert in Assumption 1.
Proposition 3 is proved in Appendix A.6. We can then initialize the Q-networks in the Bootstrapped DQN method using ensemble parameters that minimize the above upper bound. We will refer to this method as ExPerior-Paree bound, we can match the discrete prior distribution Pe.... to the max-entropy prior by initializing the Q-functions in the ensemble with parameters sampled from the max- max-entropy expert prior. In particular, we can apply SGLD on the log-pdf of the max-entropy prior derived in Appendix A.5. We will refer to this approach as MxPerior-MaxEnt.
As an alternative approach, instead of minimizing the above upper
Experimental Setup. One challenge in RL is the reward sparsity, where the learner needs to ex- plore the environment deeply to observe rewards. Utilizing expert demonstrations can significantly improve the efficiency of exploration. Here, we focus on "Deep Sea," a sparse-reward tabular RL environment proposed by Osband et al. [56] to assess deep exploration for different RL methods. The environment an AM A grid, where the agent starts at the top-left corner of the map, and at each time step, it chooses an action from A = {left, right} to move to the left or right col- umn, while going down by one row. In the original version of Deep Sea, the goal is always on the bottom-rige rright corner of the map. We introduce unobserved contexts by defining distribution over the goal columns while keeping the goal row the same. We consider four types of goal distributions where the goal is situated at (1) the bottom-right corner of the grid, (2) uniformly at the bottom of columns, and (4) uniformly at the bottom of any of the A columns. We set M 30 and generate N = 1,000
any of the most 4
M columns, (3) uniformly the bottom of any of the right-most 2

---

Nave Boa-DQN
ExPerior-Max Eax (Ourd)
ExPerice-Paran (Orr))
ExPLOORe
1.0
Avernge Reward
0.5
0.0
0
1000
2000 0
1000
2000 0
1000 2000
1000
2000
Episodes.
Figure 4: The average reward per episode over 2,000 episodes in "Deep Sea." The goal is located at the right column, uniformly at the right-most quarter of the columns, uniformly at the right-most half, and uniformly at random over all the columns, respectively. ExPerior outperforms the basclines in all instances.
samples from the optimal policies offline expert demonstrations. To further evaluate ExPerior and showcase its applicability to partially- oabbeelly-bbervve MDP, we also consider the "Frozen Lake" envi- ronment, which requires the learner to navigate to goal while avoiding hazards [17]. The learner cannot observe the hazard location, while the expert has access to the whole map. Taking action, reaching the goal, and hitting the hazard incur rewards of -2, 20, and -100, respectively. The frozen lake map is 5 x 5, where the hazard (weak ice) is randomly located in the interior squares. We consider different settings with 2. 5, 7, and potential locations for the hazard. At the start of each episode, the hazard will be chosen randomly within the potential locations. We generate N = 1,000 samples from noisily rational experts with different competence levels for this environment.
Baselines. We compare ExPerior to the following: (1) ExPLORe, proposed by Li et al. [38] to accel- erate off off-po reinforcement learning using unlabeled prior data. In this method, the offline demon- strations are assigned optimistic reward labels generated using the online data with regular updates. This information is then combined with the buffer data to perform off- off-policy learning. (2) Naïve Boot-DQN, which is the original Bootstrapped DQN with randomly initialized Q-networks [54].
Deep Sea Results. Figure 4 demonstrates the average reward per episode achieved by the baselines for T 2,000 episodes. For each goal distribution, we run the baselines with 30 different seeds and take the average to estimate the expected reward. ExPerior outperforms the baselines in all instances. However, the gap between ExPerior and the fully online Naïve Boot-DQN, which measures the effect of using the expert data, decreases as we go from the low-entropy setting (upper left) to the high-entropy distribution over the contexts (bottom right). This is consistent with the empirical and theoretical results discussed in section 5 and confirms expectation that the expert demonstrations may not be helpful under strong unobserved confounding (strong heterogeneity). The ExPLORe baseline substantially underperforms, even compared to the fully online Naive Boot-DQN (except for the first distribution with zero-entropy). We suspect this is because EXPLORe uses actor-critic methods its backbone model, which are shown to struggle with deep exploration [58].
We run all the baselines for 90,000 steps with 30 different seeds. Table shows the average reward after 500 evaluation steps at the end of the training. ExPerior outperforms the baselines in almost all instances except for the case of 3 = 0.1, which corresponds to a nearly random expert. On the other hand, ExPLORe achieves near-optimal results for 8 = 0.1. We hypoth- esize that ExPLORe's 's performance is mainly due to the superiority of their base actor-critic model since it can achieve near-optimal performance even when the expert trajectories are low-quality.
Frozen Lake Results
7 Conclusion
We introduce the Experts-as- Priirrs (ExPerior) framework, a novel empirical Bayes approach to address the problem of sequential decision-making using expert demonstrations with unobserved heterogeneity. We ground our methodology in the maximum entropy principle to infer a prior dis- tribution from expert data that guides the learning process in different setttings, including bandits, Markov decision processes (MDPs), and partially-observed MDPs. Our experimental evaluations demonstrate that we can effectively leverage the expert demonstrations to enhance learning effi- ciency under unobserved confounding. In multi-armed bandits, we illustrated through empirical analysis that the Bayesian regret incurred by our method is proportional to the entropy of the opti- mal action, highlighting its capacity to adapt based on the informativeness of the expert data. Our tasks.
work offers practical framework readily applied to a broad spectrum of decision- One limitation of our work is the limited set of experiments, especially the lack of experiments with cuman--n-the-lowe Future directions include extending to more complex environments and further investigating the theoretical properties of our RI algorithm.
10

---

Acknowledgements
We would like to thank Benjamin Van Roy, Emma Brunskill, Florian Shkurti, Ramesh Johari, and Sanath Kumar Krishnamurthy for their valuable discussions and insights. We also acknowledge the use of GPT-4 in Figure 1 and for assistance in editing various sections of this manuscript. KC is supported by the William R. and Sara Hart Kimball Stanford Graduate Fellowship. VS is supported by NSF Award IIS-233791 supported by an NSERC Discovery Award RGPIN-2022-0455 and an NFRF Special Call NFRFR- 2022-00526. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.
RGK is supported by Canada CIFAR AI Chair. VBM and VN were
References
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Joannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 13255:13, 2013.
[2] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanc- tot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529
(7587):484-489, 2016.
[3] David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv:1712.018155 2017.
[4] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Ried- miller. Deterministic policy gradient algorithms. In International conference machine learning, pages 387-395. Pmlr, 2014.
[5] Timothy Lillicrap. Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 01.5509.9777 2015.
[6] Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[7] Andrew Wagenmaker and Aldo Pacchiano. Leveraging offline data in online reinforcement learning. In International Conference on Machine Learning, pages 2023.
PMLR,
[8] Yuda Song, Yifei Zhou, Ayush Sekhari, Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprint arXiv:2210.06711 2022.
[9] Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pages 1577 1594. PMLR, 2023.
[10] Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE inter- national conference on robotics and automation (ICRA), pages 6292 2292-6299. IEEE, 2018
[11] Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, et al. Scaling data-driven robotics with reward sketching and batch reinforcement learning. arXiv preprint arXiv: /2200, 2019.
[12] Richard A. Briesch, Pradeep K. Chintagunta, and Rosa L. Matzkin. Nonparametric Discrete Choice Models With Unobserved Heterogeneity. Journal of Business & Economic Statistics, 28(2):291-307, 2010. ISSN 0735-0015. Publisher: [American Statistical Association, Taylor & Francis, Ltd.].
[13] Lantao Yu, Tianhe Yu, Chelsea Finn, and Stefano Ermon. Meta-inverse reinforcement learning with probabilistic context variables. Advances in neural information processing systems, 32, 2019.
11

---

[14]
Nathan Kallus and Angela Zhou. Minimax-Optimal Policy Learning Under Unobserved Con- founding. Management Science, 67(5):2870-2585 2890, May 2021. ISSN 0025-1909, 1526-5501. doi: 10.1287/mmsc.2000..0 20200.c62
[15] Andrew Bennett, Nathan Kallus, Lihong Li, and Ali Mousavi. Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, pages 1999-2007.
PMLR, March 2021. ISSN: 2640-3498.
[16] Sanjiban Choudhury. Mohak Bhardwaj,, Sankalp Arora, Ashish Kapoor, Gireeja Ranade, Se- bastian Scherer, and Debadcepta Dey. Data-driven planning via imitation learning. The Inter- national Journal of Robotics Research, 37(13-14):1632-166 2018.
[17] Andrew Warrington, Jonathan W Lavington, Adam Scibior. Mark Schmidt, and Frank Wood. Robust asymmetric learning in pomdps. In International Conference on Machine Learning, pages 11013-11023, PMLR, 2021.
[18) Aaron Walsman, Muru Zhang, Sanjiban Choudhury, Dieter Fox, and Ali Farhadi. Impossibly good experts and how to follow them. In The Eleventh International Conference on Learning Representations, 2022.
[19] Idan Shenfeld, Zhang- Wei Hong, Aviv Tamar, and Pulkit Agrawal. TGRL: An algorithm for teacher guided reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pages 31077-31093, PMLR, 23-29 Jul 2023.
[20] Junzhe Zhang, Daniel Kumor, and Elias Bareinboim. Causal imitation learning with unob- served confounders. Advances in neural information processing systems, 33: 12622223-2 2020.
[21] Gokul Swamy, Sanjiban Choudhury, Bagneell and Steven Z. Wu. Sequence model imitation learning with unobserved contexts. Advances in Neural Information Processing Systems, 35: 17665-17676, 2022.
[22] Botao Hao, Rahul Jain, Tor Lattimore, Benjamin Van Roy, and Zheng Wen. Leveraging demonstrations to improve online learning: Quality matters. In International Conference on Machine Learning, pages 12527-12545. PMLR, 2023.
[23] Botao Hao, Rahul Jain, Dengwang Tang, and Zheng Wen. Bridging imitation and online reinforcement learning: An optimistic tale. arXiv preprint arXiv:2303.11306 2023.
[24] Luca Weihs, Unnat Jain, lou-Jen Liu, Jordi Salvador, Svetlana Lazebnik, Aniruddha Kemb- havi, and Alex Schwing. Bridging the imitation gap by adaptive insubordination. Advances in Neural Information Processing Systems, 34:19134-19146, 2021.
(25) Leonardo Cella, Alessandro Lazaric, and Massimiliano Pontil. Meta-learning with stochastic linear bandits. In International Conference on Machine Learning, pages 1360-1370. PMLR, 2020.
[26] Leonardo Cella and Massimiliano Pontil. Multi-task and meta-learning with sparse linear bandits. In Uncertainty in Artificial Intelligence, pages 1692-1702. PMLR, 2021.
[27] Jay Mardia, Jiantao Jiao, Ervin Tánczos, Robert D Nowak, and Tsachy Weissman. Concentra- tion inequalities for the empirical distribution of discrete distributions: beyond the method of types. Information and Inference: A Journal of the IMA, 9(4):113-850 2020.
[28] Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems, 26, 2013.
[29] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzami, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 2017.
[30] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint wxXiv:2006.09359 2020.
[31] Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint
arXiv: 7077 00777088877 2017.
12

---

[32]
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Hor- gan, John Quan, Andrew Sendonaris, Ian Osband, al. Deep q-learning from demonstrations. In Proceedings of the AAAI conference on artificial intelligence, 2018.
[33] Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. Advances in neural information processing systems, 31, 2018.
[34] Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. arXiv preprint arXiv: /803. 1347, 2018.
[35] Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson. A survey of meta-reinforceemett learning. arXiv preprint arXiv:2301. 0.8088, 2023.
[36] Vinay Kumar Verma, Dhanajit Brahma, and Piyush Rai. Meta-learning for generalized zero- shot learning. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 6062-6069, 2020.
[37] Julie Jiang, Kristina Lerman, and Emilio Ferrara. Zero-shot meta-learning for small-scale data from human subjects. 2023 IEEE 11th International Conference on Healthcare Informatics (ICHI), pages 311-320. IEEE, 2023.
[38] Qiyang Li, Jason Zhang, Dibya Ghosh, Amy Zhang, and Sergey Levine. Accelerating ex- ploration with unlabeled prior data. Advances in Neural Information Processing Systems, 36, 2024.
[39] Russell Mendonea, Abhishek Gupta, Rosen Kralev, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Guided meta-policy search. Advances in Neural Information Processing Systems, 32, 2019.
[40] Allan Zhou, Eric Jang, Daniel Kappler, Alex Herzog, Mohi Khansari, Paul Wohlhart, Yunfci Bai, Mrinal Kalakrishnan, Sergey Levine, and Chelsea Finn. Watch, try, learn: Meta-learning from demonstrations and reward. arXiv preprint 1906.0/900..3330
2019.
[41] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off- policy meta-reeiforcemmnt learning via probabilistic context variables. In International con- ference on machine learning, pages 5331-5340. PMLR, 2019.
[42] Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining leam in-context reinforcement learning. Ad- vances in Neural Information Processing Systems, 36, 2024.
[43] Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, Zheng Wen, al. tutorial on thompson sampling. Foundations and Trends@ in Machine Learning, 11(1):1-96, 2018.
[44] Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes.
arXiv preprint arXiv: /502 02259, 2015.
[45] Bradley P Carlin and Thomas A Louis. Empirical bayes: Past, present and future. Journal of the American Statistical Association, 124452))126-12988 2000.
[46] Michael C Fu. Chapter 19 gradient estimation. Simulation, 13:575-616, 2006.
[47] R Tyrrell Rockafellar. Convex analysis, volume 11. Princeton university press, 1997.
[48] Miroslav Dudík, Steven Phillips, and Robert E Schapire. Maximum entropy density esti-
mation with generalized regularization and an application species distribution modeling. Journal of Machine Learning Research, 2007.
[49] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference muchine learning (ICML-I1), pages 681-688, 2011.
[50] Daniel Russo and Benjamin Van Roy. An information--theorticc analysis of thompson sam- pling. The Journal of Machine Learning Research, 17(1):24422477) 2016.
[51] Eyal Even-Dar, Shie Mannor, and Yishay Mansour. Pac bounds for multi-armed bandit and markov decision processes. In Computational Learning Theory: 15th Annual Conference on Computational Learning Theory, COLT 2002 Sydney, Australia, July 8-10, 2002 Proceedings 15, pages 255-270. Springer, 2002.
13

---

[52] Vikranth Dwaracherla and Benjamin Van Roy. arXiv:2002.07882, 2020.
Langevin
dqn.
arXiv preprint
[53] Haque Ishfaq, Qingfeng Lan, Pan Xu, A. Rupam Mahmood, Doina Precup, Anima Anandku- mar, and Kamyar Azizzadenesheli. Provable and practical: Efficient exploration reinforce- ment learning via langevin monte carlo. In The Twelfth International Conference Learning Representations, 2024.
[54] lan Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems. 29, 2016.
[55] lan Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep rein-
forcement learning. Advances Neural Information Processing Systems, 31, 2018.
[56] Ian Osband, Benjamin Van Roy, Daniel J Russo, Zheng Wen, al. Deep exploration via randomized value functions. J. Mach. Learn. Res., 20(124):1-62, 2019.
[57] Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy. Approximate thompson sampling via epis- temic neural networks. In Uncertainty in Artificial Intelligence, pages 1586-1595. PMLR, 2023.
(58) Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Ka- trina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, et al. Behaviour suite for reinforcement learning. arXiv preprint arXiv:I908.033: 2019.
[59] Jonathan M Borwein and Qiji J Zhu. Techniques of variational analysis, 2004.
14

---

A Proofs
A.1 Notation
We assume is measurable set with an appropriate r-algebra and there exists probability measure Ao on C. We denote (P(C, po) as the space of all measurable functions C + R such that Ilfll (Sc d bounded measurable functions from C to R. Unless stated otherwise, we assume the probability measures are absolutely continuous w.r.t. μo, and their density functions are in L', We may abuse the notation and use the same symbol for a probability measure and its RakonnNikodym derivative w.r.t. Noo- Finally, we use E [-] to denote expectation under the probability measure μoo
1/p < oc. Moreover, we define L (C. (o) as the space of all essentially
A.2 Useful Lemmas
Here, we state and prove a set of results that will be useful for the rest of this section. The first one Fenchel's duality theorem:
Lemma 4 (Fenchel's Duality [59]). Let X and Y be Banach spaces, let f X R U {+oo} and g: g:Y } be convex functions and let A : X + Y be a bounded linear map. Define the primal and dual values p, d E [-o0, +o0] by the Fenchel problems
p = inf f(x) + g(Ax)
zEX
= sup - --' - 9-(5""
--EY-
where fr and 9" are the Fenchel conjugates of f and g defined as f*(x*)= = suP (x", x)) f(x) (similarly for g), X* the dual space of X and (,. its duality pairing, and A Y - X* the adjoint operator of A, i.e. yA"
(y" Ax). Suppose A dom(f) n cont(g) + 0, where dom( (f) : E X; (xx < jo and cont(g) are the continuous points of g. Then, strong duality holds, iee., p d.
Proof. See the proof Theorem 4.4.3 Borwein and Zhu [59].
We can use Fenchel's duality to solve generalized maximum entropy problems. In particular, we prove a generalization of Theorem in [48] for density functions in L' (o)
Lemma 5. For any function μ. E L (C,Ho), define the extended KL divergence as
(b) = DKL (n po) If f||l|11 =
100
Moreover, assume set of bounded feuture functions m1, m2,...,, M N: C - IR is given and denote
m as the vector of all N features. Consider the linear function Am T, (C, (C, po) + R. defined as
Vu E (C, po): Am(u) == (E [m1 [4] E [m2 p...... E [mN 14)
We define the generalized maximum entropy problem as the following:
inf
vb(1)) + (Am(H)),
(5)
(C.ma)
for an arbitrary closed proper convex function Ç TR.
Then the following holds:
The dual optimization of (5) is given by sup *
log [exp mm [a]] (-(())) (-a),
(6)
where 5* is the convex conjugate function of
2. Denote a², o²,.... as sequence in N converging to supremum (6), and define the fol- lowing Gibbs density functions
Gi (c) exp m() a)
E exx mm' ax)
Then,
inf ((u) + (Am(H)) = lim (( + (Am (C
n00
15

---

Proof. Part 1: We first derive the convex conjugate of sb. Note that (c,)) (c, Ho) with the pairing
vh E I pc, po), (h, p) = h(c) plc) dpo-
Hence, by Donsker and Varadhan's variational formula 4h (c) (5 (h)
sup (C,Ia)
(h, 11) - (1) log E (exp {1})
Moreover, the adjoint operator of Am is given by An -R (C R):
Va E RN CEC: C: Am (a) (c) = m (c) a.
(8)
Using (7) and (8) and Lemma concludes the proof.
Part Denote the primal and dual objective functions by
P(μ) (b))) [ (Am())):
D(x) == - log [exp (m T a}] - (-x),
and their optimal values as P and D For any E L Ho), note that
DKL (v |l (o) DKL (v (Gibbs)) = [ ulogu - (fe v voloo / log Hights duo)
(m(c) aa) v(c) pp log fexp (m aa]]
Am(v) log exp m a)l
(9)
Using (9), we can re-write the dual objective function as:
Va RM, ve (C, D(a)= -DKL (v || (Gibts) DKL (v l| mo) Am(u) Ta -(((v))))
(10)
Moreover, note that
A (v) (-(aa) Am
ssup (z, -a) -[(x))
S (m)
( Ta (A)m())) -a) - ((Am(u))
= <((m(l)))
(11)
Combining (10) and (11), we get
Va R.N . E (o) D(a) S -DKI. (v || HGBbss) DKI. (u (o) + ((Am(u))
= -KI. (v || HGitta)) + P(v).
(12)
Now, fix arbitrary j E
0, and consider sequence of u²t u2.... L (C, such that for all
P(μ)) - P 2)
(13)
We can re-write (13) using the fact p* D* lim D(a")):
VjEN: lim P( - D(a") < 27
(14)
In particular, by setting v غd in (12) and combining the result with (14), we get
an |Gbbbs
Vj ejeh N: lim DKL. n=
25
Hence, lim lim DKL. (NL (g = 0. From properties of the KL divergence, it follows that limy-ro P(w²) lim P a concluding the proof.
16

---

A.3 Max-Entropy Prior
Proposition Let N (Dgl be the number of demonstrations in Dg. For each C and demon- stration Tg ($1, d1, 82, d2,.... SH, QH, SH+1) E De. define (c) as the (partial) likelihood of Tp under c:
H II h=1
mep (c)
PE (ap 8h (sh+1 8h, c).
(15)
Denote m(c) E R.V as the vector with elements m. (c) for TE Dr. Moreover let X R be the optimal solution to the Lagrange dual problem of (2). Then, the solution to optimization (2) is as follows:
exp (m(c) Jnn Eewuo lexp {m(c) aml)'
(MM(c)
lim
where {an Jo is sequence converging the following supremum:
N ai X
sup
log Eecpo (exp { ((()) Tafl +
log
OERN
Proof. We first simplify the KL-divergence between the empirical distribution of the expert trajec- tories PE and the marginal likelihood PE (:: 1):
PE(
DKL (Pr Pe((ii
E PE( log
PE (T(1); 14)
ED
=- log N
Σ
log PE (7(1);
(PE(T())= = 7))
N
lo N
Σ
log
E m 14]
log p (
N
(9) EDE
By (1) and (15)
Using the above equality, re-write the definition of uncertainty set P(c) as
P(()
Σ
log E. (m, pd] log N
1
Σ
log (81) s $0
N
N
TED:
SIEDE
Therefore, we can re-write the optimization (2) as
μME arg min (() HET. (C.po)
1
Σ
log [m, 11] E --logg
1
Σ
log (s1) s 0,
N
rEDp
(16)
where the extended KL divergence ((x) is defined as:
(()) DEL (N (o) If |A]| 1, +
O.W.
Note that P(c) is convex set. To see this, consider μl, P2 E P(c). Then, for any 0 < A < 1, we have μ (1 A)yl Afz P(e) since E [m, -4] is linear μ and log convex. Moreover, is easy to see there exists strictly feasible solution for (16) (e.g., consider the true distribution Jr over C). Thus, strong duality holds, and we can form the Lagrangian function as
Σ - log [ma 14] - ((+ log N + Eogp(ss)
L(, = (())
N
TEDF
BIED
17

---

Given that X E R is the optimal solution to the Lagrange dual problem, the maximum entropy prior PME will be the solution
inf L(, (C.HO
inf (()) + X² (C.H)
Σ
log IE [m, u1)
u1) constant in μ.. (17)
N
TED
Now, for each x E R.V, define the convex function ((x) == A
Morcover, for E (C, po), define Am(u) = (E [m,0) (4], mm, 14] :.... E [M, wl). Then,
N
(E - log m.)
L(p, A*) = (b( + (Am(
(18)
Combining (17) and (18), the maximum entropy prior UME is the solution to
inf (5() + (Am(H)). (C,'(
Using Lemma and noting that
A
N 2t )
(²x²)
Σ i=1
-1 log
N
concludes the proof.
A.4 Regret Bound for K-armed Bandit
Theorem 2. Consider a stochastic K K-armed bandit and let be the empirical expert policy. Assume that (i) the mean reward function is bounded in [0, for all arms, (ii) T Z min plaj. (ii)) the expert is optimal, i.e., Ya A: p(a) Pg (a: n²) and B oc, and (iv) the learner follows Algorithm Then, with probability least 1 8,
Reg /Tlog (TK/5)
Σ
p(a)
(1
p(a)
[Vpla)+
p(a) + p(d')
p(a) pla')
a.a' aaatt
Proof. Fix 8 E 10 and C. Let & be the event that (a Vo(a)
log K))(4 2nela)
for all
arms E A. all t T, and all T' N. where (a) is the number of times that arm was pulled by time t. Note that since T Z each arm will be pulled at least once and (a 2 1.
We first show that IP (E) 2 Hoeffding's 's inequality, have
Fix T, arm a, and & < T. Suppose ni (a( =j for SJST. By
P (V((a)-va - vo(a) S log (4T*K/8)
1 2T
(19)
2j
Now, using the union bound over all episodes and all actions, we get
log (2T K/0)
3a A,T E N.TNN
Vi(a) Vo(a))
2j
20
log (2T' K//)
Vi(a) vv(a)l
2j
j=!
By (19)
274K
x 2
Σ
5
2T
2T
T=1
T=1
which concludes that P (8) - 5.
18

---

The rest of the proof computes the regret for when & holds. For simplicity and without loss of generality, we assume all expert probabilities are dividable by Pmin- Recall that we follow de- terministic sampling approach and choose each arm according to its relative frequency p(-) for multiple batches, where each batch loops over all active actions. Let la be the episode in which we eliminate in favour of another arm. Then, it is easy to show that
Paii
Va' active arms by to p(a') to S ne, (t, (a'),
(20)
This lower bound corresponds the case where no other arm is eliminated before eliminating a. Moreover, we have an upper bound for ne, (a) considering the worst-case scenario in which the only remaining arms are a and des where ag is the optimal action for unobserved context
nt. (a) S p(a)
ta.
(21)
p(a) ((a)
Now, let Reg,(a) (a) be the total regret contributed by the arm a for given context c ~ C. We can upper bound the regret
(eg,(a) The (a) (Vo(an) - -V((a)
(1)
log (4T*K/8)
log (4T*K/0)
x 2n (a)
(V
+
2n,,6 (a)
2ne (ac)
p(a)
log (/7)
log (4T*K/8)
$ p(a) + p(an)
ta
By (21)
211 (a)
2n (ac)
p(a) ha
1
22og (4T4K/0)
+
p(a) + p(ac)
(a)
new (ac)
p(a) p(a) + p(a.)
1 tap(a)
1
2log (4T*K/0)
to
By (20)
tapla.)
p(a)
2t (47*K/8)
p(a) p(ac)
V p(a)
+1 p(ac)
+
p(a)
1 p(ac)
2TTlo (/8)
p(a) + p(a))
p(a)
where (i) holds since the confidence intervals of arm and a overlap at episode ta (otherwise, would have been eliminated before f,,), and (ii) follows from the fact that ta I T. Finally, we upper bound the Bayesian regret by taking the expectation of Enta. Reg over C. Note that since the expert is optimal, we have p(a) μ²" (a. a) for all & E A.
(6)
Reg = Eewa.
Σ (a)
Σ μ¹ (ac = a')
max =o!
Σ (a)
d'EA
apda
Σ p(a')
Σ (ag
max
cia==a
a'EA
aza'
where (i) follows by partitioning into (e; e CEC, C, de = =a) and choosing the worst-case con- text in each partition.
19

---

From above, we have
Reg S
Σ p(a')
max Σ (a) ataa
e'EA
p(a))(a)
(V VD(a) p(a) *V p(af)
v 27 (47*K/8)
Σ
d'EA p(a) p(a)
(4))
8Tlog (4TK/6)
Σ
p(aa)p(a)
(Veto) 1 +
Volai
1
p(a) + +pld))
Vola)
a,a' E.Aapaaa
p(a')
p(a)
v 8T'lo (4TK/5)
Σ
p(a) + p(a') (Vo(a)+ vp(a")
p(a)+ p(a')
a,a' EAaapaa
where (1i) holds since 4K/6 > 1. Replacing pla)+p(a.)
pla)
with p(a') pla)+p(a.)
concludes the proof.
A.5 Max-Entropy Expert Posterior for MDPs
Proposition (Max-Entropy Expert Posterior for MDPs). Consider contextual MDP M (S.A, R, H, p. p²). Assume the transition function T does not depend on the context variables. Moreover, assume the reward distribution Gaussian with unit variance and Assumption 1 holds. Then, the log-pet log-pdf posterior function under the maximum entropy prior is given as:
1
Fh max Es (Q d'; 0)] Q (s/2 of 0)
v θ log UME (0 HT)
a'EA
h=1
exp (B Q (s, a; 0)}
Σ of rEDs
II
EdeE exp {3 Q (s, a'; 0)} + constant in 0
(s,a)ET
(22)
1 T where Hr = ={ oh rho the h+1 h=1) 1-1) is the history of online interactions, DE is the expert
demonstration data, B is the competence level of the expert Assumption and {ar, reDs are derived from Proposition 1.
Remark. We note that, in principle, the ExPerior framework allows for context transition functions. In this case, the log-pdf in (22) provides an optimistic upper bound on the true posterior log-pdf function. See Hao et al. [23] for a similar analysis. We leave the general case for future work. Note that second term of (22) simply the log-pdf of the max-entropy prior.
Proof. Since the transition function is context-independent, the likelihood of an expert trajectory тв can be simplified as:
H Yc EcEC: my (c) II PE (an Sh: c) II (sh+1 T
Sh. an)
(23)
The second term in (23) is constant in c. This implies that the likelihood function my (c) will depend e only through the expert policy, which itself is function of optimal Q-functions by Assumption 1. Note that the second term in the definition of m., can be simply removed since we re-weight the parameters a in the optimization step (3) of Proposition 1. Hence, assuming the deep Q-network is expressive enough, without loss of generality, we can re-define the likelihood function of an expert trajectory TE (81, 01, 82, 022 12...... 8H, as, 8H+1) as
H
exp [B. Q (Sh, ag 0))
vA E 3
my (0)
II
Σ
Ened exp (3 Q (8), a' 0))
hm
20

---

We can now write the log-pdf of the posterior distribution of 0 given Hr:
log JAME (0) Hr))
log P (Hr 0) + log HMM (0) + constant in O
logp (st) log R (rå s aj 0) + log T (sh++1
s/ a/) + log MME const.
L H
ΣΣ log R (rh si o 0) + log JME (0) + const.,
(24)
Now, given the Bellman equations, we can write the mean value of the reward function as
E S,a E A: [R a; 0)] = Q (s, a 0) - max Es [Q (s', a'; 0)]
a'eA
The reward distribution is Gaussian with unit variance. Therefore,
Vs E ArER: R(r 8, 0) - (Q(s,a;
0) max Es (Q (8', a'; 01. (25)
d'EA
Moreover, by Proposition the log-pdf of the maximum entropy expert prior is given as
exp (3 Q (8, a; 0)}
vO O /MM/ JMM (
Σ
o m- (0())=
Σ Of II
Eared exp (3 Q (8, a'; 0)}
rEDn
(s.a)Er
(26)
Combining (24) to (26), we conclude the proof.
A.6 Ensemble Marginal Likelihood
Proposition 3. Consider contextual MDP M = (S. H.P.p.)) Assume the transition function T does not depend on the context variables and Assumption I holds. Then, the negative marginal log-likee nigllikel of expert data DE under the ensemble prior 140... is upper bounded by
L 3 log Σ exp ( 0mmm)) -B.
loog PE (De; (0 1
Q
B- Q (=..a; 0ans)
(E
aa'EA
where B is the competence level of the expert in Assumption 1.
Proof. Recalling (1), the log-likelihood of the expert trajectories DE under 140 is given by
Fl
-log PE (De; po
Σ
llog Ee~poenn
p(s
pF
a
S).
8
h=
Σ r(EE
log Eo Beea
II
Pe
s/ 0)
+ constant in 0cus
(p. T do not depend on 0)
H
Σ
log
Σ
II
PE
(By Definition of mm
gD
1=1
A-1
1.
H
<
Σ
log pe
(o/))
By Jensen's 'ss inequality
(9EDE
j=1
h=1
log Σ exp 3. Q (s.a'' a.: 0cms) () -3
Σ
Σ
3 Q a; Oens)
T,
(s.0)Er
No'EA
By Assumption
21

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
To the best of our knowledge theoretically and empirically validate our claims.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our work in the concluding section of the manuscript.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: provide proofs for the thcoretical results in this work.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yss]
Justification:
The experiments reproducible by virtue of the code provided. Our code is available at https: / /ig thub coom/vblmmeee heablm/expeiorrrr
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Ye]]
Justification: We release code for this work. Our code is available at https: /ttps://githult com/vablnnexperrir
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We release code reproduce all the results which provides sufficient detail to answer the above questions. Our code is available at https: //gi tiub..b.... com/vdblm/ experior
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer:
[Ye]]
Justification: The paper reports emror bars and confidence intervals.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
22

---

Answer: [Yes]
Justification: We have used 110 GPU-hours for all the experiments on Quadro RTX 6000.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / ceups/sccccoooeeiiiicicccceeeeee
Answer: [Yes]
Justification: We have reviewed the NeurIPS code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our algorithms evaluated on synthetic and simulated RL-based environ- ments and we do not anticipate any direct or indirect potential positive or negative societal impacts.
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We do not anticipate misuse from the release our models due to the nature of the evaluations conduct synthetic and RL-based datasets.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the relevant papers in cases where we needed to reproduce their code for comparison against our work.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification: We provide software code to reproduce our results.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We do not study our methodology using data from human subjects.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We do not use data that requires IRB approval to study our methodology.
23


Reviews:
[{"review_id": "UrOE3v5BDk", "reviewer": "Reviewer_11BA", "summary": "This paper studies online transfer learning in an unknown Markov decision process. The learner has access to demonstration trajectories generated by an expert, who has access to an independent context variable. The expert can observe the values of the latent context to make a decision, but such information is unobserved to the learner. The authors propose a learning strategy that allows the learner to accelerate its online learning process by leveraging the confounded expert's trajectories. More specifically, the expert selects values of the action following a specific parametric family of sofmax policy. Using this parametric information, the learner can extrapolate an informative prior about the underlying system dynamics (i.e., transition function and reward function) from the confounded trajectories and utilize the extrapolated prior to improve future learning. The authors then derive a Bayesian regret bound for the proposed method. Simulation results support the proposed approach.", "questions": "- Could the regret bound in Theorem 2 outperform standard bandit regret when the expert's demonstration is benign? Could the authors elaborate on this?", "limitations": "The authors have adequately addressed the limitations of the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-organized and clearly written. Simulations are comprehensive, supporting the proposed approach.\n- Theoretica regret analysis is provided. I have not read through all the proofs for Theorem 2. But the result seems reasonable.", "weaknesses": "- The proposed method requires the expert to follow a specific form of policy to generate the demonstration data. This parametric restriction is rather strong and may not necessarily hold in many applications.\n- It is unclear to see from Theorem 2 under which condition the expert's demonstration could accelerate the online learning process. It would be appreciated if the authors could further elaborate on this point.\n- Some related references are missing. Extrapolating knowledge from confounded data is one of the main problems in causal inference. There is a growing line of work studying learning from confounded demonstration data in canonical RL tasks, including:\n\n1. Kallus, Nathan, and Angela Zhou. \"Confounding-robust policy improvement.\" Advances in neural information processing systems 31 (2018).\n2. Zhang, Junzhe, and Elias Bareinboim. \"Near-optimal reinforcement learning in dynamic treatment regimes.\" Advances in Neural Information Processing Systems 32 (2019).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your helpful comments. We are pleased that you find the paper clearly written, well-organized, and with comprehensive simulations. We address your questions in the following.\n\n> The proposed method requires the expert to follow a specific form of policy to generate the demonstration data. This parametric restriction is rather strong and may not necessarily hold in many applications.\n\nWe agree with you that Assumption 1, about the expert policy being noisily rational, may not hold in practice. Appendix C.1, Table 2 on Page 22 considers misspecified expert models. In particular, we looked into “optimal” experts, “noisily rational” experts with misspecified values of $\\beta$, and a new type of experts we called “random-optimal,” who take completely random actions with some fixed probability and optimal actions otherwise. In all instances, our methods, which rely on the noisily rational experts, achieve near-optimal results. This observation justifies the use of Assumption 1, even in instances where it does not necessarily hold.  We would appreciate it if the reviewer has any suggestions for including other types of misspecification in Table 2.\n\n> It is unclear to see from Theorem 2 under which condition the expert's demonstration could accelerate the online learning process. It would be appreciated if the authors could further elaborate on this point.\n\nAs discussed in lines 302-312, as well as Figure 3 (b), the extra term in the regret bound of Theorem 2, i.e.,\n$$\\sum\\_{a, a’ \\in \\mathcal{A}, a \\neq a’} \\sqrt{\\frac{p(a)}{p(a) + p(a’)} \\left(1 - \\frac{p(a)}{p(a) + p(a’)}\\right)} \\left[\\sqrt{p(a)} + \\sqrt{p(a’)}\\right]$$\nclosely resembles the variance (or the entropy in the case of $K = 2$) of the optimal actions under the distribution of unobserved factors. In other words, **if the variance of the optimal actions is small, i.e., the effect of unobserved factors is negligible, the regret will be close to zero**. We will elaborate more on this point in the revised version of the paper.\n\n> Some related references are missing. Extrapolating knowledge from confounded data is one of the main problems in causal inference. There is a growing line of work studying learning from confounded demonstration data in canonical RL tasks ...\n\nThank you for providing the missing related references. We will include those papers in the revised manuscript. We emphasize that the first work, Kallus et al. (2018), only considers the offline setting, while the latter assumes the availability of rewards in the offline observational data and does not assume the demonstrations are generated by experts. Our work, on the other hand, integrates **offline expert** demonstration data with **online** reinforcement learning.\n\n> Q: Could the regret bound in Theorem 2 outperform standard bandit regret when the expert's demonstration is benign? Could the authors elaborate on this?\n\nYes, as discussed above and in lines 302-312 of the manuscript, the standard bandit regret is bounded by $\\tilde{\\mathcal{O}}\\left(KT\\right)$ for $K$ number of actions and $T$ episodes. The bound in Theorem 2, however, is\n$$\\tilde{\\mathcal{O}}\\left(\\sqrt{T} \\cdot \\underbrace{\\sum\\_{a, a’ \\in \\mathcal{A}, a \\neq a’} \\sqrt{\\frac{p(a)}{p(a) + p(a’)} \\left(1 - \\frac{p(a)}{p(a) + p(a’)}\\right)} \\left[\\sqrt{p(a)} + \\sqrt{p(a’)}\\right]}\\_{\\text{term} I}\\right)$$\nWhen the expert’s demonstration is benign, e.g., the variance of the optimal actions under unobserved confounding is small, the second term (term I) will be smaller than $\\sqrt{K}$. As an example, consider a $K$-armed case, where a given fixed arm $a_1$ is optimal regardless of the unobserved heterogeneity. Then, for all $a \\neq a_1$, we have $p(a) = 0$, resulting in term I to be zero. This is a case when the expert data is extremely informative and hence gives us zero regret.\n\nWe hope our comment addresses your questions. Please let us know if we can answer any further questions that might improve your assessment of the work.\n\n*1. The big-O notation $\\tilde{\\mathcal{O}}$ in this comment ignores the log terms.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6STpYZxc2b", "reviewer": "Reviewer_hfW9", "summary": "This paper attempt to leverage offline demonstrations to speed up online learning under unobserved heterogeneity, and unlike zero-shot meta reinforcement learning, the proposed ExPerior does not require the task labels (reward labels). ExPerior utilizes expert data to establish an informative prior distribution for online exploration. The experimental results on multi-armed bandits and MDPs showcase the superiority against current baselines.", "questions": "1.\tWhat does \"Heterogeneity\" mentioned in the paper represent? Does it refer to the differences between the transition functions and reward functions in MDPs? The paper should explain this.  \n2.\tThe paper mentions that the off-policy meta RL method PEARL requires task labels. Does the task label here refer to the reward label? In fact, the trajectories required by PEARL only contain (s, a, r, s') information.", "limitations": "A/N", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The setting proposed in this paper is realistic and significant. To solve the problem that the learner faces uncertainty and variability in task parameters, the paper proposes two approaches, parametric and non-parametric, and provides sufficient theoretical basis. Experiments under multi-armed bandits and MDPs also prove its effectiveness.", "weaknesses": "1.\tJust as stated at the end of the paper, the experiments conducted in the paper are limited, and the experiments under MDPs even made the assumption that the transition functions are invariant to the task variables, which makes the experimental environment overly simple. In more difficult experimental benchmarks such as MuJoCo, it is questionable whether the proposed method can show the expected effect. \n2.\tDue to the deficiency mentioned in the first point just now, the paper also cannot further compare with modern offline meta RL algorithms (such as PEARL, FOCAL, CORRO) under complex MDPs, making the baselines seem a bit scarce. \n3.\tThe introduction of related concepts in the paper is slightly insufficient. Besides, there is a clerical error in the upper right part of Fig. 1: the first \\mu_{\\rm{ME}} should be \\mu_{\\theta^*}.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We are encouraged that you find the paper to have a realistic and significant setting and sufficient theoretical and experimental results. We address your questions below.\n\n\n> Just as stated at the end of the paper, the experiments conducted in the paper are limited, and the experiments under MDPs even made the assumption that the transition functions are invariant to the task variables, which makes the experimental environment overly simple. In more difficult experimental benchmarks such as MuJoCo, it is questionable whether the proposed method can show the expected effect.\n\nWe agree that the current experiments do not consider complex high-dimensional settings. Please see the General Response we provided for our detailed plan to scale our approach and include such experiments in the revised version.\n\n> Due to the deficiency mentioned in the first point just now, the paper also cannot further compare with modern offline meta RL algorithms (such as PEARL, FOCAL, CORRO) under complex MDPs, making the baselines seem a bit scarce.\n\nThank you for mentioning the works FOCAL [1] and CORRO [2]. We will cite these in the related work section of the revised paper. However, independent of the complexity of the MDPs, the mentioned baselines are not immediately applicable for our problem setup. PEARL, FOCAL, and CORRO all require reward signals to be available in the offline data. Moreover, FOCAL and CORRO assume the offline data is diverse enough to learn the optimal policies solely based on the offline data and do not continue the learning during the online phase. On the other hand, our setting only assumes offline expert demonstration data (with no reward) and requires using the reward signals during the online phase. This is why we did not include those baselines in our experiments, rather than the complexity of the MDPs.\n\n> The introduction of related concepts in the paper is slightly insufficient. Besides, there is a clerical error in the upper right part of Fig. 1: the first \\mu_{\\rm{ME}} should be \\mu_{\\theta^*}.\n\nCould you please elaborate on which parts of the introduced concepts are slightly insufficient so we can explain them in more detail? Also, thank you for finding the typo in $\\mu_{\\rm{ME}}$. We will fix it in the updated version.\n\n> Q: What does \"Heterogeneity\" mentioned in the paper represent? Does it refer to the differences between the transition functions and reward functions in MDPs? The paper should explain this.\n\nYes, your understanding is accurate. We will elaborate more on what we mean by \"Heterogeneity\" in the introduction.\n\n> Q: The paper mentions that the off-policy meta RL method PEARL requires task labels. Does the task label here refer to the reward label? In fact, the trajectories required by PEARL only contain (s, a, r, s') information.\n\nThank you for pointing this out. This confusion is probably due to the paper's phrasing. Quoting the paper (lines 85-86): \"Similarly, Zhou et al. [40] and Rakelly et al. [41] require the task label and reward labels.\". This should have instead been, \"Similarly, Zhou et al. [40] and Rakelly et al. [41] require the task label and reward labels, *respectively*.\" PEARL requires offline reward information, which is not accessible in our setting. We will fix this confusion in the revised version.\n\nWe hope our response addressed your comments. Should you have any additional questions that could assist in improving your evaluation of the work, please feel free to let us know.\n\n===================================\n\n[1] Li, Lanqing, Rui Yang, and Dijun Luo. \"Focal: Efficient fully-offline meta-reinforcement learning via distance metric learning and behavior regularization.\" arXiv preprint arXiv:2010.01112 (2020).\n\n[2] Yuan, Haoqi, and Zongqing Lu. \"Robust task representations for offline meta-reinforcement learning via contrastive learning.\" International Conference on Machine Learning. PMLR, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pzr640sXKi", "reviewer": "Reviewer_2ZF5", "summary": "The paper proposes a 2-stage learning method for sequential decision making, wherein step 1 involves leaning a prior from expert demonstrations and step 2 is online RL where that learnt prior is used. The main problem being addressed is that of heterogeneity of data and/or contexts provided by the expert and those encountered by the online learning agent.", "questions": "In addition to those raised in the Limitations section:\n\n1. Do you have an intuition on how badly \"misspecified\" the learnt expert prior can get until learning for the online agent becomes very difficult? Is there a way to detect a poorly specified prior, and potentially reverting to an alternative (e.g. uninformative) after a few steps online?\n2. For the \"parametric\" ExPrior - why score gradients? Have you thought about or tried continuous relaxation when the problem is discrete?\n3. On line 112 \"Note that since the task variable is unobservable, the learner's policy will not depend on it\" - does it depend implicitly though, e.g. can the learner infer and integrate over that inferred task distribution?", "limitations": "There are no specific negative societal impact that need to be discussed.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Relevant problem: Finding useful, scalable and efficient ways to leverage offline expert demonstrations for online decision making is of key interest to the community. \n\n- The proposed method combines imitation learning with online model-based RL, paying attention to the issue of heterogeneity between the available contexts that gave rise to the offline data, that might be unavailable during the online RL step.\n\n- The problem is well-motivated and the approach is easy to understand conceptually; contributions are clearly outlined. \n\n- Presentation (writing, figures etc) is mostly clear, with some minor comments included in the below section.", "weaknesses": "Clarity\n- Notation: even with the notation paragraph, I find it quite difficult to follow. E.g. I think some `t` superscripts are missing (the definition of the q-function); should $\\mathcal{C}$ be part of the definition of the MDP?\n- Terminology: I'm a little confused with the terminology \"parametric\" vs \"non-parametric\". Unless I'm missing something, the latter is still parametric as you are optimising over a family of distributions?\n- Typo: in figure 1, annotation below \"Step 3\" is repeated, I guess one should be $\\mu_{\\theta}*$.\n- Figure 1: clarity can be improved if you write down what is being learnt and what is being updated.\n- I found the algorithm box quite useful, will be great to have it in the main paper (e.g. next to Figure 1).\n\nMax-entropy prior:\n- Clearly, if $\\mu_0  \\in \\mathcal{P}(\\epsilon)$, then we get a trivial solution, $\\mu_{ME}=\\mu_0$ (usually Uniform), so do you need an assumption on ${P}(\\epsilon)$ to get a non-trivial solution? \n- If $\\mathcal{C}$ is unbounded, then you can't really define a uniform distribution on it, how do you deal with that?\n- Can you handle improper priors?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. We are pleased that you consider the paper well-motivated, relevant, and with a conceptually easy-to-understand approach. We have addressed your questions below.\n\n> Notation: even with the notation paragraph, I find it quite difficult to follow...?\n\nWe assume the states are partitioned by the horizon $[H]$, i.e., each state $s \\in \\mathcal{S}$ is only reachable in one specific horizon $h \\in [H]$. This can be achieved by re-defining the state space as $\\mathcal{S} \\times H$. This way, none of the value functions or Q-functions need to depend on the horizon. The superscript $t$, however, corresponds to the episode number and not the horizon. The value/Q-functions are defined independently of the episode. The only variable that gets updated over episodes is the learned policy $\\pi^t$. Also, we agree that $\\mathcal{C}$ should be part of the definition of the MDP. We will include these comments in the notation page of the revised manuscript, as there will be more space (10 pages instead of 9).\n\n> Terminology: I'm a little confused with the terminology \"parametric\" vs \"non-parametric\"...\n\nLooking at the max-entropy expert prior in Proposition 1 (line 189), you can see that.\n$$\\mu\\_{\\text{ME}}(c) = \\lim\\_{k \\to \\infty} \\frac{\\exp \\left( \\mathbf{m}(c)^\\top \\boldsymbol{\\alpha}_{k} \\right)}{\\mathbb{E}\\_{c'\\sim \\mu\\_{0}} \\left[ \\exp \\left( \\mathbf{m}(c')^\\top \\boldsymbol{\\alpha}\\_{k} \\right) \\right]}$$\nwhere each $\\boldsymbol{\\alpha}\\_k$ is an $N$-dimensional parameter with $N$ being the number of demonstrations. In other words, the length of parameters required to model the max-entropy prior increases by the number of samples. This is what we mean by a \"non-parametric\" approach, in contrast to \"parametric\" models with a fixed set of learnable parameters. Moreover, we do not assume any specific form for the family of distributions in $\\mathcal{P}(\\epsilon)$, except belonging to $L^1(\\mathcal{C}, \\mu\\_0)$ defined in lines 577-583.\n\n> Typo: in figure 1, annotation below \"Step 3\" is repeated, I guess one should be \\mu_{\\theta^\\star}.\n\nThis is a mistake, and we will fix it in the revised version. Thank you for pointing this out.\n\n> Figure 1: clarity can be improved if you write down what is being learnt and what is being updated.\n\n$\\mu\\_{\\theta^\\star}$, $\\mu\\_{\\text{ME}}$ are learned from data. The posterior distributions  $\\mu\\_{\\theta^\\star}(\\cdot | \\text{history})$, $\\mu\\_{\\text{ME}}(\\cdot | \\text{history})$ are being updated, and the policy $\\pi\\_c(\\cdot | s)$ is learned/calculated given the sampled Q-functions. We will differentiate between those with different colours in Figure 1.\n\n> I found the algorithm box quite useful, will be great to have it in the main paper (e.g. next to Figure 1).\n\nWe moved algorithm boxes 1 and 2 to the appendix solely for the lack of space. If the paper gets accepted, we will include them in the main paper. (The camera-ready version allows for one more page.)\n\n\n> Clearly, if $\\mu\\_0 \\in \\mathcal{P}(\\epsilon)$, then we get a trivial solution, $\\mu\\_{\\text{ME}} = \\mu\\_0$ (usually Uniform) ...?\n\nYour assessment is correct. If the uniform distribution is already in the set of feasible prior functions $\\mathcal{P}(\\epsilon)$, then  $\\mu\\_{\\text{ME}} = \\mu\\_0$. However, this is not a trivial solution since the uniform distribution will maximize the marginal likelihood of the data and is a plausible distribution. This scenario can happen if the effect of unobserved confounding is extreme, i.e., the unobserved factors can change the optimal policies arbitrarily and provide no information on the task distribution. For example, consider a multi-armed bandit setting with $K$ actions, where there are $K$ task types, each with a different arm as the optimal action. The expert demonstration data will contain each of the arms with equal probability. In other words, the expert data provides no information, and the best thing we can do is to have a non-informative prior like the uniform prior distribution.\n\n\n> If $\\mathcal{C}$ is unbounded, then you can't really define a uniform distribution on it ...? Can you handle improper priors?\n\nNote that our result does not force $\\mu\\_0$ to be a uniform distribution or $\\mathcal{C}$ to be bounded. It only requires $\\mathcal{C}$ to be a measurable set with an existing measure $\\mu\\_0$. In the case of unbounded $\\mathcal{C}$, one can choose $\\mu\\_0$ to be a Gaussian distribution. However, our current results, like Lemma 5 on page 14, require the priors to be in $L^1\\left(\\mathcal{C}, \\mu\\_0\\right)$, i.e., $\\mathcal{C}$ should have a finite measure under the feasible priors. For that reason, the current theoretical results do not handle improper priors.\n\n> Q. Do you have an intuition on how badly \"misspecified\" the learnt expert prior can get ...?\n\nThis is the exact reason we have provided the non-parametric max-entropy approach. Table 3 (Page 22) shows that misspecified priors result in large regrets. In cases where the practitioner is not quite sure about the parametric form of the prior, we suggest using the non-parametric max-entropy approach.\n\n> Q. For the \"parametric\" ExPrior - why score gradients? Have you thought about or tried continuous relaxation when the problem is discrete?\n\nIn the parametric ExPerior, we do not assume the parameter set $\\boldsymbol{\\Theta}$ is necessarily continuous, as long as we can solve the optimization in line 175 (potentially without gradient-based methods).\n\n> Q. On line 112 \"Note that since the task variable is unobservable ...\n\nWe meant explicit dependence here. The learner policy only depends on the history of interactions and the current state. As you mentioned, one can use the history to infer/integrate the task distribution. We will clarify more on this.\n\nWe hope our comment addresses your questions. Please let us know if we can answer any further questions that might improve your assessment of the work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o1mKxw7Xtz", "reviewer": "Reviewer_FGqG", "summary": "Imitation learning in a setting where experts have contextual information not available to the imitator. Imitator sees state-action (no reward) trajectories for the expert data. The paper proposes to use the expert data as prior and then do meta-RL to learn a policy.", "questions": "- Why is human-in-the-loop highlighted as a limitation?\n- Can you give a more fleshed out example of how the problem of unobserved contextual information might appear in a realistic problem?\n- Can you lay out an idea on how to scale the method?", "limitations": "Limitations are discussed but some of the limitations highlighted seem a little strange. See questions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The problem setting considered is important and there isn't too much prior work in it.\n- The proposed method seems sensible.\n- The paper provides theoretical and empirical justification for the proposed method.", "weaknesses": "- The empirical evaluation is limited to simple domains and it is not discussed why that is the case. It would be good to include an explanation on why this was not scaled up to some more complex environments.\n- Comparison to ExPLORe: if there is a way to use the same actor-critic algorithm for the proposed method and the comparison that would strengthen the paper. As the text says, now it is not clear whether the difference comes from the online RL part or from the difference in how they use the expert data.\n- ADVISOR [1] targets the same problem setting. Differences to that algorithm should be discussed. And maybe a pointer should be given on why their method is demonstrated in high-dimensional domains and this isn't. Not saying that this has to be evaluated in high-dimensional domains, but that it is not immediately clear why this isn't.\n- Figure 4\n    - Explain the pane meanings explicitly in the caption.\n    - Explain the axes explicitly in the caption. I.e., are these learning curves?\n    - How come the naive baseline does worse in the low entropy setting than in the high entropy one?\n\n[1] Luca Weihs, Unnat Jain, Iou-Jen Liu, Jordi Salvador, Svetlana Lazebnik, Aniruddha Kembhavi, and Alex Schwing. Bridging the imitation gap by adaptive insubordination. Advances in Neural Information Processing Systems, 34:19134–19146, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and in-depth feedback. We are encouraged that you find the problem setting important, and the proposed method sensible with theoretical and empirical justification. We address your questions below.\n\n> The empirical evaluation is limited to simple domains and it is not discussed why that is the case...\n\nPlease see the General Response for our detailed answer.\n\n> Comparison to ExPLORe: if there is a way to use the same actor-critic algorithm for the proposed method ...\n\nThank you for mentioning this interesting point. As we have discussed in the paper, we agree with you that some of the results might be due to the difference between the base actor-critic method in ExPLORe and the bootstrapped DQN method used in our approach. However, note that in Table 1, on the left side (Fixed # Hazard = 9), under $\\beta = 0.1$, the ExPLORe method achieves near-optimal results while our approaches suffer large regrets. This particular example shows that, at least in the Frozen Lake environment, the baseline actor-critic method is superior to the baseline bootstrapped DQN since the expert data is relatively uninformative for $\\beta = 0.1$, i.e., the taken actions are almost random. As we increase the value of $\\beta$ and the expert data gets more informative, our proposed approaches outperform the naive bootstrapped DQN, showing that the main performance comes from the proposed method of learning the priors and not the base bootstrapped DQN algorithm. We leave the variant of ExPerior that relies on actor-critic methods to future work.\n\n> ADVISOR [1] targets the same problem setting. Differences to that algorithm should be discussed ...\n\nThe main difference to ADVISOR is that they assume having access to the expert policy $\\pi^{\\text{teach}}$, i.e., they have a human-in-the-loop setting during training and can query the expert for new states. On the other hand, our approach only assumes a finite offline data from the expert trajectories. In other words, there is no expert demonstration for new states that the learner can encounter. We will elaborate on the difference in the revised version. Please see our General Response for the high-dimensional domains.\n\n> Figure 4: (1) Explain the pane meanings explicitly in the caption. (2) Explain the axes explicitly in the caption. I.e., are these learning curves?\n\nThank you for the suggestions. We will include the meaning of each panel (low-entropy to high-entropy setting) in the caption. The y-axis refers to the expected reward for some fixed number of evaluation episodes at each training episode $t \\leq T$. We will include that in the caption as well.\n\n> Figure 4: How come the naive baseline does worse in the low entropy setting than in the high entropy one?\n\nThis is an astute observation. The reason is that as the entropy of the tasks gets higher, i.e., the goal location gets uniformly distributed over all the columns, the task will intrinsically get easier. Since the agent always starts from the top-left corner, it is much harder for it to explore and reach the bottom-right corner compared to bottom-middle or bottom-left cells. This is why, on average, the naive baseline gets higher rewards in the high-entropy case compared to the low-entropy setting (the goal column always being in the right corner).\n\n\n> Q. Why is human-in-the-loop highlighted as a limitation?\n\nBy human-in-the-loop, we mean the requirement of the algorithm to query experts during training or inference. Our approach, on the other hand, only requires an offline dataset of expert demonstrations, which is a strictly simpler requirement. It might be difficult to access an online expert (e.g., a clinical doctor) during training. This is what we mean by a limitation. Having said that, there are benefits in human-centric AI and looping humans in decision-making that we haven’t discussed in our paper. We will elaborate on this point in the paper’s Conclusion.\n\n> Q. Can you give a more fleshed out example of how the problem of unobserved contextual information might appear in a realistic problem?\n\nIn clinical applications, the objective descriptions of a patient are often documented through electronic health records (EHR). However, the subjective opinion of a doctor after having conversations with the patient and observing their visual clues or other information, such as their lifestyle, might lead to potentially different diagnoses. Even by adding clinical notes to the EHR data, it is fundamentally challenging to capture all the observations made by the doctors that influence their decisions. Another example is large language models, which are trained on datasets that do not capture contextual information. For example, suppose we ask an expert about the location of a historical building. If the expert remembers that as part of her internal knowledge, she will answer immediately. Training LLMs on such datasets might result in hallucination as the model does not capture the contextual information – When asked about the location of another historical building, the model might think like an expert and generate some response based on their internal knowledge, which can be wrong. On the other hand, what you might expect from such a model is to search on the internet to respond. Please See [1] and [2] (Figure 7) for more detail.\n\n> Q. Can you lay out an idea on how to scale the method?\n\nPlease see the General Response for our detailed answer.\n\nWe hope our response addressed your comments. If you have any additional questions that could help you improve your evaluation of the work, please feel free to let us know.\n\n===================================\n\n[1] Johnson, Daniel D., et al. \"Experts Don't Cheat: Learning What You Don't Know By Predicting Pairs.\" arXiv preprint arXiv:2402.08733 (2024).\n \n[2] Wang, Kuan, et al. \"Adapting LLM Agents with Universal Feedback in Communication.\" ICML 2024 Workshop on Foundation Models in the Wild.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their helpful and thorough feedback. We are happy that they found our work clearly written, well-organized, and with comprehensive simulations (Reviewer ${\color{red} \text{11BA}}$), significant with a realistic setting and sufficient theoretical basis (${\color{blue} \text{hfW9}}$), well-motivated, easy to understand, with clearly-outlined contributions (${\color{orange} \text{2ZF5}}$), and sensible with an important setting and theoretical and empirical justification (${\color{ForestGreen} \text{FGqG}}$). We discuss the common concern on the simplified experiments and ongoing plans to scale our method, ExPerior, to high-dimensional settings. We answer each reviewer's specific questions and concerns separately.

**Limitated Empirical Evaluation (${\color{blue} \text{hfW9}}$, ${\color{ForestGreen} \text{FGqG}}$)**

1. First, we emphasize that we have run our proposed methods on bandits, MDPs and POMDPS where, in practically all instances, our proposed algorithm outperforms the baselines. To elaborate, within each setting we study:
  a. Multi-armed bandits (with various prior functions, misspecified expert models, misspecified prior functions, and under both exact posterior sampling and stochastic gradient Langevin dynamics),
  b. Markov Decision Processes (MDPs) in the Deep Sea and Frozen Lake environments, and
  c. Partially Observable MDPs (POMDPs) in the Frozen Lake environment.
To our knowledge, most papers in the literature only consider a subset of those settings (bandits, MDPs, or POMDPs) for their experiments, and we view the breadth of applicability of our approach to be a considerable strength of ExPerior.

2. The environments used in our experiments, i.e., *Deep Sea*, *Frozen Lake*, or similar grid environments, are standard in the literature, particularly for imitation learning from experts under unobserved confounding [1,2,3,4].

We are currently working on expanding the framework to study its performance on more high-dimensional problems and/or more complex environments such as MuJoCo. Therefore, we are working on providing such experiments in the revised version. 

**Scaling the Method (${\color{ForestGreen} \text{FGqG}}$)**

ExPerior, derives a prior distribution over Q-functions from expert demonstrations and initializes the Q-networks by sampled parameters from such prior. For high-dimensional settings, the Q-networks will have a large number of parameters; to handle this, we intend to:

1. Pre-train a representation network using self-supervised learning on the offline expert data, e.g., by pre-training a convolutional module to capture the meaningful state representation from images of the Frozen Lake environment (instead of using the grid representation) or the MuJoCo environment.
2. Construct the Q-network by freezing the pre-trained module and adding a learnable linear layer on top of it so it will capture different Q-functions for different values of unobserved factors. This enables us to learn and sample from a prior distribution defined over a simple linear layer instead of the entire Q-network.


===================================

[1] Warrington, Andrew, et al. "Robust asymmetric learning in pomdps." International Conference on Machine Learning. PMLR, 2021.

[2] Shenfeld, Idan, et al. "Tgrl: An algorithm for teacher guided reinforcement learning." International Conference on Machine Learning. PMLR, 2023.

[3] Hao, Botao, et al. "Bridging imitation and online reinforcement learning: An optimistic tale." arXiv preprint arXiv:2303.11369 (2023).

[4] Johnson, Daniel D., et al. "Experts Don't Cheat: Learning What You Don't Know By Predicting Pairs." arXiv preprint arXiv:2402.08733 (2024).

Meta-review: The paper studies sequential decision-making in the presence of expert demonstrations based on unobserved contextual information. The reviewers have identified some valuable aspects of the paper:

- Well-motivated problem
- Good presentation

However, they have also noted several concerns:

- Assumption 1 may be very limiting
- Limited experimental evaluation in terms of environments and a lack of comparison with relevant baselines
- The conditions under which performance improvement is possible (as outlined in Theorem 2) are quite extreme (the expert always plays the optimal action).
Decision: Accept (poster)