Sequential Decision Making with Expert Demonstrations under Unobserved Heterogeneity

Authors: vahid  balazadeh
keertana  chidambaram
viet  nguyen
rahul g krishnan
vasilis  syrgkanis
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 02:58:57

Abstract: We study the problem of online sequential decision-making given auxiliary demonstrations from _experts_ who made their decisions based on unobserved contextual information. These demonstrations can be viewed as solving related but slightly different tasks than what the learner faces.  This setting arises in many application domains, such as self-driving cars, healthcare, and finance, where expert demonstrations are made using contextual information, which is not recorded in the data available to the learning agent. We model the problem as a zero-shot meta-reinforcement learning setting with an unknown task distribution and a Bayesian regret minimization objective, where the unobserved tasks are encoded as parameters with an unknown prior. We propose the Experts-as-Priors algorithm (ExPerior), an empirical Bayes approach that utilizes expert data to establish an informative prior distribution over the learner's decision-making problem. This prior enables the application of any Bayesian approach for online decision-making, such as posterior sampling. We demonstrate that our strategy surpasses existing behaviour cloning and online algorithms, as well as online-offline baselines for multi-armed bandits, Markov decision processes (MDPs), and partially observable MDPs, showcasing the broad reach and utility of ExPerior in using expert demonstrations across different decision-making setups.

Main paper: 
 introduction: 
 Reinforcement learning (RL) has found success in complex decision-making tasks, spanning areas such as game playing [1, 2, 3] , robotics [4, 5] , and aligning with human preferences [6] . However, RL's considerable sample inefficiency, necessitating millions of training frames for convergence, remains a significant challenge. A notable body of work within RL has been dedicated to integrating expert demonstrations to accelerate the learning process, employing strategies like offline pretraining [7] and the use of combined offline-online datasets [8, 9] . While these approaches are theoretically sound and empirically validated [10, 11] , they typically presume homogeneity between the offline and online datasets. A vital question arises about the effectiveness of these methods when expert data embody heterogeneous tasks, indistinguishable by the learner.An important example of such heterogeneity is in situations where experts operate with additional information not available to the learner, a scenario previously explored in imitation learning with unobserved contexts [12, 13, 14, 15] . Existing literature either relies on the availability of experts to query during training [16, 17, 18, 19] or focuses on the assumptions that enable imitation learn- ing with unobserved contexts, sidestepping online reward-based interactions [20, 21] . Recent contributions by Hao et al. [22, 23] suggest using offline expert data for online RL, albeit without accounting for unobserved variations. Our work addresses the more general challenge of online decision-making given auxiliary offline expert data with unobserved heterogeneity. We view such demonstrations as solving related yet distinct problems from those faced by the learner, where differences remain invisible to the learner. For instance, in a personalized education scenario, while a learning agent can observe characteristics like grades or demographics, it might remain oblivious to factors such as learning styles, which are visible to an expert teacher and can significantly influence teaching methods. Naïve imitation without access to this "private" information will only learn a single policy for each observed characteristic [24] , leading to sub-optimal actions. On the other hand, a purely online approach requires extensive trial and error to result in meaningful decisions.We integrate offline expert data with online RL, treating the scenario as a zero-shot metareinforcement learning (meta-RL) problem with an unknown distribution over unobserved contextual variables. Unlike typical meta-RL frameworks where the learner is exposed to multiple instances during training (different students in our example) to learn the underlying distribution [25, 26] , our approach only leverages offline expert data to infer the distribution of unobserved factors, embodying a zero-shot meta-RL paradigm [27] .Contributions. We define a Bayesian regret minimization objective and consider unobserved variables as parameters under an unknown prior distribution. We use empirical Bayes to derive an informative prior over the unobserved variables from expert data. We use the learned prior distribution to drive exploration in the online RL task, using approaches like posterior sampling [28] . We propose two procedures to learn such a prior: (1) a parametric approach that can utilize any existing knowledge about the parametric form of the prior distribution, and (2) a nonparametric approach that employs the principle of maximum entropy when such prior knowledge does not exist. We call our framework Experts-as-Priors or ExPerior for short. See Figure 1 for a goal-oriented RL example. ExPerior outperforms existing offline, online, and offline-online baselines in multi-armed bandits, Markov decision processes (MDPs), and partially observable MDPs. For multi-armed bandits, we find the Bayesian regret incurred by ExPerior is proportional to the entropy of the optimal action under the prior distribution, aligning with the entropy of expert policy if the experts are optimal. We introduce a frequentist algorithm for multi-armed bandits and prove a Bayesian regret bound proportional to a term that closely resembles the entropy of the optimal action. Our results suggest using the entropy of expert demonstrations to evaluate the impact of unobserved factors. 
related work: 
 Our work is an addition to the recent body of reinforcement learning research that leverages offline demonstrations to speed up online learning [29, 10, 30, 7, 9] . Classic algorithms such as DDPGfD [31] and DQfD [32] achieve this by combining imitation learning and RL. They modify DDPG [5] and DQN [1] by warm-starting the algorithms' replay buffers with expert trajectories and ensuring that the offline data never gets overridden by online trajectories. Closely related to our study is the meta-RL literature, which aims to accelerate learning in a given RL task by using prior experience from related tasks [33, 34, 35] . These papers present model-agnostic meta-learning training objectives to maximize the expected reward from novel tasks as efficiently as possible.Two unique features distinguish our problem from the settings considered above. First, our setting assumes heterogeneity within the offline data and with the online RL task that is unobserved to the learner, while the (optimal) experts are privy to that heterogeneity. Second, we assume the learner will only interact with one online task, making our setup similar to zero-shot meta-RL [27, 36, 37] . Most similar to our work is the ExPLORe algorithm [38] , which assigns optimistic rewards to the offline data during the online interaction and runs an off-policy algorithm using both online and labelled offline data as buffers. For our setting, the algorithm incentivizes the learner to explore the expert trajectories, leading to faster convergence. We consider this work one of our baselines.Our methodology utilizes only the state-action trajectory data from expert demonstrations without task-specific information or reward labels. Other similar methods require additional offline information. For example, Nair et al. [30] assume that the offline data contains the reward labels and use that to pre-train a policy, which is then fine-tuned online. Mendonca et al. [39] require task labelling for each trajectory and use the offline data to learn a single meta-learner. Similarly, Zhou et al. [40] and Rakelly et al. [41] require the task label and reward labels. They then infer the task during online interaction and use the task-specific offline data. Lee et al. [42] requires a large amount of noisy expert data with reward labels, in addition to the optimal trajectory data, for good performance. Finally, our methodology builds on posterior sampling [43] . Hao et al. [22, 23] consider a similar problem using posterior sampling to leverage offline expert demonstration data to improve online RL. However, they assume homogeneity between the expert data and online tasks. In contrast, our setting accounts for heterogeneity. 
problem setup: 
 Decision Model for Unobserved Heterogeneity. To account for unobserved heterogeneity, we consider a generalization of finite-horizon Markov Decision Processes (MDPs) with a notion of probabilistic contextual variables [44, 13, 21] . The underlying model for the MDP will additionally depend on an unobserved variable that encapsulates the information hidden from the learner. For example, consider a personalized education setup where teaching a student corresponds to a task, and the learning agent can observe students' characteristics, like their demographic status and grades. Other factors, such as the student's learning style (e.g., visual learners or self-study), may not be readily available, even though they are important in determining the optimal teaching style.Let C be the set of all unobserved context variables that can describe the unobserved heterogeneity of the decision-making problem (e.g., the set of all possible learning styles). A contextual MDP M = (S, A, T , R, H, ρ, µ ⋆ ) is parameterized by states S, actions A, transition functionT : S × A × C → ∆(S), reward function R : S × A × C → ∆(R), horizon H > 0, initial state distribution ρ ∈ ∆(S), and context distribution µ ⋆ . We assume the transition/reward functions and µ ⋆ are unknown, and for simplicity, ρ does not depend on the context variable. For each unobserved context c ∼ µ ⋆ , we consider T episodes, where at the beginning of each episode t ∈ [T ], an initial state s 1 ∼ ρ is sampled. Then, at each timestep h ∈ [H], the learner chooses an action a h ∈ A, observes a reward r h ∼ R (s h , a h , c) and the next state s h+1 ∼ T (s h , a h , c). Without loss of generality, we assume the states are partitioned by [H] to make the notation invariant to the timestep. Let Π be the set of all Markovian policies. For a policy function π : S → ∆(A) ∈ Π and context variable c, we define the value functionV c (π) = E H h=1 r h π, c and the Q-function as Q π c (s, a) := E H h ′ =h r h ′ s h = s, a h = a, π, cfor all s ∈ S, a ∈ A. Moreover, we define the optimal policy for a context variable c ∈ C as π c := arg max π∈Π V c (π). Note that since the context variable is unobserved, the learner's policy will not depend on it. The learning agent's goal is to learn history-dependent distributions p 1 , . . . , p T ∈ ∆(Π) over Markovian policies to minimize the expected regret, defined as Reg := E c∼µ ⋆ T t=1 V c (π c ) -E π t ∼p t [V c (π t )] .In the personalized education example, the above setup assumes a fixed distribution µ ⋆ over the set of learning styles and aims to minimize expected regret over the population of students. Our setup and regret assume the unobserved factors remain fixed during training. This captures scenarios wherein the unobserved variables correspond to less-variant factors (a student's learning style is more likely to remain unchanged). No learning algorithm can control the regret value if we allow the unobserved factors to change arbitrarily throughout T episodes without access to hidden information; consider a two-armed bandit with a context value drawn with uniform probability from C = {c 1 , c 2 } that can change at each episode. Assume the expected reward of the first arm under c 1 and c 2 is one and zero, respectively, and it is reversed for the other arm. Any algorithm that does not have access to c would result in linear regret since each action is sub-optimal with a probability of 0.5, independent of the algorithm's choice.Remark. Our setup can be formulated as a Bayesian model parameterized by C, and our regret can be seen as the Bayesian regret of the learner. However, the distribution µ ⋆ is not the learner's prior belief about the true model as it is often formulated in Bayesian learning, but a distribution over potential contexts that the learner can encounter. Our setup can thus be seen as a meta-learning problem. In fact, it is zero-shot meta-learning since we do not assume having access to more than one online RL task during training -we only learn the prior distribution using the offline data.Expert Demonstrations. In addition to the online setting described above, we assume the learner has access to an offline dataset of expert demonstrations D E , where each demonstration τ E = (s 1 , a 1 , s 2 , a 2 , . . . , s H , a H , s H+1 ) refers to an interaction of the expert with a decision-making task during a single episode, containing the actions made by the expert and the resulting states. We assume that the unobserved context variables for D E are drawn i.i.d. from distribution µ ⋆ , and the expert had access to such unobserved variables (private information) during their decision-making. Moreover, we assume the expert follows a near-optimal strategy [22, 23] .Assumption 1 (Noisily Rational Expert). For any c ∈ C, experts select actions based on a distribution defined as p E (a | s ; c) ∝ exp {β • Q πc c (s, a)}, for all s ∈ S, a ∈ A, and some known competence value of β ∈ [0, ∞]. In particular, the expert follows the optimal policy if β → ∞.We assume experts do not provide any rationale for their strategy, nor do we have access to rewards in the offline data; this is a combination of imitation and online learning rather than offline RL [22] . 
experts-as-priors framework for unobserved heterogeneity: 
 Our goal is to leverage offline data to help guide the learner through its interaction with the decisionmaking task. The key idea is to use expert demonstrations to infer a prior distribution over C and then to use a Bayesian approach such as posterior sampling [28] to utilize the inferred prior for a more informative exploration. If the current context is from the same distribution of contexts in the offline data, we expect that using such priors will lead to faster convergence to optimal trajectories compared to the commonly used non-informative priors. Consider the personalized education example. Suppose we have gathered offline data on an expert's teaching strategies for students with similar observed information like grade, age, location, etc. The teacher can observe more finegrained information about the students that is generally absent from the collected data (e.g., their learning style). Our work relies on the following observation: The space of the optimal strategies for students with similar observed information but different learning styles is often much smaller than the space of all possible strategies. With the inferred prior distribution, the learner needs only to focus on the span of potentially optimal strategies for a new student, allowing for significantly more efficient exploration.We resort to empirical Bayes and use maximum marginal likelihood estimation [45] to construct a prior distribution from D E . Given a probability distribution (prior) µ on C, the marginal likelihood of an expert demonstration τ E = (s 1 , a 1 , s 2 , a 2 , . . . , s H , a H , s H+1 ) ∈ D E is given byEQUATIONWe aim to find a prior distribution to maximize the log-likelihood of D E under the model described in (1) . This is equivalent to minimizing the KL divergence between the marginal likelihood P E and the empirical distribution of expert demonstrations, which we denote by P E . In particular, we form an uncertainty set over the set of plausible priors as P(ϵ) := µ ; D KL P E P E (• ; µ) ≤ ϵ , where the value of ϵ can be chosen based on the number of samples so the uncertainty set contains the true prior with high probability [27] . However, the set of plausible priors does not uniquely identify the appropriate prior. In fact, even for ϵ = 0, P(ϵ) can have infinite plausible priors. To solve this ill-posed problem, we propose two approaches, parametric and nonparametric prior learning.Parametric Experts-as-Priors. For settings where we have existing knowledge about the parametric form of the prior, we can directly apply maximum marginal likelihood estimation to learn it. In particular, we define the parametric expert prior as the following. Note that we can calculate the gradients of the marginal likelihood using the score function estimator [46] .Definition 1 (Parametric Expert Prior). Let Θ be a set of plausible prior distribution parameters (e.g., Beta distribution parameters for a Bernoulli bandit). We call µ θ ⋆ a parametric expert prior, iff θ ⋆ ∈ arg min θ∈Θ τ ∈DElog P E (τ ; µ θ ).Nonparametric Experts-as-Priors. For settings where there is no existing knowledge on the parametric form of the prior, we can employ the principle of maximum entropy to choose the least informative prior that is compatible with expert data: Definition 2 (Max-Entropy Expert Prior). Let µ 0 be a non-informative prior on C (e.g., a uniform distribution). Given some ϵ > 0, we define the maximum entropy expert prior µ ME as the solution to the following optimization problem:EQUATIONNote that the set of plausible priors P(ϵ) is a convex set, and therefore, ( 2) is a convex optimization problem. We can derive the solution to problem (2) using Fenchel's duality theorem [47, 48] :Proposition 1 (Max-Entropy Expert Prior). Let N = |D E | be the number of demonstrations in D E . For each c ∈ C and demonstration τ E = (s 1 , a 1 , s 2 , a 2 , . . . , s H , a H , s H+1 ) ∈ D E , define m τE (c) as the (partial) likelihood of τ E under c, i.e., m τE (c) = H h=1 p E (a h | s h ; c) T (s h+1 | s h , a h , c). Denote m(c) ∈ R N as the vector with elements m τE (c) for τ E ∈ D E . Moreover, let λ ⋆ ∈ R ≥0 be the optimal solution to the Lagrange dual problem of (2). Then, the solution to optimization (2) is:µME(c) = lim n→∞ exp m(c) ⊤ αn E c ′ ∼µ 0 [exp {m(c ′ ) ⊤ αn}] ,where {α n } ∞ n='1 is a sequence converging to the following supremum:EQUATION)The proof is provided in Appendix A.3. Instead of solving for λ ⋆ , we set it as a hyperparameter and then solve (3). Even though Proposition 1 requires the correct form of Q-functions for different values of c, we will see in the following sections that we can parameterize the Q-functions and treat those parameters as a proxy for the unobserved factors. Once such a prior is derived, we can employ any Bayesian approach for decision-making. We provide a pseudo-algorithm for ExPerior in Algorithm 1. The following sections will detail the algorithm for bandits and MDPs. end for 10: end for 5 Learning in Bandits K-armed Bandits. For K-armed bandits, note that S = ∅, H = 1, and A = {1, . . . , K}. Each expert demonstration τ E = a will be the pulled arm by the expert for a particular bandit, and the (partial) likelihood function in Proposition 1 can be simplified as m τE (c) = p E (a ; c). This likelihood function only depends on the context variable c through the expert policy p E , and since p E only depends on c through the mean reward function (Assumption 1), we can consider the set of mean reward functions as a proxy for the unobserved context variables C. e.g. in a Bernoulli K-armed bandit setting, we can define Posterior Sampling. With the above parameterizations of C, we can use Proposition 1 to derive the maximum entropy prior distribution over the context parameters. However, we cannot sample from the exact posterior since the derived prior is not a conjugate prior for standard likelihood functions. Instead, we resort to approximate posterior sampling via stochastic gradient Langevin dynamics (SGLD) [49] . We call this method ExPerior-MaxEnt in our experiments. We also employ a parametric approach as discussed in section 4, which we call ExPerior-Param. In particular, we use the Beta distribution as our prior model and learn the parametric expert prior in Definition 1.C Ber = a → ⟨e a , ϑ⟩ ; ϑ ∈ [0, 1] K .In the following, we evaluate our approach compared to online methods that do not use expert data and offline behaviour cloning. We provide an empirical regret analysis for ExPerior based on the informativeness of expert data, number of actions, and number of training episodes. We also discuss the robustness of ExPerior to misspecified expert models and the advantage of ExPerior-MaxEnt to ExPerior-Param when the parametric prior model is misspecified. To characterize the effect of expert data on the learner's performance, we propose an alternative for K-armed bandits inspired by the successive elimination and derive a Bayesian regret bound for it.Experiments. We consider K-armed Bernoulli bandits for our experimental setup. We evaluate the learning algorithms in terms of the Bayesian regret over multiple (prior) distributions µ ⋆ over the unobserved contexts. In particular, we consider up to N µ ⋆ = 64 different beta distributions, where their parameters are chosen to span a different range of heterogeneity, consisting of tasks with various expert data informativeness. To estimate the Bayesian regret, we sample N task = 128 bandit tasks from each prior distribution and calculate the average regret. We use N E = 1000 expert demonstrations for each prior distribution in our experiments. We compare ExPerior to the following baselines: (1) Behaviour cloning (BC), which learns a policy by minimizing the cross-entropy loss between the expert demonstrations and the agent's policy solely based on offline data. (2) Naïve Thompson sampling (Naïve-TS) that chooses the action with the highest sampled mean from a posterior distribution under an uninformative prior. (3) Naïve upper confidence bound (Naïve-UCB) algorithm that selects the action with the highest upper confidence bound. Both Naïve-TS and Naïve-UCB ignore expert demonstrations. (4) UCB-ExPLORe, a variant of the algorithm proposed by Li et al. [38] tailored to bandits. It labels the expert data with optimistic rewards and then uses it alongside online data to compute the upper confidence bounds for exploration, and (5) Oracle-TS, which performs exact Thompson sampling with the true prior distribution µ ⋆ . For a fair comparison, we also consider a variant of Oracle-TS, which uses SGLD for approximate posterior sampling.Comparison to baselines. Figure 2 demonstrates the average Bayesian regret for various prior distributions over T = 1,500 episodes with K = 10 arms. To better understand the effect of expert data, we categorize the prior distributions by the entropy of their optimal actions into low entropy (less than 0.8), high entropy (greater than 1.6), and medium entropy. Oracle-TS and ExPerior-Param outperform other baselines, yet the performance of ExPerior-MaxEnt is comparable to the SGLD variant of Oracle-TS. This indicates that the maximum entropy prior derived from Proposition 1 closely approximates the true prior distribution, µ ⋆ , and the performance difference with Oracle-TS is primarily due to approximate posterior sampling. Moreover, the pure online algorithms Naïve-TS and Naïve-UCB, which disregard expert data, display similar performance across different entropy levels, contrasting with other algorithms that show significantly reduced regret in low-entropy contexts. This underlines the impact of expert data in settings where the unobserved confounding has less effect on the optimal actions. Specifically, in the extreme case of no unobserved heterogeneity, BC is anticipated to yield optimal performance. Additionally, Naïve-UCB surpasses UCB-ExPLORe in medium and high entropy settings, possibly due to the over-optimism of the reward labelling in Li et al. [38] , which can hurt the performance when the expert demonstrations are uninformative.Empirical regret analysis for Experts-as-Priors. We examine how the quality of expert demonstrations affects the Bayesian regret achieved by ExPerior. Settings with highly informative demon- 24.2 ± 3.9 24.2 ± 3.9 24.2 ± 3.9 24.2 ± 3.9 24.2 ± 3.9 24.2 ± 3.9 24.2 ± 3.9 24.2 ± 3.9 24.2 ± 3.9strations, where unobserved factors minimally affect the optimal action, should exhibit near-zero regret since there is no diversity in the unobserved contexts, and the experts are near-optimal. Conversely, in scenarios where unobserved factors significantly influence the optimal actions, we anticipate the regret to align with standard online regret bounds, similar to the outcomes of Thompson sampling with a non-informative prior. We conduct trials with ExPerior and Oracle-TS across various numbers of arms over T = 1,500 episodes, calculating the mean and standard error of Bayesian regret across distinct prior distributions. As depicted in Figure 3 (a), both ExPerior and Oracle-TS yield sub-linear regret relative to K and T , comparable to the established regret bound of O( √ KT ) for Thompson sampling. However, the middle panel indicates that the regret of ExPerior is proportional to the entropy of the optimal action, having an almost linear relationship. This observation seems to be in contrast with the standard Bayesian regret bounds for Thompson sampling under correct prior that have shown a sublinear relationship of O Ent(π c ) , where Ent(π c ) denotes the entropy of the optimal action under µ ⋆ [50] . We analyze this observation in section 5.1.Ablations. We also run additional experiments to assess the robustness of ExPerior to misspecified experts. We create expert data from different experts with various competence levels, such as optimal, noisily rational, and random-optimal experts, where the latter chooses an action optimally with a fixed probability and randomly otherwise. Table 1 shows ExPerior's robustness to different expert models. Setting β = 10 for training ExPerior-MaxEnt and β = 1 for ExPerior-Param achieves consistent out-performance among different expert types. Moreover, We evaluate the advantage of learning nonparametric max-entropy prior over misspecified parametric priors in Table 2 . Even though ExPerior-Param with a Beta prior outperforms ExPerior-MaxEnt, ExPerior-MaxEnt is superior to ExPerior-Param if the prior mismatches the correct form (e.g., Gaussian or Gamma). 
an alternative frequentist approach for k-armed bandits: 
 To analyze the effect of expert data on the Bayesian regret, we devise an alternative frequentist approach, based on the successive elimination algorithm [51] , which follows a similar intuition to Experts-as-Priors. In particular, we prove a bound on its Bayesian regret and show that the derived bound is proportional to a term that closely resembles the entropy of the optimal action, showing that the observation in the middle panel of Figure 3 (a) is consistent within different approaches. The idea of successive elimination is to identify suboptimal arms and deactivate them over time. In particular, it runs a uniform sampling policy among active arms and builds confidence intervals for each. It then deactivates all the arms with an upper confidence bound smaller than at least one arm's lower confidence bound. We modify this algorithm using the policy derived from expert demonstrations instead of a uniform sampling policy. Recall that in K-armed bandits, each expert trajectory τ E represents the pulled arm by the expert. Hence, the empirical distribution of expert demonstrations can be seen as a sampling policy over different arms. To simplify the analysis, we employ a deterministic sampling approach by pulling each arm a fixed number of times based on its probability. To do so, we discretize the expert policy with a step size p min , which leads to a relative frequency of ⌈ P E (a)/p min ⌉ for an arm a. In particular, we can choose p min = min a; PE(a)̸ =0 P E (a). We provide the concrete algorithm in Algorithm 2 and prove the following Bayesian regret bound: Theorem 2. Consider a stochastic K-armed bandit and let p be the empirical expert policy. Assume that (i) the mean reward function is bounded in [0, 1] for all arms, (ii) T ≥ 1 min a;p(a)̸ =0 p(a) , (iii) the expert is optimal, i.e., ∀a ∈ A : p(a) = P E (a ; µ ⋆ ) and β → ∞, and (iv) the learner follows Algorithm 2. Then, with probability at least 1δ,EQUATIONSee Appendix A.4 for the proof. Two terms in (4) depend on expert data: (1) The relative standard deviation between any two pairs of arms and (2) a scaling factor that depends on the magnitude of probability that the arms are optimal. For homogeneous demonstrations, where the expert data only includes one unique pulled arm, the standard deviation (Term 1) is zero, resulting in zero regret. However, in extreme heterogeneity, where the empirical expert distribution is uniform over the arms, we have Reg ≲ K √ KT log T . 2 Finally, to assess the relationship between the regret bound and the entropy of the expert data, we fix K = 2, T = 100, and plot the bound from (4) as a function of the entropy of the optimal action for various prior distributions. Figure 3 (b) demonstrates a linear relationship, similar to the regret incurred by ExPerior in Figure 3 (a) . This observation opens up new directions to further analyze the regret for ExPerior and similar approaches in MDPs. 
learning in markov decision processes (mdps): 
 For MDPs, we need to parameterize both the mean reward and transition functions. However, we assume the transition functions are invariant to the context variables to simplify our methodology and avoid extra modelling assumptions. Under this assumption, it is sufficient to parameterize the optimal Q-functions, e.g., using a deep Q-network (DQN) and treat those parameters as a proxy for the unobserved context variables, i.e., C MDP := {(s, a) → Q (s, a ; θ) ; θ ∈ Θ}, where Θ is the set of parameters for a DQN. We can then derive a closed-form log-pdf of the posterior distribution under the maximum entropy prior. See Appendix A.5 for details. The derived posterior log-pdf can (POMDP) ExPerior-MaxEnt -22.58 ± 1.17 6.00 ± 0.00 3. 58 5.99 ± 0.00 6.00 ± 0.00 6.00 ± 0.00 6.00 ± 0.00 -30.68 ± 12.40 -10.64 ± 16.64 -13.00 ± 19.00 6.00 ± 0.00 Optimal 6.00 ± 0.00 6.00 ± 0.00 6.00 ± 0.00 6.00 ± 0.00 12.00 ± 0.37 6.53 ± 0.31 6.00 ± 0.00 6.00 ± 0.00 then be used as the loss function for DQN Langevin Monte Carlo [52, 53] as the counterpart for Thompson sampling with SGLD. However, running Langevin dynamics can lead to highly unstable policies due to the complexity of the optimization landscape in DQNs. Instead, we use a heuristic that combines the learned prior distribution with bootstrapped DQNs [54] .(The original method of Bootstrapped DQNs utilizes an ensemble of L randomly initialized Qnetworks. It samples a Q-network uniformly at each episode and uses it to collect data. Then, each Q-network is trained using the temporal difference loss on parts of or possibly the entire collected data. This method and its subsequent iterations [55, 56, 57] achieve deep exploration by ensuring diversity among the learned Q-networks. To incorporate Bootstrapped DQN into the ExPerior framework and utilize the expert data, we can formulate the ensemble as a discrete prior distribution over the Q-networks. Let θ ens = θ 1 ens , . . . , θ L ens be the parameter vector for an ensemble of Q-functions. We can define the ensemble prior, parameterized by θ ens , asµ θens (θ) := 1 L L i=1 I θ i ens = θ for any θ ∈ Θ.Based on this prior model, we can learn the parametric expert prior using maximum marginal likelihood estimation, as formulated below. A, T , R, H, ρ, µ ⋆ ). Assume the transition function T does not depend on the context variables and Assumption 1 holds. Then, the negative marginal log-likelihood of expert data D E under the ensemble prior µ θens is upper bounded by-log PE (DE ; µ θens ) ≤ 1 L L i=1 τ ∈D E (s,a)∈τ log a ′ ∈A exp β • Q s, a ′ ; θ i ens -β • Q s, a ; θ i ens ,where β is the competence level of the expert in Assumption 1.Proposition 3 is proved in Appendix A.6. We can then initialize the Q-networks in the Bootstrapped DQN method using ensemble parameters that minimize the above upper bound. We will refer to this method as ExPerior-Param. As an alternative approach, instead of minimizing the above upper bound, we can match the discrete prior distribution µ θens to the max-entropy prior by initializing the Q-functions in the ensemble with parameters sampled from the max-entropy expert prior. In particular, we can apply SGLD on the log-pdf of the max-entropy prior derived in Appendix A.5.We will refer to this approach as ExPerior-MaxEnt.Experimental Setup. One challenge in RL is the reward sparsity, where the learner needs to explore the environment deeply to observe rewards. Utilizing expert demonstrations can significantly improve the efficiency of exploration. Here, we focus on "Deep Sea," a sparse-reward tabular RL environment proposed by Osband et al. [56] to assess deep exploration for different RL methods. The environment is an M × M grid, where the agent starts at the top-left corner of the map, and at each time step, it chooses an action from A = {left, right} to move to the left or right column, while going down by one row. In the original version of Deep Sea, the goal is always on the bottom-right corner of the map. We introduce unobserved contexts by defining a distribution over the goal columns while keeping the goal row the same. We consider four types of goal distributions where the goal is situated at (1) the bottom-right corner of the grid, (2) uniformly at the bottom of any of the right-most M 4 columns, (3) uniformly at the bottom of any of the right-most M 2 columns, and (4) uniformly at the bottom of any of the M columns. We set M = 30 and generate N = 1,000 samples from the optimal policies as offline expert demonstrations. To further evaluate ExPerior and showcase its applicability to partially-observed MDP, we also consider the "Frozen Lake" environment, which requires the learner to navigate to a goal while avoiding hazards [17] . The learner cannot observe the hazard location, while the expert has access to the whole map. Taking action, reaching the goal, and hitting the hazard incur rewards of -2, 20, and -100, respectively. The frozen lake map is 5 × 5, where the hazard (weak ice) is randomly located in the interior squares. We consider different settings with 2, 5, 7, and 9 potential locations for the hazard. At the start of each episode, the hazard will be chosen randomly within the potential locations. We generate N = 1,000 samples from noisily rational experts with different competence levels for this environment.Baselines. We compare ExPerior to the following: (1) ExPLORe, proposed by Li et al. [38] to accelerate off-policy reinforcement learning using unlabeled prior data. In this method, the offline demonstrations are assigned optimistic reward labels generated using the online data with regular updates. This information is then combined with the buffer data to perform off-policy learning. (2) Naïve Boot-DQN, which is the original Bootstrapped DQN with randomly initialized Q-networks [54] .Deep Sea Results. Figure 4 demonstrates the average reward per episode achieved by the baselines for T = 2,000 episodes. For each goal distribution, we run the baselines with 30 different seeds and take the average to estimate the expected reward. ExPerior outperforms the baselines in all instances. However, the gap between ExPerior and the fully online Naïve Boot-DQN, which measures the effect of using the expert data, decreases as we go from the low-entropy setting (upper left) to the high-entropy distribution over the contexts (bottom right). This is consistent with the empirical and theoretical results discussed in section 5 and confirms our expectation that the expert demonstrations may not be helpful under strong unobserved confounding (strong heterogeneity). The ExPLORe baseline substantially underperforms, even compared to the fully online Naïve Boot-DQN (except for the first distribution with zero-entropy). We suspect this is because ExPLORe uses actor-critic methods as its backbone model, which are shown to struggle with deep exploration [58] .Frozen Lake Results. We run all the baselines for 90,000 steps with 30 different seeds. Table 3 shows the average reward after 500 evaluation steps at the end of the training. ExPerior outperforms the baselines in almost all instances except for the case of β = 0.1, which corresponds to a nearly random expert. On the other hand, ExPLORe achieves near-optimal results for β = 0.1. We hypothesize that ExPLORe's performance is mainly due to the superiority of their base actor-critic model since it can achieve near-optimal performance even when the expert trajectories are low-quality. 
conclusion: 
 We introduce the Experts-as-Priors (ExPerior) framework, a novel empirical Bayes approach to address the problem of sequential decision-making using expert demonstrations with unobserved heterogeneity. We ground our methodology in the maximum entropy principle to infer a prior distribution from expert data that guides the learning process in different setttings, including bandits, Markov decision processes (MDPs), and partially-observed MDPs. Our experimental evaluations demonstrate that we can effectively leverage the expert demonstrations to enhance learning efficiency under unobserved confounding. In multi-armed bandits, we illustrated through empirical analysis that the Bayesian regret incurred by our method is proportional to the entropy of the optimal action, highlighting its capacity to adapt based on the informativeness of the expert data. Our work offers a practical framework readily applied to a broad spectrum of decision-making tasks. One limitation of our work is the limited set of experiments, especially the lack of experiments with human-in-the-loop. Future directions include extending to more complex environments and further investigating the theoretical properties of our RL algorithm. 
 
 Appendix: We assume C is a measurable set with an appropriate σ-algebra and there exists a probability measure µ 0 on C. We denote L p (C, µ 0 ) as the space of all measurable functions f : C → R such that ∥f ∥ p = C |f | p dµ 0 1/p < ∞. Moreover, we define L ∞ (C, µ 0 ) as the space of all essentially bounded measurable functions from C to R. Unless stated otherwise, we assume the probability measures are absolutely continuous w.r.t. µ 0 , and their density functions are in L 1 (C, µ 0 ). We may abuse the notation and use the same symbol for a probability measure and its Radon-Nikodym derivative w.r.t. µ 0 . Finally, we use E [•] to denote expectation under the probability measure µ 0 .Here, we state and prove a set of results that will be useful for the rest of this section. The first one is Fenchel's duality theorem: Lemma 4 (Fenchel's Duality [59] ). Let X and Y be Banach spaces, let f : X → R ∪ {+∞} and g : Y → R ∪ {+∞} be convex functions and let A : X → Y be a bounded linear map. Define the primal and dual values p, d ∈ [-∞, +∞] by the Fenchel problemsp = inf x∈X f (x) + g(Ax) d = sup y * ∈Y * -f * (A * y * ) -g * (-y * ),where f * and g * are the Fenchel conjugates of f and g defined asf * (x * ) = sup x∈X ⟨x * , x⟩ -f (x)(similarly for g), X * is the dual space of X and ⟨•, •⟩ is its duality pairing, and A * : Y ⋆ → X ⋆ is the adjoint operator of A, i.e., ⟨A * y * , x⟩ = ⟨y * , Ax⟩. Suppose A dom(f ) ∩ cont(g) ̸ = ∅, where dom(f ) := {x ∈ X ; f (x) < ∞} and cont(g) are the continuous points of g. Then, strong duality holds, i.e., p = d.Proof. See the proof of Theorem 4.4.3 in Borwein and Zhu [59] .We can use Fenchel's duality to solve generalized maximum entropy problems. In particular, we prove a generalization of Theorem 2 in [48] for density functions in L 1 (C, µ 0 ):Lemma 5. For any function µ ∈ L 1 (C, µ 0 ), define the extended KL divergence asψ(µ) := D KL (µ ∥ µ 0 ) If ∥µ∥ 1 = 1, +∞ o.w.Moreover, assume a set of bounded feature functions m 1 , m 2 , . . . , m N : C → R is given and denote m as the vector of all N features. Consider the linear functionA m : L 1 (C, µ 0 ) → R N defined as ∀µ ∈ L 1 (C, µ 0 ) : A m (µ) := (E [m 1 • µ] , E [m 2 • µ] , . . . , E [m N • µ]) .We define the generalized maximum entropy problem as the following: infEQUATIONfor an arbitrary closed proper convex function ζ : R N → R. Then the following holds:1. The dual optimization of (5) is given by supEQUATIONwhere ζ ⋆ is the convex conjugate function of ζ.2. Denote α 1 , α 2 , . . . as a sequence in R N converging to supremum (6) , and define the following Gibbs density functionsµ α Gibbs (c) := exp m(c) ⊤ α E [exp {m ⊤ α}] .Then,inf µ∈L 1 (C,µ0) ψ(µ) + ζ (A m (µ)) = lim n→∞ ψ(µ α n Gibbs ) + ζ A m (µ α n Gibbs ) .Proof. Part 1: We first derive the convex conjugate of ψ. Note that L 1 (C, µ 0 )⋆ = L ∞ (C, µ 0 ) with the pairing ∀h ∈ L ∞ (C, µ 0 ), µ ∈ L 1 (C, µ 0 ) : ⟨h, µ⟩ := C h(c) • µ(c) dµ 0 .Hence, by Donsker and Varadhan's variational formulaEQUATIONMoreover, the adjoint operator of A m is given byEQUATIONUsing ( 7) and ( 8) and Lemma 4 concludes the proof.Part 2: Denote the primal and dual objective functions byP (µ) := ψ(µ) + ζ (A m (µ)) , D(α) := -log E exp m ⊤ α -ζ * (-α) ,and their optimal values as P * and D * . For any ν ∈ L 1 (C, µ 0 ), note thatEQUATIONUsing ( 9), we can re-write the dual objective function as:EQUATIONMoreover, note thatEQUATIONCombining (10) and (11) , we getEQUATIONNow, fix an arbitrary ϵ > 0, and consider a sequence of µ 1 , µ 2 , . . . ∈ L 1 (C, µ 0 ) such that for all j ∈ N:EQUATION)We can re-write (13) using the fact P * = D * = lim n→∞ D(α n ):EQUATIONIn particular, by setting ν = µ j in (12) and combining the result with ( 14), we get ∀j ∈ N : lim n→∞ D KL µ j µ α n Gibbs < ϵ 2 j . Hence, lim j∈∞ lim n→∞ D KL µ j µ α n Gibbs = 0. From properties of the KL divergence, it follows that lim j→∞ P (µ j ) = lim n→∞ P (µ α n Gibbs ), concluding the proof.A. EQUATION)Denote m(c) ∈ R N as the vector with elements m τE (c) for τ E ∈ D E . Moreover, let λ ⋆ ∈ R ≥0 be the optimal solution to the Lagrange dual problem of (2). Then, the solution to optimization (2) is as follows:µ ME (c) = lim n→∞ exp m(c) ⊤ α n E c∼µ0 [exp {m(c) ⊤ α n }] ,where {α n } ∞ n=1 is a sequence converging to the following supremum:sup α∈R N -log E c∼µ0 exp m(c) ⊤ α + λ ⋆ N N i=1 log N • α i λ ⋆ .Proof. We first simplify the KL-divergence between the empirical distribution of the expert trajectories P E and the marginal likelihood P E (• ; µ):D KL P E P E (• ; µ) = τ (i) ∈DE P E (τ (i) ) log P E (τ (i) ) P E τ (i) ; µ = -log N - 1 N τ (i) ∈DE log P E τ (i) ; µ ( P E (τ (i) ) = 1 N ) = -log N - 1 N τ (i) ∈DE log E [m τ (i) • µ] - 1 N s (i) 1 ∈DE log ρ s (i) 1.By ( 1) and (15) Using the above equality, we can re-write the definition of uncertainty set P(ϵ) asP(ϵ) = µ ; - 1 N τ ∈DE log E [m τ • µ] -ϵ -log N - 1 N s1∈DE log ρ (s 1 ) ≤ 0 .Therefore, we can re-write the optimization (2) asEQUATIONwhere the extended KL divergence ψ(µ) is defined as:ψ(µ) := D KL (µ ∥ µ 0 ) If ∥µ∥ 1 = 1, +∞ o.w.Note that P(ϵ) is a convex set. To see this, consider µ 1 , µ 2 ∈ P(ϵ). Then, for any 0≤ λ ≤ 1, we have µ = (1 -λ)µ 1 + λµ 2 ∈ P(ϵ) since E [m τ • µ]is linear in µ andlog is convex. Moreover, It is easy to see there exists a strictly feasible solution for ( 16) (e.g., consider the true distribution µ ⋆ over C). Thus, strong duality holds, and we can form the Lagrangian function asL(µ, λ) := ψ(µ) + λ 1 N τ ∈DE -log E [m τ • µ] -λ ϵ + log N + 1 N s1∈DE log ρ (s 1 ) .Given that λ ⋆ ∈ R ≥0 is the optimal solution to the Lagrange dual problem, the maximum entropy prior µ ME will be the solution toEQUATIONCombining ( 17) and ( 18), the maximum entropy prior µ ME is the solution toinf µ∈L 1 (C,µ0) ψ(µ) + ζ (A m (µ)) .Using Lemma 5 and noting thatζ * (x * ) = λ ⋆ N N i=1 -1 -log - N λ ⋆ • x * i concludes the proof.A.4 Regret Bound for K-armed Bandit Theorem 2. Consider a stochastic K-armed bandit and let p be the empirical expert policy. Assume that (i) the mean reward function is bounded in [0, 1] for all arms, (ii) T ≥ 1 min a;p(a)̸ =0 p(a) , (iii) the expert is optimal, i.e., ∀a ∈ A : p(a) = P E (a ; µ ⋆ ) and β → ∞, and (iv) the learner follows Algorithm 2. Then, with probability at least 1δ, Proof. Fix δ ∈ (0, 1) and c ∈ C. Let E be the event thatV t c (a) -V c (a) ≤ log(4T 4 K/δ) 2nt(a)for all arms a ∈ A, all t ≤ T , and all T ∈ N, where n t (a) is the number of times that arm a was pulled by time t. Note that since T ≥ 1 pmin , each arm will be pulled at least once and n t (a) ≥ 1. We first show that P (E) ≥ 1δ. Fix T , arm a, and t ≤ T . Suppose n t (a) = j for 1 ≤ j ≤ T . By Hoeffding's inequality, we haveEQUATIONNow, using the union bound over all episodes and all actions, we getP ∃a ∈ A, T ∈ N, t ≤ T, j ≤ t : V t c (a) -Vc(a) > log (2T 4 K/δ) 2j ≤ ∞ T =1 a∈A T t=1 t j=1 P V t c (a) -Vc(a) > log (2T 4 K/δ) 2j ≤ ∞ T =1 a∈A T t=1 t • δ 2T 4 K By (19) ≤ ∞ T =1 δ 2T 4 K × T 2 × K = ∞ T =1 δ 2T 2 ≤ δ, which concludes that P (E) ≥ 1 -δ.The rest of the proof computes the regret for when E holds. For simplicity and without loss of generality, we assume all expert probabilities are dividable by p min . Recall that we follow a deterministic sampling approach and choose each arm according to its relative frequency p(•) pmin for multiple batches, where each batch loops over all active actions. Let t a be the episode in which we eliminate an arm a in favour of another arm. Then, it is easy to show that ∀a ′ ∈ active arms by tEQUATIONThis lower bound corresponds to the case where no other arm is eliminated before eliminating a. Moreover, we have an upper bound for n ta (a) considering the worst-case scenario in which the only remaining arms are a and a c , where a c is the optimal action for unobserved context c:EQUATIONNow, let Reg c (a) be the total regret contributed by the arm a for a given context c ∼ C. We can upper bound the regret asReg c (a) = n ta (a) (V c (a c ) -V c (a)) (i) ≤ 2n ta (a) log (4T 4 K/δ) 2n ta (a) + log (4T 4 K/δ) 2n ta (a c ) ≤ 2 p(a) p(a) + p(a c ) • t a log (4T 4 K/δ) 2n ta (a) + log (4T 4 K/δ) 2n ta (a c ) By (21) = 2log (4T 4 K/δ) • p(a) p(a) + p(a c ) • t a 1 n ta (a) + 1 n ta (a c ) ≤ 2log (4T 4 K/δ) • p(a) p(a) + p(a c ) • t a 1 t a p(a) + 1 t a p(a c ) By (20) = 2t a log (4T 4 K/δ) • p(a) p(a) + p(a c ) 1 p(a) + 1 p(a c ) (ii) ≤ 2T log (4T 4 K/δ) • p(a) p(a) + p(a c ) 1 p(a) + 1 p(a c ) ,where (i) holds since the confidence intervals of arm a and a c overlap at episode t a (otherwise, a would have been eliminated before t a ), and (ii) follows from the fact that t a ≤ T .Finally, we upper bound the Bayesian regret by taking the expectation of a̸ =ac Reg c (a) over c ∼ C. Note that since the expert is optimal, we have p(a) = µ ⋆ (a c = a) for all k ∈ A. is the history of online interactions, D E is the expert demonstration data, β is the competence level of the expert in Assumption 1, and {α ⋆ τ } τ ∈DE are derived from Proposition 1.EQUATIONRemark. We note that, in principle, the ExPerior framework allows for context-dependent transition functions. In this case, the log-pdf in (22) provides an optimistic upper bound on the true posterior log-pdf function. See Hao et al. [23] for a similar analysis. We leave the general case for future work. Note that the second term of ( 22) is simply the log-pdf of the max-entropy prior.Proof. Since the transition function is context-independent, the likelihood of an expert trajectory τ E can be simplified as:EQUATION)The second term in (23) is constant in c. This implies that the likelihood function m τE (c) will depend on c only through the expert policy, which itself is a function of optimal Q-functions by Assumption 1. Note that the second term in the definition of m τE can be simply removed since we can re-weight the parameters α in the optimization step (3) of Proposition 1. Hence, assuming the deep Q-network is expressive enough, without loss of generality, we can re-define the likelihood function of an expert trajectory τ E = (s 1 , a Now, given the Bellman equations, we can write the mean value of the reward function as ∀s ∈ S, a ∈ A : E [R (s, a ; θ)] = Q (s, a ; θ)maxa ′ ∈A E s ′ [Q (s ′ , a ′ ; θ)]The reward distribution is Gaussian with unit variance. Therefore, ∀s ∈ S, a ∈ A, r ∈ R : R (r | s, a ; θ) = N Q (s, a ; θ)maxEQUATIONMoreover, by Proposition 1, the log-pdf of the maximum entropy expert prior is given asEQUATIONCombining ( 24) to ( 26), we conclude the proof. where β is the competence level of the expert in Assumption 1.Proof. Recalling (1), the log-likelihood of the expert trajectories D E under µ θens is given by Although this bound is worse than the standard successive elimination by a factor of K, our empirical results show that ExPerior is still on par with standard regrets in the non-informative cases. 


Decision: The paper studies sequential decision-making in the presence of expert demonstrations based on unobserved contextual information. The reviewers have identified some valuable aspects of the paper:

- Well-motivated problem
- Good presentation

However, they have also noted several concerns:

- Assumption 1 may be very limiting
- Limited experimental evaluation in terms of environments and a lack of comparison with relevant baselines
- The conditions under which performance improvement is possible (as outlined in Theorem 2) are quite extreme (the expert always plays the optimal action).

Reviews: review: paper_summary: paper_summary: This paper studies online transfer learning in an unknown Markov decision process. The learner has access to demonstration trajectories generated by an expert, who has access to an independent context variable. The expert can observe the values of the latent context to make a decision, but such information is unobserved to the learner. The authors propose a learning strategy that allows the learner to accelerate its online learning process by leveraging the confounded expert's trajectories. More specifically, the expert selects values of the action following a specific parametric family of sofmax policy. Using this parametric information, the learner can extrapolate an informative prior about the underlying system dynamics (i.e., transition function and reward function) from the confounded trajectories and utilize the extrapolated prior to improve future learning. The authors then derive a Bayesian regret bound for the proposed method. Simulation results support the proposed approach.
strength_weakness: strength_weakness: - The proposed method requires the expert to follow a specific form of policy to generate the demonstration data. This parametric restriction is rather strong and may not necessarily hold in many applications.
- It is unclear to see from Theorem 2 under which condition the expert's demonstration could accelerate the online learning process. It would be appreciated if the authors could further elaborate on this point.
- Some related references are missing. Extrapolating knowledge from confounded data is one of the main problems in causal inference. There is a growing line of work studying learning from confounded demonstration data in canonical RL tasks, including:

1. Kallus, Nathan, and Angela Zhou. "Confounding-robust policy improvement." Advances in neural information processing systems 31 (2018).
2. Zhang, Junzhe, and Elias Bareinboim. "Near-optimal reinforcement learning in dynamic treatment regimes." Advances in Neural Information Processing Systems 32 (2019).
questions: questions: - Could the regret bound in Theorem 2 outperform standard bandit regret when the expert's demonstration is benign? Could the authors elaborate on this?
limitations: limitations: The authors have adequately addressed the limitations of the paper.
paper_summary: paper_summary: This paper attempt to leverage offline demonstrations to speed up online learning under unobserved heterogeneity, and unlike zero-shot meta reinforcement learning, the proposed ExPerior does not require the task labels (reward labels). ExPerior utilizes expert data to establish an informative prior distribution for online exploration. The experimental results on multi-armed bandits and MDPs showcase the superiority against current baselines.
strength_weakness: strength_weakness: 1.	Just as stated at the end of the paper, the experiments conducted in the paper are limited, and the experiments under MDPs even made the assumption that the transition functions are invariant to the task variables, which makes the experimental environment overly simple. In more difficult experimental benchmarks such as MuJoCo, it is questionable whether the proposed method can show the expected effect. 
2.	Due to the deficiency mentioned in the first point just now, the paper also cannot further compare with modern offline meta RL algorithms (such as PEARL, FOCAL, CORRO) under complex MDPs, making the baselines seem a bit scarce. 
3.	The introduction of related concepts in the paper is slightly insufficient. Besides, there is a clerical error in the upper right part of Fig. 1: the first \mu_{\rm{ME}} should be \mu_{\theta^*}.
questions: questions: 1.	What does "Heterogeneity" mentioned in the paper represent? Does it refer to the differences between the transition functions and reward functions in MDPs? The paper should explain this.  
2.	The paper mentions that the off-policy meta RL method PEARL requires task labels. Does the task label here refer to the reward label? In fact, the trajectories required by PEARL only contain (s, a, r, s') information.
limitations: limitations: A/N
paper_summary: paper_summary: The paper proposes a 2-stage learning method for sequential decision making, wherein step 1 involves leaning a prior from expert demonstrations and step 2 is online RL where that learnt prior is used. The main problem being addressed is that of heterogeneity of data and/or contexts provided by the expert and those encountered by the online learning agent.
strength_weakness: strength_weakness: Clarity
- Notation: even with the notation paragraph, I find it quite difficult to follow. E.g. I think some `t` superscripts are missing (the definition of the q-function); should $\mathcal{C}$ be part of the definition of the MDP?
- Terminology: I'm a little confused with the terminology "parametric" vs "non-parametric". Unless I'm missing something, the latter is still parametric as you are optimising over a family of distributions?
- Typo: in figure 1, annotation below "Step 3" is repeated, I guess one should be $\mu_{\theta}*$.
- Figure 1: clarity can be improved if you write down what is being learnt and what is being updated.
- I found the algorithm box quite useful, will be great to have it in the main paper (e.g. next to Figure 1).

Max-entropy prior:
- Clearly, if $\mu_0  \in \mathcal{P}(\epsilon)$, then we get a trivial solution, $\mu_{ME}=\mu_0$ (usually Uniform), so do you need an assumption on ${P}(\epsilon)$ to get a non-trivial solution? 
- If $\mathcal{C}$ is unbounded, then you can't really define a uniform distribution on it, how do you deal with that?
- Can you handle improper priors?
questions: questions: In addition to those raised in the Limitations section:

1. Do you have an intuition on how badly "misspecified" the learnt expert prior can get until learning for the online agent becomes very difficult? Is there a way to detect a poorly specified prior, and potentially reverting to an alternative (e.g. uninformative) after a few steps online?
2. For the "parametric" ExPrior - why score gradients? Have you thought about or tried continuous relaxation when the problem is discrete?
3. On line 112 "Note that since the task variable is unobservable, the learner's policy will not depend on it" - does it depend implicitly though, e.g. can the learner infer and integrate over that inferred task distribution?
limitations: limitations: There are no specific negative societal impact that need to be discussed.
paper_summary: paper_summary: Imitation learning in a setting where experts have contextual information not available to the imitator. Imitator sees state-action (no reward) trajectories for the expert data. The paper proposes to use the expert data as prior and then do meta-RL to learn a policy.
strength_weakness: strength_weakness: - The empirical evaluation is limited to simple domains and it is not discussed why that is the case. It would be good to include an explanation on why this was not scaled up to some more complex environments.
- Comparison to ExPLORe: if there is a way to use the same actor-critic algorithm for the proposed method and the comparison that would strengthen the paper. As the text says, now it is not clear whether the difference comes from the online RL part or from the difference in how they use the expert data.
- ADVISOR [1] targets the same problem setting. Differences to that algorithm should be discussed. And maybe a pointer should be given on why their method is demonstrated in high-dimensional domains and this isn't. Not saying that this has to be evaluated in high-dimensional domains, but that it is not immediately clear why this isn't.
- Figure 4
    - Explain the pane meanings explicitly in the caption.
    - Explain the axes explicitly in the caption. I.e., are these learning curves?
    - How come the naive baseline does worse in the low entropy setting than in the high entropy one?

[1] Luca Weihs, Unnat Jain, Iou-Jen Liu, Jordi Salvador, Svetlana Lazebnik, Aniruddha Kembhavi, and Alex Schwing. Bridging the imitation gap by adaptive insubordination. Advances in Neural Information Processing Systems, 34:19134–19146, 2021.
questions: questions: - Why is human-in-the-loop highlighted as a limitation?
- Can you give a more fleshed out example of how the problem of unobserved contextual information might appear in a realistic problem?
- Can you lay out an idea on how to scale the method?
limitations: limitations: Limitations are discussed but some of the limitations highlighted seem a little strange. See questions.

Comments: comment: I appreciate the authors' response. They have answered my questions. Theorem 2 states that the proposed algorithm is able to achieve performance improvement when the expert always picks the optimal action in demonstrations. This is somewhat expected since, in this case, the latent is not effective, and the demonstration data could be directly transferred. On the other hand, this is quite an extreme condition that might be hard to reach in practical conditions. I will keep my current score.
Thank you for your response, I maintain my original score.
Thank you for the response. Most of my concerns have been addressed. I'm raising my score.