Blending Imitation and Reinforcement Learning for Robust Policy Improvement
Authors: Xuefeng Liu
Takuma Yoneda
Rick Stevens
Matthew Walter
Yuxin Chen
Venue: ICLR 2024 spotlight (2024)
Keywords: imitation learning
reinforcement learning
multiple experts
Primary area: reinforcement learning

Abstract: While reinforcement learning (RL) has shown promising performance, its sample complexity continues to be a substantial hurdle, restricting its broader application across a variety of domains. Imitation learning (IL) utilizes oracles to improve sample efficiency, yet it is often constrained by the quality of the oracles deployed. To address the demand for robust policy improvement in real-world scenarios, we introduce a novel algorithm, Robust Policy Improvement (RPI), which actively interleaves between IL and RL based on an online estimate of their performance. RPI draws on the strengths of IL, using oracle queries to facilitate exploration—an aspect that is notably challenging in sparse-reward RL—particularly during the early stages of learning. As learning unfolds, RPI gradually transitions to RL, effectively treating the learned policy as an improved oracle. This algorithm is capable of learning from and improving upon a diverse set of black-box oracles. Integral to RPI are Robust Active Policy Selection (RAPS) and Robust Policy Gradient (RPG), both of which reason over whether to perform state-wise imitation from the oracles or learn from its own value function when the learner’s performance surpasses that of the oracles in a specific state. Empirical evaluations and theoretical analysis validate that RPI excels in comparison to existing state-of-the-art methodologies, demonstrating superior performance across various benchmark domains.

Published as a conference paper at ICLR 2024
BLENDING IMITATION AND REINFORCEMENT LEARN- ING FOR ROBUST POLICY IMPROVEMENT
Xuefeng Liu'; Takuma Yoneda', Rick L. Stevens', Matthew R. Walter", Yuxin Chen' 'Department of Computer Science, University of Chicago Toyota Technological Institute at Chicago
ABSTRACT
While reinforcement learning (RL. has shown promising performance, its sample complexity continues to be substantial hurdle, restricting broader application across a variety of domains. Imitation learning (IL) utilizes oracles to improve sample efficiency, yet it is often constrained by the quality of the oracles deployed. To address the demand for robust policy improvement in real-world scenarios, we introduce a novel algorithm, Robust Policy Improvement (RPI), which actively online estimate of their performance.
interleaves between IL and RL based RPI draws on the strengths of II,, using oracle queries to facilitate exploration-an aspect that is notably challenging in sparse-reward RI- h1--particulaalyy during the early stages of learning. As learning unfolds, RPI gradually transitions to RL, effectively treating the learned policy as an improved oracle. This algorithm is capable of learning from and improving upon diverse set of black-box oracles. Integral RPI are Robust Active Policy Selection (RAPS) and Robust Policy Gradient (RPG), both of which reason over whether to perform state-wise imitation from the oracles or learn from its own value function when the learner's perfor- mance surpasses that of the oracles in specific state. Empirical evaluations and theoretical analysis validate that RPI excels in comparison to existing state-of- the-art methods, showing superior performance across various domains. Please checkout our website'.
INTRODUCTION
Reinforcement learning (RL) has shown significant advancements, surpassing human capabilities in diverse domains such as Go (Silver al., 2017), video games (Berner et al., 2019; Mnih et al., 2013), and Poker (Zhao et al., 2022). Despite such achievements, the application of RI. is largely constrained by its substantially high sample complexity, particularly in fields like roboties (Singh et al., 2022) and healthcare (Han et al., 2023), where the extensive online interaction for trial and error often impractical.
Imitation learning (IL) (Osa et al., 2018) improves sample efficiency by allowing the agent to replace some or all environment interactions with demonstrations provided by an oracle policy. The efficacy of heavily relies access to oear-optimal oracles for approaches like behavior cloning (Pomerleau, 1988; Zhang et al., 2018) or inverse reinforcement learning (Abbeel and Ng, 2004; Finn et al., 201 6; Ho and Ermon, 2016; Ziebart et al., 2008). Interactive II. techniques, such as DAgger (Ross et al., 2011) and Aggre Vate()) (Ross and Bagnell, 2014; Sun et al., 2017), 7), similarly assume that the policy we train (i.e., learner policy) can obtain demonstrations from a near-optimal oracle. When we have access to rewards, the learner has the potential to improve and outperform the oracle. THOR (Sun et al., 2018) exemplifies this capability by utilizing a near-optimal oracle for cost shaping, optimizing the k-step advantage relative to the oracle's 's value function (referred to as "cost-to-ge oracle").
However, realistic settings, obtaining optimal or near-optimal oracles is often infeasible. Typically, learners have access to suboptimal and black-box oracles that may not offer optimal trajectories or quantitative performance measures varying states, requiring substantial environment interactions
"Correspondence to: Xuefeng Liu cxucfeng@uchicagee @uchicago.chus https:/mooostt-plieeympveeeeeee githuu

---

Published as a conference paper at ICLR 2024
to identify state-wise optimality. Recent approaches, including LOKI (Cheng al., 2018) and TGRL (Shenfeld et al., 2023) aim to tackle this issue by combining IL and RL. They focus on a single-oracle setting, whereas MAMBA (Cheng et al., 2020) and MAPS (Liu et al., 2023) learn from multiple oracles. These approaches demonstrate some success, but often operate under the assumption that at least one oracle provides optimal actions in any given state, which does not always hold in practice. In situations where no oracle offers beneficial advice for specific state, it is more effective learn based on direct reward feedback. Our work intend to bridge this gap by adaptively blending IL and RL in a unified framework.
Our contributions. In this paper, we present max', a learning framework devised to enable robust learning in unknown Markov decision processes (MDP) by interleaving RL and IL, leveraging multi- ple suboptimal, black-bo box oracles. Within this framework, we introduce Robust Policy Improvement (RPI) a novel policy gradient algorithm designed to facilitate learning from a set of black-box oracles. RPI comprises two innovative components:
1. Robust Active Policy Selection (RAPS), improving value function estimators of black-box oracles efficiently, and
2. Robust Policy Gradient (RPG), executing policy gradient updates within an actor-critic framework based a newly devised advantage function.
Our algorithm strikes a balance between learning from these suboptimal oracles and self improvement through active exploration in states where the learner has surpassed the oracle's s performance. We provide a theoretical analysis of our proposed method, proving that it ensures performance lower bound no worse than that of the competing baseline (Cheng et al., 2020). Through extensive empirical evaluations on eight different tasks from DeepMind Control Suite (Tassa et al., 2018) 8) and Meta-World (Yu et al., 2020), we empirically demonstrate that RPI outperforms contemporary methods and then ablate its components.
2 RELATED WORK
Online selection of suboptimal experts. CAMS (Liu et al., 2022b;a) learns from multiple subopti- mal -bxck-box&& experts to perform model selection based on a given context, but is only applicable in stateless online learning settings. Meanwhile, SAC-X (Riedmiller et al., 2018) 1) learns the intention policies (oracles), each of which optimizes their own auxiliary reward function, and then reasons over which of these oracles to execute as a form of curriculum learning for the task policy. LfGP (Ablett et al., 2023) combines adversarial IL with SAC-X to improve exploration. Defining auxiliary rewards requires the task to be decomposed into smaller subtasks, which may not be trivial. Further, they query the intention policies several times within a single episode. Unlike CAMS and SAC-X, which rely on selecting expert policies to perform sub-tasks, our approach trains an independent learner policy. It acquires expertise from sub- sub-optimal experts using only a single oracle query per episode, thus having the potential to surpass these oracles through global exploration.
Policy improvement with multiple experts. Recent works attempt to learn from suboptimal black-box oracles while also utilizing rewards observed under the learner's policy. Active offline policy selection (A-OPS) (Konyushova al., 2021) utilizes policy similarities to enhance value predictions. However, A-OPS lacks a learner policy to acquire expertise from these offline policies. ILEED (Beliaev et al., 2022) distinguishes between oracles based on their expertise at each state but is constrained to pure offline IL. settings. InfoGAIL (Li et al., 2017) conditions the learned policy on latent factors that motivate demonstrations of different oracles. OIL (Li et al., 2018) tries identify and follow the best oracle in a given situation. SFQL (Barreto et al., 2017) proposes generalized policy improvement with successor features. MAMBA (Cheng et al., 2020) utilizes an advantage function with geometric weighted generalization and achieves a larger policy improvement over SFQL, while addressing the above two important questions with theoretical support. MAPS (Liu et al. 2023) improves on the sample efficiency and performance of MAMBA by proposing active policy selection and state exploration. However, even when the quality of the oracle set is poor, these algorithms will still resort to imitation learning with the inferior oracles. In contrast, our algorithm performs self improveeett employing imitation learning only on states for which an oracle outperforms the learner.
2

---

Published as a conference paper at ICLR 2024
3 PRELIMINARIES
We consider a finite-horizon Markov decision process (MDP) M, A.,, with state space S, action space A, unknown stochastic transition dynamics P S A A(S), unknown reward function r x A + [0, 1], and episode horizon H. We define total number of training steps (rounds) N and assume access to (possibly empty) set of K oracles, defined II {m"]] Jk-1 where TT : :S + (AA) The generalized Q-fumction with respect to a general function f S -+Ri defined as:
Q' (s, r - Es--ps,,af (s')]
When f() is the value function of a particular policy я, the generalized Q-function can be used to recover the policy's Q-function Q" to f as
(s, a). We denote the generalized advantage function with respect
A (s, Q' (s, f() (r, B. (s')) (s).
Given an initial state distribution do E (()), let dā denote the distribution over states at time t under policy The state visitation distribution under can be expressed as d #1 DE di The value function of the policy ı under dg is denoted as:
v (do) Eo V (s)
E so Bro 880
Σ r=
(St. (st. ar)
where p" (Te st) the distribution over trajectories T; = {st, at...., -H-1, a= under policy π. The goal find policy arg max, J (x) maximizing the expected return
(TT) Eg [V" (s)].
(1)
POLICY IMPROVEMENT WITH PERFECT KNOWLEDGE OF ORACLE SET
We now present reinforcement learning framework in the presence of imitation learning oracle set, which is inspired from Cheng et al. (2020); Liu et al. (2023). In this section, we assume that we have perfect knowledge of the underlying MDP and each oracle's value function. We will relax these assumptions in the next section.
Max-following. Given collection of As imitation learning oracles IIo - (= ]E(K)) the max- following policy is greedy policy that selects the oracle with the highest expertise in any given state. The max-following policy sensitive to the quality of the oracles. Specifically, if all oracles perform worse than the learner policy at given state, the max-following policy will still naively imitate the best (but poor) oracle. Instead, would be more prudent to follow the learner's guidance these cases.
Definition 4.1. (Extended Oracle Set). Let II = {n" { _I = { J be the learner's policy class, where denotes that the policy has been updated for rounds. We define the extended oracle set at the n-th round as
]E[K] be the given black- oracle set,
ETE = IT U { (
Remark 4.2. The learner policy in the extended oracle set is updated at each round.
4.1 MAX* AGGREGATION
Based on the extended oracle set, we first introduce the advantage function A and the baseline value function ft follows:
Definition 4.3. (A* Advantage Function). Given i oracles n¹ T'.... T and the learner policy an we define advantage function as
A" (s, == (s, a) + E. (s')] - ft (s),
(3)
where f (s) is the baseline value function, defined as:
max 1 (s), where [V*] RE[[TT]] == |va' ,..., Dark Vun
f' (s)
kE[[nT]]
3

---

Published as a conference paper at ICLR 2024
f (s) focuses exclusively on optimizing oracle selection for single state, assuming that the selected policy will be followed for the remainder of the trajectory. To optimize the oracle selection for every encountered state, we introduce the max max--follownng policy, which acts greedy policy, adhering to the optimal policy within the extended oracle set for any given state.
Definition 4.4. (Max*-Follae -Following Policy). Given extended oracle set II the max" -followig policy
T (a | s) == Tha (a 8), where k' := arg max V ( K K VVn. (5)
k
Proposition 4.5. Following x² is as good or better than imitating the single-best policy in IIf
With a slight abuse of notation, we use A ()) to denote the generalized advantage function of the policy л 8. As proved in the Appendix, the function A" (s, TT 2 (Appendix C.I) and that the value function for the max* maxt-following policy satisfies V= (s) 2 f (s) = max (s) (Appendix C.I). This indicates that following T" is at least as good as or better than imitating single best policy in ITe Thus, TO is a valid approach to robust policy learning in the multiple oracle setting. The max -following policy H² is better than the max-following policy when the learner's 's policy is better than any oracle for a given state. On the other hand, when the value of specific oracle V always better than all other policies for all states, T simply reduces to the corresponding oracle x². This is not ideal because the value (s) assumes to keep rolling out the same oracle π from state s until termination, without making improvement by looking one step ahead and searching for better action. To address this, we propose the max" muuu"-aggeggtton policy as follows.
Definition 4.6. (Max* -Aggregation Policy"). For state s, the max" aggregattion policy T performs one-step improvement and takes the action with largest advantage over f",
(a) s) Sa=ar, where a' arg maxA" (s, a) and ô is the Dirac delta distribution. aEA
(6)
Although the max -following policy x improves upon the max-following policy, it does not perform self-imprrvemmnt. In contrast, the max' -aggregation policy σ0 looks one step ahead and makes the largest one-step advantage improvement with respect to f*. Thus, in the degenerate case where x² is equivalent to the single best policy, T outperforms the best single policy in 11tt for all states, Since A (s, T0) A (s, )T) 2 0 for any state s by Corollary C.2 and Proposition 4.5, we conclude that the max aggregattion policy π is a suitable policy benchmark for the robust policy learning setting as well. We note that the baseline (*)(s corresponds the value of choosing the single-best policy in II at state 8 and rolling it out throughout the rest of the episode. In contrast, π²" and π 3 optimize the oracle selection every remaining step in the trajectory. This work is therefore built .
Remark 4.7. (Empty Oracle Set) Up to this point, we have primarily assumed non-empty oracle set TI and an extended oracle set of size |T| 2 2. Given an empty oracle set IIº, I will only contain the learner policy. In this case, f = V and x will not improve, while @ reduces to pure reinforcement learning, performing self- eelf-improvement by using the advantage function
5 ROBUST POLICY IMPROVEMENT WITH BLACK-BOX ORACLE SET
Improving a policy from the max* t baseline f (Eqn. 4) is the key to learning robustly via IL and RL. This requires knowledge of the MDP and the oracles' value functions, however, the oracles are presented to the learner as black-box policies with unknown value functions.
A critical challenge use I (s) max V (s) baseline is that it changes as training goes. whereas MAMBA assumes a static baseline function. In the following analysis we resort to a slightly weaker baseline, f == max H / (s), where m < N is an intermediate step in the learning process, and N is the total number of rounds. Similarly, we define A (s,a) := r as + Es (s')] - f (s), as the corresponding advantage function, and TM as the corresponding max* -aggregation policy by setting A = A in Definition 4.6. In the following, we use the baseline value fi.. and reformulate the problem in an online learning setting (Ross et al.. 2011; Ross and Bagnell, 2014; Sun et al., 2017; Cheng et al., 2020; Liu et al., 2023) for black-box oracles. Following MAMBA's analysis, we first assume that the oracle value functions are known but the MDPi is unknown, followed by the case that the value functions are unknown.
When we exclude the learner's policy from the extended oracle set, this reduces to the max-aggregation policy, which was used in MAMBA (Cheng et al., 2020).
4

---

Published as a conference paper at ICLR 2024
Unknown MDP with known value functions. If the MDP is unknown, can regard d as an adversary in online learning and establish the online loss for round n as
ln (m) = - -- Eaw [A (s, a).
Lemma C.1 and Proposition 4.5 suggest that making &n (TT) small ensures that V (do) achieves better performance than fmm (do) for n. Averaging over N rounds of online learning, we obtain
1 N
Σ N
V (do) = Jm (do) + AN EN (II²) Regrett
where Regrety = N (En= lu (Tn) - -mmiin E En= Pn (7)) depends the learning speed of an online algorithm, AN 1 Σ N (T)) is the loss of the baseline max' -aggregation policy mm
and (II == min N (E (T) Σn en (TTT)) expresses the quality of oracle class, where _1 is specified in Definition 4.1. I
m E -1 we have eN (1I O. Otherwise, e (11 > 0. By Proposition 4.5, A (s, m 2 0 and, in turn, A 2 O. If π E _1f, using no-regret algorithm to address this online learning problem will produce policy that achieves performance of at least Egwdo f (s)] + AN + O(1) after iterations.
Unknown MDP with unknown value function. practice, the value functions of the oracle set are unavailable. ' and A* need to be approximated by I and Á We compute the sample estimate of the gradient as follows:
Vln (Tn) = -aa))==EB--aae HEn-dd Eaa EasT,
V Flggm n. (a A' (s, as.a).
The approximation of the value function and gradient introduces bias and variance terms in the online learning regret bound Regrety, We propose general theorem to lower bound the performance:
Proposition 5.1 (Adapted from Cheng et al. (2020)). Define AN. eN (II)), fin and Regrer as above, where fm == melxe(l V (8) for m I N, and Regretf corresponds to the regret of first-order online learning algorithm based on Eqn. 9. It holds that
{ Rswdd (s (s)] + E [AN -NN (TTF) Regret C)
E
max *Ta (do) nE(N]
where the expectation is over the randomness in feedback and the online algorithm.
Remark 5.2. W.L.o.g. we assume (s) =- arg [a] V (s), which corresponds to the baseline function considered in MAMBA. Note that f (s) admits a weaker baseline value than f (s) for m < n. but no weaker than the max value of any oracle, fö (s). Therefore, as the learner improves (s), max max' aagrregation will have an improved lower bound over Cheng et al. (2020). Consider scenario where m = o(N). In round m, we instantiate f and perform I-step advantage improvement over / Since f > fo (s) when (+) > [ú(()) (s), ~ d=nn, we can view max -aggregation as adding improved learner policies into II at the end of each round and perform 1-step improvement over f on the expending oracle set. As Eg f (s)) improves, will lead to the improvement over the original bound in Proposition 5.1.
ROBUST POLICY IMPROVEMENT VIA ACTIVELY BLENDING RL AND IL
In this section, we present RPI, an algorithm for robust policy improvement that builds upon the max* -agrregation policy. RPI consists of two main components: Robust Active Policy Selec- tion (RAPS) and Robust Policy Gradient (RPG) that enable the algorithm to combine the advantages of reinforcement and imitation learning.
6.1 ROBUST ACTIVE POLICY SELECTION
To improve the sample efficiency in learning from multiple oracles and lower the bias in Regret in Proposition 5.1 caused by the approximator of the max" baseline function ft, we propose a robust active policy selection strategy. We employ ensemble of prediction models to estimate the value
5

---

Published as a conference paper at ICLR 2024
Algorithm 1 Robust Policy Improvement (RPI)
Input: Learner policy T1, oracle set II = (Tt frrh KEjAy function approximators {V+ ]EE[K] Vn.
Output: The best policy among {TT,,.., TT).
1: for n = .,...., N do
2:
Construct an extended oracle set IT = _ n. Tn ke[mT]]
3: Sample le E [H - uniformly random.
4: Roll-in Ta up to tg, select kt (Eqn. 10), and roll out ~h to collect the remaining data D
5: Update 17 using D*
6:
Roll-in Tn for full H-horizon to collect data D',-
7: Update Vn using D/=-
8: Compute advantage À GAE+ (Eqn. 11) and gradient estimate 9n. (Eqn. 14) using D
9: Update n. to Tn+1 by giving In to first-order online learning algorithm.
function for a policy (Liu et al., 2023), where we estimate both the mean 1/ (s) and the uncertainty oh (s) for a particular state s. We generate a few independent value prediction networks that are initialized randomly, and then train them using random samples from the trajectory buffer of the corresponding oracle **
In the single oracle case, the motivation of rolling in learner policy and rolling out oracle policy (referred to RIRO) in prior work (e.g.. DAgger, AggrevateD)
address the distribution shift. In our work, in addition addressing distribution shift, we aim to improve the value function estimator V of the most promising oracle on the switch state & to reduce the bias term of f Moreover, we seek reduce the roll-out cost associated with querying oracles, particularly when the learner exhibits higher expected value for the switching state. In such cases, we roll-out the learner to collect additional data to enhance its policy. We achieve this goal by comparing the UCB of oracle policies' value function and LCB of learner policy to improve the estimation of f*. We design the strategy follows:
Let (s)s) (s) + (s)(s, (s) function for state s, respectively. We obtain the best oracle n² for state as follows:
(s) (s))) be the UCB and LCB of policy k's value
k. arg max 1i (s), 122 (s) VK (s) V kK+ (s)
(10)
[E[[ITE]]
where px++1 is the confidence-aware value function approximator for the learner's policy, while [V*] KE[A] represents the value function approximators associated with oracle policies.
Remark 6.1. The insight behind using confidence-aware policy selection strategy in RAPS is improve the estimate of the value function of the most promising oracle given state. This necessitates accounting for estimation uncertainties, which leads to the adoption of UCB-based approach to identify the optimal oracle. Using LCB for the learner encourages gracle-guided exploration unless we are certain that the learner surpasses all oracles for the given state. We empirically evaluate this in Section 7.2.
Remark 6.2. MAPS (Liu al., 2023) introduced active policy selection strategy by selecting the best oracle to roll out and improve the value function approximation on state 8t according to fo In this work, we empirically improve such strategy by utilizing the learner policy in _1.
6.2 ROBUST POLICY GRADIENT
We now propose robust policy gradient based on a novel advantage function, denoted by A GAE+ and novel max actor-critic framework.
A GABi advantage function. The policy gradient methods maximize the expected total reward by repeatedly estimating the gradient g E H-1 rv]. The policy gradient has the form g E, [Vglog Ta (a.])))A.] (Sutton et al., 1999; Greensmith et al., 2004; Schulman et al., 2015; 2017),, where Ty is a stochastic learner policy and Å, is an estimator of the advantage function at timestep and [[]]] indicates the empirical average over finite batch of samples, for algorithm that

---

Published as a conference paper at ICLR 2024
Dense reward tasks
Sparse reward tasks
Figure I:
We consider 8 tasks from DeepMind Control Suite and Meta- World. Extended results on different variants of these tasks provided in Appendices E.2 & E.3.
alternates between sampling and optimization. A, measures whether the action better or worse than the current policy. Hence, the gradient term Volog To (at (&x) A, points in the direction of increased To (ar(se) if and only if Ar Á (st, az) > 0. For A, we propose a novel advantage function A GAE based on general advantage estimation (Schulman et al., 2015), the max' baseline f' and the A" advantage function (3).
A, GAE(7,A)+ 8, (7 8 + + 87-1, where 8, ra + (8++1) (st), (11)
where T < H, and y and are the predefined parameters that control the bias-variance tradeoff. In this work, we use 0.9 and 7 1, and thus simplify A,
GAE().A)= as A,
GAB+
We propose variant of the max' baseline f' that includes confidence threshold I for an oracle's value estimate:
Th #
(s) if oh (s) > s,, where arg ke[ V (s).
ft (s)
(12)
max V (s), otherwise.
Remark 6.3. We use threshold to control the reliability of taking the advice of oracle, where lower value indicates greater confidence. In our experiments, we use I', 0.5, which we have found to exhibit robust behavior (Appendix E.5).
Finally, we have the n-th round online loss as
é, (mT) = -H BeatrrWeang Bawals
[
GAB (s, |==an!
(13)
and gradient estimator
On Vln (mT) HE Eawarss
Vlog (a s)A, (s, T=N.
GAE+
(14)
Max* actor-critic. We note that the RPG component (Algorithm 1, lines 8-9) can be viewed as variant the actor-critic framework, with the actor sampling trajectories that are then evaluated by the maxt critic based on the A GAE+ advantage function (11). The policy gradient in Eqn. 14 enables the learner policy T. to learn from high-performing oracles and to improve its own value function V for the states in which the oracles perform poorly.
Remark 6.4. When 7 1, Eqn. 11 disregards the accuracy of f', but has high variance due the sum of the reward terms. When = O, it introduces bias, but has much lower variance. Moreover, when À = 0 and 7 1, the loss (13) of RPI reduced the loss (7) under max max' -aggregation (6). and the performance bound for the max -aggregation policy and RPI will be equal. Thus, performing no-regret online learning with regards to Eqn. 13 has the guarantee in Proposition 5. and Remark 5.2. However, when À > O, RPI will optimize the multi-stepe advantage over f in Eqn. 13, while the max" max" -ggregatton policy @ only optimizes the one-step advantage over ft. Thus, RPI will have smaller e (112) term than max -aggregation, which improves the performance lower bound in Proposition 5.1.
Imitation, Blending and Reinforcement. Instances of f* Eqn. 11 may involve combination of oracles' and learner's value functions. In a case that this does involve the learner's value function-this is likely in the early stage of training since the learner's performance is Poor-RP1 performs imitation learning on the oracle policies. Once the learner policy improves and f* becomes identical to the learner's value function, RPI becomes equivalent the vanilla actor-critic that performs self-improvement. When it is combination of the two, RPI learns from blending of the learner and oracles.
7

---

Published as a conference paper at ICLR 2024
100
MAPS
Best poturns
Best return
DMC Stite
200
Best
Best
0
56
Training step
Training step
Training step
Training Step
Cheetah (dense)
(b) Walker-walk (dense)
(c) Caripole (dense)
(d) Pendulum (sparse)
MAPi NAMBA
Mett World
Best returry
ar
Best retum
relum
Best relurn
300
40
Training step
Training step
Truming step
Training step
(0) Paneet-open (dense)
(f) Drawer-close (dense)
(g) Buttonn-peass (sparse)
(b) Window-close (sparse)
Figure 2: Main results. A comparison between RPI with five baselines and the best oracle (horizontal line) on Cheetah, Cartpole, Pendulum, and Walker-walk from DMC; and Window-close Button- press, Faucet-open, and Drawer-close from Meta-Wort in terms of best-return with three diversified oracles. The shaded area represents the standard error over five trials. RPI scores the best in all benchmarks.
7 EXPERIMENTS
Environments. We evaluate our method on eight continuous state and action space domains: Cheetah-run, CartPole- CartPole-singuup. Pendulum-swingup. and Walker-walk from the DeepMind Control Suite (Tassa et al., 2018); and Window-close, Faucet-open, Drawer-close and Button-prest from Meta-World (Yu et al., 2020). In addition, we conduct experiments a modified sparse reward Meta-World environment, which considered be a more challenging task. We set ensemble size of value functions to five. Appendix D provides further details.
Oracles. We implement our oracles as policies trained using PPO (Schulman et al., 2017) with generalized advantage estimate (GAE) (Schulman al., 2015) and SAC (Haarnoja et al., 2018). We save the policy weights at different points during training to achieve oracles that perform differently in different states. Each environment is provided with three diversified oracles.
Baselines. We compare RPI with five baselines: (1) PPO with GAE as a pure RL baseline; (2) Max- Aggregation (Cheng al., 2020) pure IL baseline (a multiple-oracle variant of (egreVaTe(D)) (3) a variant of LOKI adapted to the multiple-orace oracle setting that initially performs pure IL and then pure RL; (4) MAMBA: (5) MAPS (the current state-of-the- methody"; and also the best oracle in the oracle set as a reference. We matched the number of environment interactions across algorithms'. Appendix D provides further details.
7.1 MAIN RESULTS
Figure 2 visualizes the performance of RPI and the baselines. The results show that RPI surpasses the baselines on all domains, despite variations in the black-box oracle set. Notably, the Ra-based PPO-GAE baseline outperforms the IL methods in the later stages of training in most of the dense reward environments, while based approaches perform better in the sparse reward domains Pendulum-swingup (Fig. 2(d)) and window-close (Fig. 2(h)) are particularly difficult domains that involve non-trivial dynamics and sparse reward (i.e., the agent receives a reward of only when the pole is near vertical, the window is closed exactly). Due to the sparse reward, the IL-based approaches are significantly more sample efficient than the RL-based approach, but their performance plateaus quickly. RPI initially bootstraps from the oracles, and due their suboptimality, it switches to self- eelroimmrveeeent (i.e., learning from its own value function), resulting in better performance than both IL and RL methods. These results demonstrate the robustness of RPI as actively combines the advantages of IL and RI. to adapt various environment.
*Our experimental setup including the oracle differs from that of MAPS. In this work, the learner for all baselines has access to approximately the same numher transitions and the learner does not have access to the oracle's 's trajectory. We reproduce the baseline performance for the MAPS' setting in Appendix E. *PPO has slightly smaller number of interactions due to the lack of oracles' value function pre-training.
8

---

Published as a conference paper at ICLR 2024
Bost return
600
Best returm
Best returm
Best
246
300
Training step
Training step
Training step
Training step
(6) Learning state-wise expertise.
(b) RAPS vs APS.
(c) Confidence-awarc RPG
(d) Confidenso-awarc RAPS
Figure 3: Results of ablation studies on the Cartpole environment.
7.2 ABLATION STUDIES
Learning wise oracle expertise. In Fig. 3(a), we examine the ability of RPI to aggregate the expertise of multiple oracles on Cartpole. We created three diversified oracles and find that RPI achieves return of 645 when it is able query all three oracles, while the best return falls below 600 when given access to only a single oracle. This result demonstrates RPI's utilization of the oracles' state-wise expertise, it achieves better performance when given access to more oracles.
Ablation robust active policy selection. In order to understand the effectiveness of RPI's robust active policy selection strategy (RAPS), we compare it to active policy selection (APS) (Liu al., 2023) (without the learner in RIRO (Algorithm line 4)) Cartpole. Fig. 3(b) shows that RAPS has the advantage of selecting the learner policy to roll out in states for which it outperforms the oracles, resulting in self-improvement. This leads RAPS outperforming the APS-based approach.
in RPI. (/) RPG: We first perform ablation on Cartpole investigate the benefits of using a confidence threshold on an oracle's 's value estimate for RPG (Eqn. 12). We see in Fig. 3(c) that the confidence threshold enables RPG benefit from both state-wise imitation learning from oracles with high confidence and the execution of reinforcement learning when oracles exhibit high uncertainty. Without the threshold, RPG is more vulnerable to the quality of oracle set. (2) RAPS: We then consider the benefits of reasoning over uncertainty to the policy selection strategy, comparing uncertainty-awant
Confidence-awarence
RPI-LCB/UCB (Eqn. 10) to RPI-MEAN, which does not consider uncertainty. Fig. 3(d) demonstrates the benefits of using LCB//CB for policy selection. Addition results in Appendix E.6 reveal that
outperforms RPI-MEAN across all benchmarks by an overall margin of 40%, supporting the advantage of incorporating confidence to policy selection.
Visualizing active IL and RL. Figure 4 visualizes the active state-wise imitation and reinforcement process employed by RPI in the gradient estimator on Pendulum. The figure includes three oracle policies (in blue, orange, and green) and the learner's policy (in red). Each oracle exhibits different expertise at different stages. In the beginning, RPI only imitates the oracles, which initially have greater state-wise expertise than the learner. As the learner improves, the frequency with which RPI samples the leaner policy increases, corresponding to self-improvement. As training continues, the expertise of the learner increasingly exceeds that of the oracles, resulting in RPI choosing self-improvement more often than imitating the oracles.
selection
Training step Figure 4: and RL.
8 CONCLUSION
We present max robust framework for IL and RL in the presence of a set of black-box oracles. Within this framework, we introduce RPI, a policy gradient algorithm comprised of two novel components: robust active policy selection strategy (RAPS) that enhances sample efficiency and robust policy gradient (RPG) for policy improvement. We provide a rigorous theoretical analysis of RPI, demonstrating its superior performance compared the current state-of-the-ar Moreover, empirical evaluations on a diverse set of tasks demonstrate that RPI consistently outperforms all IL and RL baselines, even in scenarios with limited oracle information (favoring RL) or sparse rewards (favoring IL). RPI effectively adapts to the nature of the domain and the quality of the oracles by actively interleaving IL and RL. Our work introduces new avenues for robust imitation and reinforcement learning and encourages future research on addressing more challenging tasks in robust settings, such as handling missing state or oracle information.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank Ching-An Cheng for constructive suggestions. This work is supported in part by the RadBio-Al project (DE-AC02-06CHI1333) U.S. Department of Energy Office of Science, Office of Biological and Environment Research, the Improve project under contract (75N91019F0013) 1233228CCA00000,, DE-AC02-06 01-12-00--H111 BB-CCH11335 1357, DE-AC52-07NA27344, DE- collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration, Al-Assisted Hybrid Renewable Energy, Nutrient, and Water Recovery project (DOE DE- DE-EE0009505),
AC05-000R22725), the Exascale Computing Project (17-SC-20-SC),
and NSF HDR TRIPODS (2216899).
REFERENCES
Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), 2004. 1
Trevor Ablett, Bryan Chan, and Jonathan Kelly. Learning from guided play: Improving exploration for adversarial imitation learning with simple auxiliary tasks. IEEE Robotics and Automation Letters, 1(3):1263- 0211112667777 1270, 2023. 2, 13
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS). 2017. 2
Mark Beliaev, Andy Shih, Stefano Ermon, Dorsa Sadigh, and Ramtin Pedarsani. Imitation learning by estimating expertise of demonstrators. In Proceedings of the International Conference on Machine Learning (ICML), pages 1732-1748, July 2022. 2, 13
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d. O. Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning. arXiv preprint 19Xiv:/912.0660 2019.
Kianté Brantley, Amr Sharaf, and Hal Daumé III. Active imitation learning with noisy guidance. arXiv preprini arXiv:2005. /2800 2020. 13
Ching-An Cheng, Xinyan Yan, Nolan Wagener, and Byron Boots. Fast policy learning through imitation and reinforeement. arXiv preprint 0rXiv:1805.10413. 2018. 2, 13, 16
Ching-An An Cheng, Andrey Kolobov, and Alekh Agarwal. Policy improvement via imitation of multiple oracles. In Advances in Neural Information Processing Systems (NeurIPS), pages 5587-5598, 2020. 2, 4, 5, 8, 13, 14, 15, 16, 17
Hal Daumé, John Langford, and Daniel Marcu. Search-based structured prediction. Machine Learning, 75(3):297-325, 2009.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In Proceedings of the International Conference Machine Learning (ICML), pages 49-58, 2016. 1
Evan Greensmith, Peter I. Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(9), 2004.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the International Conference Machine Learning (ICMI., pages 1861-1870, 2018.
Dong Han, Beni Mulyana, Vladimir Stankovic, and Samuel Cheng, A survey on deep reinforcement learning algorithms for robotic manipulation. Sensors, 23(7):3762, 2023.
10

---

Published as a conference paper at ICLR 2024
Nicklas Hansen, Yixin Lin, Hao Su, Xiaolong Wang. Vikash Kumar, and Aravind Rajeswaran. Modem: Accelerating visual model-basee reinforcement learning with demonstrations. arXiv preprint arXhh:2212.055690 2022. 13
Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. 13
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances Neural Information Processing Systems (NeurIPS), pages 4565 4573, 2016. 1
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning, In Proceedings of the International Conference on Machine Learning (ICML), 2002. 15
Ksenia Konyushova, Yutian Chen, Thomas Paine, Caglar Gulcehre, Cosmin Paduraru, Daniel Mankowitz, Misha Denil, and Nando de Freitas. Active offline policy selection. In Advances in Neural Information Processing Systems (NeurIPS), pages 24631-24644, 2021. 2, 13
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arX:v:2110.06769, 2021. 13
Guohao Li, Matthias Mueller, Vincent Casser, Neil Smith, Dominik Michels, and Bernard Ghanem.
Oil: Observational imitation learning. arXiv preprint arXiv: 1803.01/0 2018 2
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual
demonstrations. Advances in neural information processing systems, 30, 2017. 2
Xuefeng Liu, Fangfang Xia, Rick Stevens, and Yuxin Chen. Cost-effective online contextual model
selection. arXiv preprint (xXxx:2207000033 2022a. 2, 13
Xuefeng Liu, Fangfang Xia, Rick L Stevens, and Yuxin Chen. Contextual active online model selection with expert advice. In Proceedings of the ICML Workshop on Adaptive Experimental Design and Active Learning in the Real World, 2022b.
Xuefeng Liu, Takuma Yoneda, Chaoqi Wang, Matthew R Walter, and Yuxin Chen. Active policy improvement from multiple black-box oracles. In Proceedings of the International Conference on Machine Learning (ICML), pages 11320-22332 2023. 2, 3, 4, 6, 9, 13, 14, 15, 16, 17, 18
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint arXiv: 1312.560, 2013.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application reward shaping. In Proceedings of the International Conference on Machine Learning (ICML), pages
1999. 15
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics, 7(1-2): 1-179, 2018.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988. 1, 13
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In Proceedings of the International Conference on Machine Learning (ICML), pages 4344 4353, 2018. 2,
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pages 661-668 2010. 13
Stephane Ross and Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint 106.5974 2014. 1,4

---

Published as a conference paper at ICLR 2024
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pages 627-635, 2011. 1,4, 13
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint 2015. 6, 7, 8, 13, 17
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: orXiv:/707.06:: 1707.06347, 2017. 6, 8, 13, 17
Idan Shenfeld, Zhang-Wer Wei Hong, Aviv Tamar, and Pulkit Agrawal. TGRL: An algorithm for teacher guided reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML). 2023. 2, 13
David Silver, Julian Schritwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature, 550(7676):354-333 2017.
Bharat Singh, Rajesh Kumar, and Vinay Pratap Singh. Reinforcement learning in robotic applications:
comprehensive survey, Artificial Intelligence Review, pages 1-46, 2022. 1
Yuda Song, Yifei Zhou, Ayush Sekhari, J Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid rl: Using both offline and online data can make rl efficient. arXiv preprint drX11:2210.06718. 2022. 13
Wen Sun, Arun Venkatraman, Geoffrey Gordon, Byron Boots, and J Andrew Bagnell. Deeply AggreVaTell Differentiable imitation learning for sequential prediction. In Proceedings of the International Conference on Machine Learning (ICML), pages 3309-3318, 2017. 1, 4, 13, 16
Wen Sun, Andrew Bagnell, and Byron Boots. Truncated horizon policy search: Combining reinforcement learning & imitation learning. arXiv preprint /r05: 2018. 1, 13
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems (NeurIPS), 1999. 6, 13
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. DeepMind control suite. arXiv preprint
2018. 2, 8, 17
Ronald J Williams. Simple statistical frllient algorithms for connectionist reinforcement learning, Machine Learning, 8(3):229-256, 1992. 13
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. World:Wrld: A benchmark and evaluation for multi-task and meta reinforcement lcarning. In Proceedings of the Conference on Robot Learning (CoRL), pages 1094-1100, 2020. 2, 8, 17
Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforcement learning. arXiv preprint arXiv:2302.00933 2023. 13
Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg. and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 2018.
Enmin Zhao, Renye Yan, Jinqiu Li, Kai Li, and Junliang Xing. AlphaHoldem: High-performance artificial intelligence for heads-up no-limit poker via end-to-end reinforcement learning. In Proceedings of the National Conference on Artificial Intelligence (AAAI), pages 4689-4697, 2022.
Brian D. Ziebart, A. Maas, J. Andrew Bagnell, and Anind. K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the National Conference on Artificial Intelligence (AAAI), 2008.
12

---

Published as a conference paper at ICLR 2024
A SELECTIVE COMPARISON AGAINST RELATED WORKS
Table 2: A qualitative comparison of related algorithms. The publication years are included in parentheses for reference. Algorithms designed to fit a particular criterion are marked by "V"; criteria that are not explicitly considered in the algorithm design are marked by "x".
Sample efficiincy (in multiple oracles)
Algorithm
Criterion
Online
Statefal
Active
Internctive
Multiple uracles
Robast
(Pomerlean, 1988)
Behuvioral Cloning
IL
X
X
REINFORCE (Willamss 1992)
R1.
X
X
X
[Sution et al., 1999)
(Ross Ragnell, 2010)
SMILe
IL
DAgger
(Ross al.,
GAB
(Schulman al.,
(Schulman al., 2015)
AggreVaret>
[San et al., 2017)
(Hester al., 2018)
Offinee - online RL.
THOR
(Sun al. 2018)
RR.
LOKI
(Chhnng et al. 2018)
HLL
SAC-X
(Riedmiller et al. 2018)
RL
LEAQU
(Bmmlly et al., 2020)
II.
(Cheng et al. 2020)
MAMBA
RL+
A-OPS
Policy Sel.
(Kongushosa
ILEED
(Beliaev 2022)
Officee oaline RL.
(Kostikav aall.222222 2021)
CAMS
Model Sel.
2022a)
(Hassen al., 2023)
MoDem
Offinee -i caline XL
Hybrid RL [Song al., 2012)
Online RL with offline data
PEX
Offinee - coline RL
EZlling al., 2003)
TGRL
(Shenfeld
IL
LIGP
MAPS
IL+RE
[Liu al,, 2023)
(Ques)
IL+RR
A.1 ADDITIONAL NOTES ON RELATED WORK
MAMBA addressed the challenge of learning from multiple sub-optimal oracles and tackled two fundamental questions: what constitutes a reasonable benchmark for policy improvement, and how to systematically combine sub-optimal oracles into a stronger baseline. MAMBA proposed a max- aggregated baseline and suggested policy improvement from it natural strategy for combining these oracles to form a better policy. In addition, they introduced a novel Max-aggregation of Multiple Baseline approach and provided a theoretical performance guarantee for it. However, one limitation of MAMBA is its high sample complexity. requires prolonged rounds to identify the optimal oracle for given state due to its strategy of uniformly sampling oracle, resulting in larger
13

---

Published as a conference paper at ICLR 2024
accumulation of regret. MAPS aims to enhance the sample efficiency of MAMBA by introducing Max-aggregation Active Policy Selection with theoretical support, and it empirically outperforms MAMBA. Nevertheless, both MAMBA and MAPS share common limitation in non-robustness. They are susceptible to the quality of the oracle set, and both algorithms may fail in cases where the oracle set is of poor quality. Our work addresses this robustness challenge by proposing novel max" framework. Inspired by the maxaaggregation policy from MAMBA, we introduced a max--aggregation policy based on a novel extended oracle set. This enables the policy undergo self-improvemmet even when the oracle set is poor. Additionally, we proposed novel algorithm, RPI, with Robust Active Policy Selection improve active policy selection from MAPS. Theoretical analyses were provided for both the max" framework and the RPI algorithm.
B ADDITIONAL BACKGROUND
B.1 ADDITIONAL. ALGORITHMS LEARNING FROM MULTIPLE ORACLES
In this section, we introduce few basclines that learn from set of black-box oracles II {**) /
Single-best expert n²: The first baseline that we consider imitates a single oracle that achieves the best performance hindsight among the oracle set, i.e. π- arg max ao [V* (so)]. After figuring out the single-best expert, this strategy simply keeps rolling out the expert. In practice, this often inadequate as it neglects the potential benefits of suboptimal oracles at the state level.
Max-following =": Given a collection of k imitation learning oracles II = { ]еe[K]. the max- following policy (Cheng et al., 2020; Liu al., 2023) a greedy policy that selects the oracle with the highest expertise in any given state:
(a () == nk (al kt := arg max V (s)
kE[K]
where (*(s) V** (s) is the value function for oracle k E [K].
Max-aggregation π max; Max- aug-eggregttion (Cheng et al., 2020) performs one-step improvement based on the max-following policy n." Denote a natural value baseline fruux (st) for IL with multiple oracles as
mmax (8t)
max V (s). kE(K] K]
(15)
We then denote the max-aggregation policy as
max (a s) Sama., where a" arg max A g (s,a), aEA
A mmX (8, a) = r (8,a) a) E pmax - fmax (s)
and ò is the Dirac delta distribution.
(16)
The max max-aggregatie aaxaegggegttinn policy is function of fmaa and, in turn, requires knowledge of the MDP and each oracle's value function (Eqn. 15). However, in the episodic interactive II setting, oracles provided as black boxes and their value functions are unknown. MAMBA (Cheng et al., 2020) and MAPS (Liu et al., 2023) deal with this by reducing IL to an online learning problem and adapt the online loss defined at round n as:
Ln (T;; A) (1 A)HE
[ART
(s.
AE [A1 (s,
(17)
Here, A mix (s) a A-weighted advantage defined as:
A (s.a):= Σ NA (s,
(18)
which integrates various i-step advantages:
fT (Se, (r) = En ( (r az) + ... (8t+i, + maa (sc- - (st).
(
14

---

Published as a conference paper at ICLR 2024
B.2 LIMITATIONS OF THE PRIOR ART
MAPS (Liu et al., 2023) and MAMBA (Cheng et al., 2020) suffer from two limitations related to their non-robustness to the choice of the oracle set. First, their online loss function (17) relies on predetermined À value that combines imitation learning and reinforcement learning, making their performance sensitive to the quality of the oracle set. Second, their gradient estimator utilizes the aggggeggent policy x= and the value bascline function frm., both of which are dependent the given black-box oracle set. the oracle set includes only adversarial oracles, these methods will still try to perform imitation learning, thereby impeding policy enhancement.
B.3 VALUE FUNCTION APPROXIMATOR FOR DISCRETE ENVIRONMENT
In the interactive episodic MDP, we roll out a selected oracle k, resulting in (s;) trajectories Tikk, .2. .N starting from state & for round N. We determine estimate for the return in state 8, by averaging the returns obtained across the trajectories:
Na(5.)
T. (8t)
1
Σ Xr (8),0))))
(19)
N& (8e)
B.4 ACTIVE POLICY SELECTION
To address the sample efficiency challenge in learning from multiple experts, we reference active policy selection technique in MAPS work to select the best oracle hst for state as follows:
(st(s) + (s)) + t (st)
2H log 3 Nu(a()
S discrete S continuous
k, arg kE[K]
(20)
C PROOFS
In the following, we provide proofs for theoretical claims in the main paper.
Lemma C.1. (Kakade and Langford, 2002; Ng et al., 1999) Let f S - R such that f ($H) = For any MDP and policy
v (do) f (do) HEggdd A A' (s,
(21)
From Lemma C.I, we get the following corollary:
Corollary C.2. (Cheng et al., 2020) If f is improvable with respect to n, then V (s) 2 (s), Vs E S.
Corollary C.2 indicates that policy outperforms all policies in II if, for every state, there is baseline value function f superior to that of all policies (f(s)) 2 V (s), Vk E | E S), while
f can be improved by π (i.e., A (s, π) 2 0).
C.1 PROOF OF PROPOSITION 4.5
Proof. Without loss of generality, let assume the optimal oracle is oracle 1 (the first oracle) in oracle set II,
A (8,7*) = (s, + En Be Ba [f (s') f (s)
(22a)
2 (8, + Earry Be V (8/) V (s) n (
(22b)
[v (5/) V (s)
(22c)
A v (s,x²) > 0,
(22d)
15

---

Published as a conference paper at ICLR 2024
where the last step follows since x² (als) n (als) n (als). Since have A (s, 2 together with Lemma C.I, we have
V 2 f (8)
max (s). [E[[TT]]
(23)
V= (s) 2 f (s) indicates that following ' is equally good or superior to imitating a single best policy in 11
C.2 PROOF OF PROPOSITION 5.1
We denote So (8) == arg max V (s). According to Theorem of Cheng et al. (2020), we obtain
E
max V (do) 2 Bando (5o (8) + E [AN - 6N (TT²) Regret C nE(N]
(24)
Now let II for RPI as [E If [s)]+EB [AN eN (11-) Regret Since 1E lf (s)] 2 Eg Lfö (s)],
II Tm- Following the same reasoning strategy as above, we will have lower bound
we have performance lower bound of RPI no worse than MAMBA.
Remark. MAPS (Liu et al., 2023) retains MAMBA's lower bound but enhances sample efficiency and reduces the bias in Regretf. The inherent uncertainty of the optimal policy x* E II makes an unbiased f estimate challenging. The regret term Regret is bounded by:
E [ 0o = + VoN)
where B+ is the estimation bias that results from selecting the non-optimal policy #* in II for given state, and f is the value estimation error w.r.t. the true value for given state of selected policy 2* and v represents the variance term.
MAPS improves upon MAMBA's sample complexity, reducing bias in its regret bound via an active policy selection mechanism. Our work builds on MAPS, emphasizing empirical enhancements in active policy selection with the integration of the learner policy in ITF
D EXPERIMENTAL DETAILS
D.1 BASELINES
Aggre VggreVaTeD (Sun et al., 2017) is differentiable version of Aggre which focuses on a single oracle scenario. AggreVa VaTeD allows us to train policies with efficient gradient update procedures. AggreVaTeD models the policy using differentiable imitation learning. By applying differentiable imitation learning, it minimize the difference between the expert's 's demonstration and the learner policy behavior. Aggre VaTeD learns from the expert's demonstration while interact with the environment to outperform the expert.
deep neural network and trains the policy
Max-Aggregation We have developed variant of the Max-aggregation policy outlined in Equation (16) that is specifically designed for pure imitation learning using multiple oracle sets. When utilizing single oracle, it effectively reduces to VateD. Our approach builds on the existing MAMBA framework by setting the lambda value in the loss function to zero. While max- aggregation may always yield the optimal policy, it offers the advantage of being able achieve results with fewer samples, making it a more sample- -fffccefffiient option.
LOKI-variant LOKI (Cheng et al., 2018) is strategy for policy learning that combines the imitation learning and reinforcement learning objectives in two-stage manner for the single oracle setting. In the first stage, LOKI performs imitation learning for a small but random number of iterations and then switches to policy gradient reinforcement learning method for the second stage. LOKI is able to outperform a sub-optimal expert and converge faster than running policy gradient from scratch. In this work, propose variation of LOKI that adapts multiple experts. During the first-half of training (i.c., the first stage) we perform Max Max- aggregation style imitation learning, and then perform pure reinforcement learning the second stage.
16

---

Published as a conference paper at ICLR 2024
PPO-GAE Schulman al. (2015) proposed the generalized advantage estimator (GAE) as means of solving high-dimenn sighddimensional continuous control problems using reinforcement learning. GAE is used to estimate the advantage function for updating the policy. The advantage function measures how much better a particular action is compared to the average action. Estimating the advantage function with accuracy in high-dimensional continuous control problems is challenging. In this work, propose PPO-GAE, which combines PPO's policy gradient method with GAE's advantage function estimate, which is based a linear combination of value function estimates. By combining the advantage of PPO and GAE, PPO-GAE (Schulman et al., 2017) achieved both sample efficiency and stability in high- -ighddimensional continuous control problems.
MAMBA MAMBA (Cheng et al., 2020) is the SOTA work of learning from multiple oracles. It utilizes mixture of imitation learning and reinforcement learning to learn a policy that able to imitate the behavior of multiple experts. MAMBA is also considered as interactive imitation learning algorithm, it imitates the expert and interact with environment to improve the performance. MAMBA randomly select the state as switch point between learner policy and oracle. Then, it randomly selects the oracle to roll out. It effectively combines the strengths of multiple experts and able to handle the case of conflicting expert demonstrations.
MAPS MAPS (Liu et al., 2023) is policy improvement algorithm that performs imitation learning from multiple suboptimal oracles. It actively chooses the oracle to imitate based on their value function estimates and identifies the states that require exploration. By introducing two variations, Active Policy Selection (APS) and Active State Exploration (ASE), MAPS improves the sample efficiency of MAMBA. The MAPS variant selects the most promising oracle, denoted as k., for rollout, utilizing the resulting trajectory to refine the value function estimate vrk. (st). This approach aims to minimize the chances of selecting an inferior oracle for given state st, thereby reducing both the sample complexity and gradient estimation bias. On the other hand, the ASE variant of MAPS deliberates whether to continue with the current policy switch to what is believed be the most promising oracle, similar to APS, by leveraging an uncertainty measure over the current state. In this study, we adopt MAPS variant as our bascline method.
D.2 GYM ENVIRONMENTS
We evaluate RPI and compare its performance the aforementioned baselines the Cheetah- run, CartPole-swingup. Pendulum-swingup. and Walker-walk tasks from the DeepMind Control Suite (Tassa et al., 2018) and Window-close, Faucet-open, Drawer-close and Button-press from WeradWorld (Yu et al., 2020). In addition, we conduct experiments on a modified sparse reward Meta-Workd environment, which is considered be more challenge task.
D.3
SETUP
Setup. In order to ensure a fair evaluation, all baselines are assessed using an equal number of environment interaction steps (training steps). Each training iteration involved a policy rollout for the same number of steps. We note that there is discrepancy in the amount of data available to the learners of RPI and PPO-GAE. RPI (MAPS, MAMBA, Max-Aggregation uses some of the interactions to learn the value function for each Oracle, which results in relatively less data for its learner, whereas PPO-GAE utilizes all the environment interactions to update all benefits for its learner policy. Thus, in this work, we balance the transition buffer size for each algorithm make them have approximately same number of stored transitions for learner policy improvement. We average the result based 5-10 trials.
D.4 IMPLEMENTATION DETAILS OF RPI
We provide the details of RPI in Algorithm 1 as Algorithm 2. Algorithm 2 closely follows Algorithm 1 with a few modifications as follows:
In line 5, we use buffer with a fixed size (DD, = 19, 200) for each oracle, and discard the oldest data when fills up.
17

---

Published as a conference paper at ICLR 2024
Algorithm 2 Robust Policy Improvement (RPI)
Input: Learner policy T1. oracle set II = (Tth (rr KEEKK function approximators {V ]EE[K] Vn
Output: The best policy in {T1, TN
1: for = ...,.... N
do
2:
Construct an extended oracle set IT = [ ,....., x², Ta ke([TT]
3: Sample le [H -11 uniformly random. We have buffer with fixed size (12xl 19, 200)
for each oracle, and we discard the oldest data when it fills up.
4: Switch to n².
5: Roll-in Ta up to le. select k; (10), and roll out xh to collect data D
6:
Update f2k, using D*
7:
Roll-out the learner policy nn for specified steps (2, 048), and add them the buffer D
8:
Update Vn using D
9:
Compute advantage A
GAE+
(1I) and gradient estimate 9n (14) using D
10:
Perform PPO style policy update on policy Ta to Tn+1-
In line 7, we roll-out the learner policy until the buffer reaches fixed size ([D" = 2,048), and then empty once we use the roll outs update the learner policy. This stabilizes the training compared to storing a fixed number of trajectories in the buffer, as MAMBA does.
In line 10, we use PPO with max" actor-critic style policy update.
We pretrain the value function V of oracle ic before the main training loop, with trajectories generated by rolling out x* from the initial states. In the main training loop, we train 1i using the corresponding rolled-out trajectories, bootstrapped only by itself. This is the same strategy as in MAMBA and MAPS. Similarly, we train the learner value function V, using only trajectories rolled-in with Tny bootstrapped only by itself.
D.5 HYPERPARAMETERS AND ARCHITECTURES
Table 3 provides a list of hyperparameter settings we used for our experiments. We use an ensemble of MLPs to predict oracle's value. With five identical MI.Ps that are separately initialized at random, we train each MI.P separately for the same dataset. At inference time, we collect predictions from all MLPs for single input and compute the mean and standard deviation.
D.6 COMPUTING INFRASTRUCTURE AND WALL-TIME COMPARISON
We conducted our experiments cluster that includes CPU nodes (approximately 280 cores) and GPU nodes (approximately 110 Nvidia GPUs, ranging from Titan to A6000, set up mostly in 4- and 8-GPU configurations). Based on the computing infrastructure, we obtained the wall-time comparison Table as follows.
E SUPPLEMENTAL EXPERIMENTS
E COMPARING RPI AGAINST BASELINES WITH A DATA ADVANTAGE
In the main paper, followed an experimental setup where we assumed that the learner had access to approximately the same quantity of transitions, while lacking access to the oracle's offline trajectory. However, some of the baseline algorithms, such as MAPS (Liu et al., 2023) were originally evaluated under different setting in the literature: They assume that the learner from the oracles' pre-traie pre-trained offline dataset. In this section, we run MAPS under such a setting, in order to provide more comprehensive evaluation that is consistent with the literature. Note that under this experimental setup, MAPS has approximately twice the amount of data compared to RPI. We refer this variant as MAPS-ALL.
access additional data
In contrast to MAPS our original configuration, the performance of MAPS-AN SALPS-ALLAE doubles in the Cheetah environment (as shown in Figure 5(a)) and the Pendulum environment (as depicted in Figure
18

---

Published as a conference paper at ICLR 2024
Parameter
Value
Shared
Learning rate
x 10-1
Optimizer
Adam
Nonlinearity
ReI.U
# of functions in value function ensemble
5
Oracle
oruclees oracle (K)
for oracle ( (1z*1)
episodes to
value pretraining
Learner
Horizon of MetaWorld and DMControl (H)
300, 1000
Replay for learner policy (IP,,)
GAF gamma (y)
0.995
GAE lambda (A) for AggreVaTeD and Max-Aggregation
0
V.Kiivariant
or
RPI
0.9
training steps (rounds) (N)
100
# of episodes to perform RIRO (Alg 1. line 4) per training steration
4
mini batch size
128
epochs perform gradient updates per taaining itenttoon
Table RPI Hyperparameters.
Methods
Cheetah
Cartpole
Walk Walker
Pendulum
MAPS
18m
1h 10m
41m
17m
MAMBA
1h 25m
59m
14m
21m
LOKI-variant
1b 17m
Ih 36m
2h 11m
1h 12m
PPO-GAE
54m
49m
MAX MgX-aggroaaiion
1h 5m
Ih 34m
2h 25m
1h 13m
RPI
57m
18m
Table Wall-time comparison between different methods.
5(c)). Moreover, the performance MAPS-ALL surges between middle and end of rounds in the Pendulumn environment. This behavior mirrors what reported in the original MAPS paper as well. In the Walker-walk environment (as illustrated in Figure 5(b),, MAPS-ALE demonstrates an approximate 10% improvement. For the Cartpole environment (Figure 5(d)), MAPS-ALL's performance increases by around 20%. MAPS-ALL exhibits overall performance similar to that of its original paper, with any differences caused by the difference in the oracle set. Notably, result, MAPS-ALL distinctly outperforms RPI only in the Pendulum environment. RPI's performance remains comparable MAPS-ALL in the Cheetah environment and significantly surpasses the MAPS-ALL baseline in the Walker-Walker Walker and Cartpole environments, despite utilizing much less data.
E.2 META-WAR WORLD EXPERIMENTS (DENSE REWARD)
In Fig. 6, we conducted additional experiments comparing RPI and state-of-the-art (SOTA) methods (MAMBA, MAPS), as well as the best-performing oracle and PPO-GAE, across the Meta-World benchmarks. The tasks are including (1) window-close, (2) faucet-opent (3) drawer-cle closee, and (4) button-press. RPI demonstrates superior performance compared to all baselines in the majority of environments, with the exception of the button-press task.
19

---

Published as a conference paper at ICLR 2024
E-K NAMBA
Best retumm
2P1 MAPS-LLL
Best
Best
Training step
Training step
Training step
Training step
(a) Chectah (donse)
(b) Walker-walk (dense)
(c) Pendulum (sparse)
(d) Cartpolo (dense)
Figure 5: Running MAPS in the original paper's setting.
HATS MAUURA
MARS
MAUUA
Boo returo
Host rettrm
Best retlum
600
Bos
460-
Training stop
Training stop
Training step
Training step
(a) Window-close (dense)
(b) Faueet-open (dense)
(c) Drawer-close (dense)
(d) Buttou-press (dense)
Figure 6: Experimental results on the Meta- World benchmark with dense reward.
E.3 META WORLD EXPERIMENTS (SPARSE REWARD)
In Fig. 7, further demonstrate the advantages of imitation learning, we modified the World environment to create more challenging sparse reward environment. In this environment, the agent only receives reward of I upon success; otherwise, receives a reward of 0. We then compared the performance of the RPI and state-of-the-an (SOTA) imitation MAMBA, MAPS, as well as the pure RL method PPO-GAE, and the best-performing oracle across the Meta-World benchmarks. The tasks include (I) window-cle crawer-close and (4) button press, these sparse reward environments, when provided with good oracle, the imitation learning-bascl approach demonstrates its advantage over the pure RL approach. RPI, MAPS, and Mamba outperform PPO-GAE by factor of the but t ton-tress enviromment. When provided with bad oracle, RPI can still outperform MAPS and MAMBA in the faucet-oper open environment. Moreover, even with a poor oracle, RPI outperforms both basee approaches (MAMBA, MAPS) and the RL-bbased approach (PPO-GAF)) in the wi wwndowwwae close environment, showcasing that RPI enjoys benefits from both RL and IL aspects.
methods
(2) Saucet-opent (3)
Best reture
Best eeturr
Best returrs
Best
Training step
Training
Training stop
Training stop
(a) Window-close (sparse)
(h) Faucet-open (sparse)
(<) Drawer-close (sparse)
(d) Button-press (sparse)
Figure 7: Experimental results on the Meta- World benchmark with sparse reward.
E.4 SINGLE/EMPTY ORACLE SET
In Fig. 2, mainly discuss experiment multiple oracle set. In Fig. 8, demonstrate that RPI is also robust enough to handle single or empty oracle set.
Single oracle setting, In Fig, 8(b), demonstrate that RPI outperforms all other baselines in single oracle setting as well. This is consistent with the results observed the multiple-expers setting. Fig. 8(c) demonstrates that providing oracle with mediocre performance to RPI boosts the performance rather than providing a optr-optimal oracle. Since we train oracles' value functions from the oracle rollouts, the value function of the near-optimal oracle may not have the "bad" states
20

---

Published as a conference paper at ICLR 2024
that the learner policy would encounter in the early stage. This leads to inaceuracy in the predicted values for such states. In comparison, the value function for a mediocre oracle would be able to produce accurate predictions on such states.
No oracle environment. When there are no experts available, the performance of imitation learning- based approaches will inevitably degrade. However, as shown in Fig. 8(a), RPI can adapt to such scenario by regressing to pure reinforcement learning. Since we extend RPI based on PPO-GAE, achieves a similar level of performance to PPO-GAE when the oracle is empty.
Fre-GAZ
164
106-
Bost roturn
Best retumn
MAMARA
Training step
Training step
Training step
(4) No oracles
(b) Good single oraele
(c) H0HHeellul oracle ablation
Figure 8: Ablation study. (a) Comparing RPI and PPO-GAE under no oracle Pendulum environment. (b) Comparing RPI and six baseline under single expert Cheetah environment. (c) ablation study on oracle quality under Cheetah environment.
E.5 ABLATION ON CONFIDENCE THRESHOLD r,
In Table 5, we perform an ablation study of threshold r. in Eqn. 12 robust policy gradient component. We treat I, as a hyperparameter the choice of which depends on the user's risk aversion. Empirically, we find that a setting of T, = 0.5 robustly works well nearly every setting in our experiments across Deepmind Control suite, Meta-Wan World (Dense reward), and Meta-world (Sparse reward) environment, the one exception being the Pendulum domain. To better understand how the choice of I effects overall performance, the Deepmind Control Suite using different values for I', E [0, 0.5, 3, 5]. As the following table shows, setting r = 0.5 yields the best performance for all but the Pendulum environment.
conducted set of experiments in which we RPI on
In practice, one can first use roll outs each oracle to estimate the standard deviation and associated confidence intervals of their ensemble values. A conservative user could then start by setting r, based on probabilistic lower-bound of a and subsequently tune the hyperparameter according to user's risk aversion preference.
Enviromment
Round
r, 0
r, = 0.5
P, = 3
T,
Cheetah
100
252.7 a 23.2
291.2 36.3
251.4 1
$3.4
81.3 20.8
Walker-walk
100
6.5
402.2 57.7
2
31.8
38.2 1.9
Pendulum
100
34.2
45.6
+.1
Cartpole
100
445.7 + 13.5
670.4 110.1
394.8 + 50.6
301.7 + 60.0
303.2 4.00
Table 5: Tuning the confidence threshold r..
E.6 ABLATION ON UCB/LCB POLICY SELECTION
Environment
Round
RPI-RAPSILCEAUCB)
RPI-MEAN
Cheelah
100
291.2 a 36.3
263.0 a 33.7
Walker-walk
Pendulum
100
54.3
53.8 a
Cartpole
100
670.4 = 110.1
354.3 + 65.2
Overall
100
1418.1
1013.8
Table 6: Ablation study confidence aware UCB/LCB vs MEAN policy selection
21

---

Published as a conference paper at ICLR 2024
EPI LC
Best return
retarn ". s.
Best
Best
190
Training step
Training step
Training step
Training step
(a) Cheetah
(b) Curtpole
Pendulum
(d) Walker-walk
Figure 9: Experimental results ablation study on confidence aware UCB/LCB vs MEAN policy selection.
We conducted an ablation study that compares RPI-LCB/UCB, which takes uncertainty into account follows:
K arg max (v)()), V2)),,.... (K)))).,,,,(((())
(s))
against RPI-MEAN, which does not consider uncertainty as:
K = arg max V (8)))...., (x))()))
The experimental results presented in Fig. 9 and Table demonstrate that the RPI-LCB/UCB strategy outperforms RPI-MEAN across all benchmarks by overall margin of 40%. This highlights the significance of incorporating uncertainty in the policy selection strategy.
22


Reviews:
[{"review_id": "1JcNT1cjiL", "reviewer": "Reviewer_yEJM", "summary": "This work proposes a new policy gradient algorithm named Robust Policy Improvement (RPI), which learns policies through IL and RL. They propose the Robust Active Policy Selection (RAPS) to improve value function efficiently and then use the Robust Policy Gradient (RPG) to update policies, which is a variant of the actor-critic method. In addition to some theoretical proof of the optimality of the method, some experiments show that RPI outperforms other baselines to a degree.", "questions": "As listed in weakness.\n\nBy the way, why are the links in the paper invalid? (such as citation and equations)", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Paper writing is clear to understand. And the method is natural.\n2. The theoretical analysis is adequate.", "weaknesses": "(1) Novelty. \n- The policy improvement of perfect knowledge is similar to making ensembles of several imitated policies. The theoretical analysis of the method seems a little redundant. \n- The exploration in RPI is uniformly random sampling (line 3 of Alg.1). \nThese seem trivial.\n\n(2) The experimental setting is not clear and sufficient.\n- How many demonstrates for imitation learning? How many online interactions for reinforcement learning? The x-axis of the curves is training step, what is 100 training step means? And do the baselines use the same examples and online interactions?\n- The selected tasks are easy. In meta-world, as I know, button-press and window-close are not hard. What about bin-picking and door-open? In DMC, what about hopper or even humanoid?\n- Lack of baselines. Except for MAPS, all the baselines are from before 2020. I think there are more works than listed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2lc5YR5LXL", "reviewer": "Reviewer_6RE6", "summary": "This paper investigates the setting of learning from multiple oracles that can also be framed as an imitation learning setting. The work proposes a novel procedure for combining learning from oracles with active exploration using a framework called robust active policy selection in combination with a robust policy gradient. The method interoperates between imitation learning and reinforcement learning and smoothly transitions between the two. A theoretical introduction of the presented method is followed by empirical evaluations that show improved performance over several baselines.", "questions": "Q1. What is the relationship between the studied setting and offline RL? Offline RL, similar to the studied setting, tries to blend behavioral cloning with value function learning. In that sense, is any offline RL algorithm a candidate for learning from multiple experts? I’m not claiming that offline RL needs to be added in prior work if it turns out to be unfitting but it would be great if the authors could elaborate on this point.\n\nQ2. Does the max-following policy execute a mixture over policies or does it, given a state, fix a policy and then execute this policy throughout time? This sentence is slightly confusing me here: “Specifically, if all oracles perform worse than the learner policy at a given state, the max-following policy will still naively imitate the best (but poor) oracle.” If the former is true, then it would be better or equal to the best poor oracle.\n\nQ3. I’m confused about Line 5 of the algorithm. $D^k$ contains data from possibly two different policies, however it is used to update a single value function. Is this a typo or am I misunderstanding the algorithm here? I would think this should only be taking the second portion of the roll-out data, not the roll-in data. Could you elaborate why this is okay to do?\n\nQ4. In the policy selection step, the text states “Using LCB for the learner encourages exploration unless we are very sure that the learner surpasses all oracles for the given state”. What exactly does the term exploration refer to here? Is this just meant as exploration for the learner’s policy?\n\nQ5. To clarify that I am understanding this correctly. In Eq 14, the gradient is computed with respect to the learner’s actor policy, but with the overall multi-step advantage from the collection of value functions correct? I’m not 100% sure that I understand why this is okay since $\\hat{f}^+(s_{t+1})$ and $\\hat{f}^+(s_{t})$ can be from different policies, right? My guess is that it is fine because they are both independent of the action but I would appreciate some clarification.\n\nQ6. Did you use the original implementations for MAPS and MAMBA or did you reimplement the methods? Do you have an intuition why in the present comparisons these methods perform significantly worse than in the previous literature?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Motivation\n* The work is well-motivated as an approach to learn from multiple oracles and improve upon them. Building frameworks that can blend between existing knowledge via oracles and exploration using a learners policy seems like a good way to enhance the setting of only learning from oracles.\n\nStructural clarity\n* The paper is well structured and well written. The flow is very clear. I do have to say that I did get lost in the notation details at times because there is a lot of notation that is quite similar. The switch from theoretical analysis to empirical investigation is handled nicely in the text.\n\nMathematical rigor\n* The work uses strict mathematical definitions and states its assumptions clearly. The proof for proposition 4.5 is intuitive. I did not carefully check the proof of proposition 5.1 in the Appendix. However, it seems reasonable that an online no-regret learning algorithm would obtain high value.\n\nNovelty\n* The work provides a sufficiently novel version of an existing algorithm that enables a mixture of online and offline optimization. Some of the novelty might be slightly exaggerated though and could be made more clear, see Weakness: Contextualization with prior work).\n\nExperimental evaluation\n* The experiments consider a sufficient number of environments and baselines and all evaluation is done over a reasonable number of random seeds. Ablations give insight into the behavior of the algorithm and provide intuition that the algorithm works as intended.", "weaknesses": "Contextualization with prior work  \nBefore I start, I would like to mention that I am not familiar with this sub-field of RL but do know the standard online and offline RL literature quite well.\n* The prior work section is rather brief with a total of 6 citations. I am not familiar with the exact sub-field but this seems rather little given the literature is several years old.\n* My first main concern is related to the contextualization of various parts of the paper to previous work. I think it could be made clearer which parts are inspired by previous work. For instance, the ideas of UCB value selection and multi-step advantage over several value functions seem already present in previous work but this is not immediately apparent from the current manuscript. I understand that the present manuscript defines these things with the inclusion of an additional learner policy, which requires the development of novel machinery. However, I think it would be good to give credit to the general formulations in previous work where appropriate.\n\nExperimental evaluation\n* My second main concern is the following. The experimental setting seems very similar to that studied in Liu et al., since environments and baselines are relatively similar. However, it seems that the performance of the reported baselines in this paper differs significantly from the performance reported in the previous study of MAPS. Both MAPS and MAMBA seem to be unable to even achieve performance close to the best oracle while in the MAPS paper both algorithms perform significantly better on the tasks that seem to be identical to the ones chosen here. See Q6.\n\nTextual clarity suggestions\n* This might be a me thing but the term robust policy gradient has been used several times in the literature before. It might make sense to consider a less generic name for the algorithm. Examples:\n\nXuezhou Zhang et al. Robust Policy Gradient against Strong Data Corruption. ICML 2021.\nYue Wang and Shaofeng Zou. Policy Gradient Method For Robust Reinforcement Learning. ICML 2022.\n\n* I’m being a little nitpicky here but the policy gradient theorem with baseline is definitely older than 2015 (P6 section 6.2). It might be good to cite relevant prior work here.\n* Some of the notation can be quite confusing and I lost track of what index refers to which object several times. This is mostly because lot’s of variables are indexed by multiple things but all objects are somewhat similar. I don’t really have a good suggestion on how to fix this though other than reducing the size of separate, less relevant sections.\n\nOverall, I do think that the community will benefit from the publication of this work. I am inclined to raise my score if my two main concerns can be addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o0uicd7xDd", "reviewer": "Reviewer_oW5n", "summary": "This paper proposes the max+ algorithm, which is a hybrid imitation + reinforcement learning algorithm which utilizes a set of expert policies to aid a policy in initial learning and exploration, and gradually transitions to reinforcement learning as the learner policy improves. The method maintains a value estimate for each expert policy as well as the learner's, and uses the value functions to construct an update rule that improves over the best expert in the set. The authors then apply their proposed method to the continuous control domain on DM Control Suite and Metaworld tasks, where they demonstrate that their method achieves greater learning speed than prior baselines.", "questions": "- How would the following naive baseline perform? Initialize from the expert policy, and then finetune from there with RL. It seems like this baseline could perform quite well in the proposed experimental setup, given that PPO-GAE is one of the baselines and performs quite well.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is clear and well written. The authors have done a good job of presenting an overview of their method, giving theoretical justification of why the method works, and benchmarking against several state-of-the-art baselines.\n- The experiments are detailed, and I appreciate the ablation studies which show the usefulness of the robust policy selection rule vs naive rule, and the transition from utilizing the expert policies early in training while transitioning smoothly to RL as the policy improves.\n- I am not familiar enough with the literature in this subdomain to judge the degree of novelty in the approach, but based on the information presented in the paper the proposed update rule (the robust policy gradient, and robust policy selection) is novel.", "weaknesses": "- For reproducibility purposes, it would be great if the authors could include a table of algorithm hyperparameters used in the appendix, and hyperparameter sweeping strategies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uuJu6PLrpZ", "reviewer": "Reviewer_EG6V", "summary": "This paper proposes a method that combines RL and IL to learn a policy that improves upon a set of imperfect oracles. The method extends the MAMBA algorithm [1] by incorporating the learner policy into the set of oracles and using uncertainty-aware value-estimates to define the exploration strategy and advantage estimator. These changes enable the proposed method to behave like an RL algorithm in cases where the oracles do not provide beneficial supervision. The proposed method achieves good results in mujoco and meta-world tasks, beating most of the oracle policies and baselines.", "questions": "- Equation (12)\n    - What does equation (12) mean when the \"when\"-clause equates to false?\n    - What does $\\hat{V}_{\\mu}^{|\\Pi^{\\varepsilon}|}$ mean?\n- How many members are there in the value-function ensemble?\n- How many oracles does each experiment have?\n- Why are the replay buffers for the oracles so small? The oracle policies do not change, so the data does not become more off-policy as the training proceeds.\n- What are the x-axes in the result figures? Surely the algorithms do not converge in 100 gradient steps?\n- How many timesteps are sampled for value-function pre-training? How would the learning curves look like if these steps were included on the x-axis? In general, using environment steps as the x-axis would make it easier to compare methods that are otherwise so different.\n\n\n[1] Policy Improvement via Imitation of Multiple Oracles, Cheng et al. 2020\n\n[2] Policy Gradient Methods for Reinforcement Learning with Function Approximation, Sutton et al. 2000", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method beats a range of relevant baselines in a diverse set of benchmarks.\n- The sections 1 to 5 are generally well written and reasonably easy to follow. Caveat: I did not verify the derivations myself.", "weaknesses": "## Presentation\n- The method description has missing details. These are also issues with soundness of the paper.\n    - The necessary details for understanding equation (12) are not clearly presented. See questions.\n    - The value ensemble is central to the proposed method, yet there are only two sentences at the bottom of page 5 discussing how the value ensemble is trained.\n- The theory section borrows from MAMBA, which is only natural since these methods are closely related, but it would have been clearer to emphasize the points of departure from MAMBA more and perhaps move the parts that are more closely shared to the appendix.\n- Nitpick:\n    - The definition of policy gradient with advantage is cited for Schulman, but there are prior works considering policy gradients with advantage estimation, e.g., [2].\n    - The abstract mentions real-world scenarios but the paper does not talk about them.\n\n\n\n## Soundness\n- Computing the update requires evaluating the value ensembles for all of the oracles. Compared to normal RL algorithm, e.g., PPO, this means number of oracles times the size of ensemble more forward passes. Depending on the number of oracles and ensemble members, this may equate to a lot more expensive loss evaluation. In addition to the missing details about the ensemble size and number of oracles, it would be good to include some wall-time comparisons between the different methods.\n\n## Experimental evaluation\n- Cartpole is a bit too easy environment for the otherwise well-executed ablation study.\n- In the appendix, it says that the value ensembles are pre-trained before the main loop starts. That sounds like the proposed method uses more samples (and more compute) than the baselines. This should be mentioned in the main paper and discussed as a limitation.\n\n## Summary\nThe proposed method seems to do well, but the paper is missing too many details and is too hard to follow at parts that it is hard to evaluate whether the contribution is impactful or not. I think including a thorough discussion about the relationship of the proposed method and MAMBA as well as limitations would go a long way to make this easier to understand. Additionally, adding all of the missing details that I have listed here and in the questions would help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)