Hybrid Policy Optimization from Imperfect Demonstrations
Authors: Hanlin Yang
Chao Yu
peng sun
Siji Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: reinforcement learning
sparse reward
exploration
learning from demonstrations
TL;DR: Proposed a method to accelerate online agent learning in sparse reward environments by using only a small number of imperfect demonstrations.

Abstract: Exploration is one of the main challenges in Reinforcement Learning (RL), especially in environments with sparse rewards. Learning from Demonstrations (LfD) is a promising approach to solving this problem by leveraging expert demonstrations. However, expert demonstrations of high quality  are usually costly or even impossible to collect in real-world applications. In this work, we propose a novel RL algorithm called HYbrid Policy Optimization (HYPO), which uses a small number of imperfect demonstrations to accelerate an agent's online learning process. The key idea is to train an offline guider policy using imitation learning in order to instruct an online agent policy to explore efficiently. Through mutual update of the guider policy and the agent policy, the agent can leverage suboptimal demonstrations for efficient exploration while avoiding the conservative policy caused by imperfect demonstrations. Empirical results show that HYPO significantly outperforms several baselines in various challenging tasks, such as MuJoCo with sparse rewards, Google Research Football, and the AirSim drone simulation.

Hybrid Policy Optimization from Imperfect Demonstrations
Hanlin Yang Sun Yan-sen University
Chao Yu* Sun Yat-sen University
Peng Sun ByteDance
Siji Chen
Sun Yat-sen University
yuchao3@mail sysu. edu.
Abstract
Exploration is one of the main challenges in Reinforcement Learning (RL), es- pecially in environments with sparse rewards. Learning from Demonstrations (LfD) is promising approach to solving this problem by leveraging expert demon- strations. However, expert demonstrations of high quality are usually costly or even impossible collect in real-world applications. In this work, propose a novel RL algorithm called HYbriid Policy Optimization (HYPO), which uses a small number of imperfect demonstrations accelerate agent's 's online learning process. The key idea is to train an offline guider policy using imitation learning in order instruct update of the guider policy and the agent policy, the agent can leverage suboptimal demonstrations for efficient exploration while avoiding the conservative policy caused by imperfect demonstrations. Empirical results show that HYPO signifi- cantly outperforms several baselines in various challenging tasks, such MuJoCo with sparse rewards, Google Research Football, and the AirSim drone simulation.
online agent policy to explore efficiently. Through mutual
Introduction
Reinforcement Learning (RL) (Sutton & Barto 2018) plays important role in solving real-world control problems with large state-action spaces. In RL, an agent learns a srcision-making policy through interaction with the environment, based learning goal. However, pre-defining reward function relies heavily on human knowledge that varies from to another, and sometimes carefully hand-tuned reward function could lead undesired even hazardous policies (Devidze et al.) 202T). Therefore, a more intuitive way of reward definition utilize the completeness of the task, e.g., the distance that robot walks or scoring goal in football games. While being unbiased towards the learning goal, learning with such sparse rewards could be challenging due to the inefficient exploration in the large problem space.
reward function that indicates the agent's 's
In recent years, there has been intensive research interest in Learning from Demonstrations (LfD) promising approach to addressing sparse reward tasks by leveraging the expert demonstrations. Methods like DDPGID (Vecerík et al. 2017). DQDD (Hester et al.] 2018) and AWAC (Nair et al., 2020) normally require that the demonstrations consist of complete trajectories including state, actions and rewards, and thus are not applicable in real-world applications when it is only possible to observe the state and action information in the demonstrations. Other methods based on Imitation Learning (IL), such as Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon 2016) and Adversarial Inverse Reinforcement Learning (AIRL) (Fu al. 2017), usually require that the demonstrations be generated by an optimal expert, which are difficult to achieve in complex tasks.
More recent approaches such Policy Optimization with Demonstrations (POfD) (Kang al.
Conresponding author. Code is available at httbtggs///ggtuuu...oo//oo//////
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

2018) and Learning Online with Guidance Offline (LOGO) (Rengarajan et al. 2022) can result in an excessively conservative policy (Nair al. 2020), due to the negative impact of imperfect demonstrations. Specifically, when the agent performs better than the suboptimal expert policy, simply using the demonstration-gul guided reward or distilling knowledge (e.g., POAD and LOGO) from the suboptimal policy will cause an excessively conservative policy of the online learning agent.
In this paper, we propose HYbrid Policy Optimization (HYPO), a novel RL algorithm that is capable of achieving near-optimal performance in sparse reward environments using only a small number of imperfect demonstrations with low quality and incomplete trajectories. The basic idea is to learn an offline imitation policy (i.e., the guider) to guide the learning of the agent and an online learning policy (i.e,, the agent) to interact with the environment. These two policies are updated mutually to help the agent learn from both the expert demonstrations and the environment more efficiently. The key insight of HYPO that, the guider imitates the expert to guide the learning of the agent during the initial stage, and then the agent learns to outperform the expert through online interactions with the environment. We evaluate HYPO on wide variety of control tasks, including MuloCo with sparse rewards, Google Research Football and AirSim UAV simulation. Besides, we also investigate how the number and quality of the demonstrations can influence the performance of HYPO and other baseline methods. The results show that 1) HYPO can learn successful policies in these challenging tasks efficiently, and 2) HYPO is minimally affected by the number and quality of the demonstrations.
2
Related Work
2.1 Offline RI.
Offline RL. (Levine et al. 2020) enables to learn policies using only offline data, without any online interaction with the environment. However, the related offline RL methods, such as BCQ (Fujimoto) et al. 2019), BEAR (Kumar et al.) 2019). BRAC (Wu et al.] 2019), and Fisher-BRC (Kostrikov et al., 2021 still suffer from the issue of distributional shift, i.c., the policy is trained under one distribution but evaluated on a different distribution. A common way to mitigate this issue is to restrict the policy distribution be similar to the policy that generates the offline data. However, this constraint can result in excessively conservative policies. In contrast, HYPO can avoid the distribution shift problem since the offline trained guider has no interaction with the environment, but only used to guide the online agent for more efficient exploration. In addition, unlike the existing offiine RL approaches that normally require a large number of data with complete trajectories, thus have limited applications when collecting a large number of perfect demonstrations is either time-consuming or even infeasible, HYPO is able to learn a near-optimal policy given only small number of incomplete demonstrations generated by suboptimal policy.
2.2 IL
IL methods are designed to mimic the behaviors of experts. As one of the most well-known IL algorithms, Behavior Cloning (BC) learns a policy by directly minimizing the discrepancy between the agent and the expert in the demonstration data. However, BC has been shown to suffer from issues such as compounding errors and the inability to handle the distributional shift |Ross al.) (20TT). As a recent BC-based method, DWBC Xu et al. (2022) trains a pure offline BC task to attain the expert performance by using discriminator that distinguishes the optimal expert data and the suboptimal data. Different from DWBC, HYPO can Iearn policies that outperform the expert by using online interaction with the environments and offline guidance from the guider. Moreover, unlike BC-based methods that always require optimal demonstrations, HYPO is capable of learning with only imperfect demonstrations. As another type of IL, Inverse RL (IRL) (Ng et al., 2000) learns a reward function that explains the expert behavior and then uses this reward function to guide the agent's learning process. Some popular approaches use adversarial training to train a policy that is similar to the expert policy, but at the same time is robust to distributional shift, such as the GAIL (Ho & Ermon, 2016) and AIRL (Fu et al. 2017). However, unlike HYPO, all these methods assume that the expert actions be optimal.

---

2.3 LID
The goal of LfD is to use demonstration data to aid agent online learning, so as to overeome the exploration problem in RL, especially in environments with sparse rewards Schaal (199)). DQID (Hester al. 2018) and DDPGfD (Vecerik al., (017) rely on complete offline expert demonstrations with associated rewards to accelerate the online learning process in discrete and continuous action spaces, respectively. POfD Kang et al. (2018) combines the benefits of both imitation learning and RI. to learn policies from both demonstration data and online interactions by learning from shaped reward averaged over the original environment reward and a demonstration- guided reward. However, as the environment reward is much sparser than the demonstration -uuiddddnggddd reward, such a mixed reward may cause the learning objective to deviate from the goal of the optimal policy, Another LfD approach is LOGO (Rengarajan et al. 2022) which pre-trains expert policy to guide the learning of agent, and uses a trust region controlled by hyper parameeer to restrict agent learning. Unlike LOGO that the final policy may be excessively conservative due to the fixed suboptimal expert, HYPO enables the agent to outperform the expert through updating the guider and the agent mutually.
3 Preliminaries
3.1 Markov Decision Process
We consider the standard Markov Decision Process (MDP) (Sutton & Barto 2018 the mathe- matical framework for modeling sequential decision-making problems, which is defined by tuple (S, AA do 7). where S is finite set of states, A is finite set of actions, P S A x S + IR is the transition probability function, r S + R is the reward function, dg is the initial distribution, and [0, 1) is the discount factor. An agent interacts with the environment based policy x A + [0, 1]. which maps from state to distribution over actions. The performance of i is evaluated by its expected discounted reward J(T):
Σ Y'r (sc. ac)
( Ex [r(s,a)] E (s0.000
(1)
where (so, ag, S1....) is a trajectory generated from interaction with the environment, i.c., so ~ do, at T (-][s;) and 8t41 P(-Ist, az). The value function of a policy is defined V=
Erar |E 7 ( (s) = s]. i.e. the expected cumulative discounted reward obtained by the
policy from the state Correspondingly, the action-value function is defined (=, E,sz |E00 ('r(st, dt)[su s, ag = a). and the advantage function is A (s, a) = The objective of RL algorithm is to discover the optimal policy x² that can maximize the expectation of discounted return ((T), i.e., n J
3.2 Learning with Offline Demonstrations
BC and GAIL are well-known baseline methods for learning with offline demonstrations. The difference that BC is a pure offline method without any interactions with the environment, while GAIL is an online learning method through generating and learning from the trajectories.
BC. BC is used for matching the distribution over actions, with its objective given by:
C
IE
log (a]s)],
(s.0)~D
where D is the offline demonstrations. Optimizing objective 2 will make the distribution of actions Ty match the one in the expert policy that generates the state-action pairs in D.
The discriminator aims to distinguish between the offline expert demonstrations and the trajectories sampled by online agent. The objective of the discriminator in GAIL is given by:
GAIL.
L LA-GAIL min E (s.a)>D
log d(s, a))
E
log (1-d(s.a) d(s.a)))
d(s
(s.a)~B
where is the offline demonstrations, B in the replay buffer consists of trajectories sampled by the agent, is the discriminative classifiers d S A (0, 1). Optimizing objective (3) will
3

---

make the learned discriminator assign 1 to all transitions from the offline demonstration D and 0 to all transitions from B. During training, the discriminator will assign a large reward to agent when d(s,a)=B($, a) is close to 1, which is inspired by the generative adversarial training.
The discriminator can be interpreted as new reward function r(s, a) = log (1 - d(s, a)) providing learning signal to the agent. In this way, the agent will receive larger reward if its trajectories are more like the expert, leading to against the discriminator.
4 Hybrid Policy Optimization with Imperfect Demonstrations
Before presenting detailed descriptions of our method, we use Figure to illustrate the over- all motivation. In the early learning stage, the agent usually suffers from pointless exploration (i.e., brown curves) due to the sparsity of the re- ward signal. In such stage, the agent is generally worse than the expert, and thus can gain benefi- cial guidance from the expert demonstrations to attain relatively fair performance that is close to the expert. In the later learning stages, we expect the agent to surpass the expert through exploration in the environment, hence being too close to the expert provement. However, without constraining the deviation to the expert, the agent still en- counter performance collapse due to the point- less exploration caused by the sparse reward. We tackle these challenges by updating the two policies of the guider and agent mutually, to provide the agent with an appropriate guidance. In this way, the agent able to outperform the expert to achieve a near-optimal performance.
random agent
suboptimal expert
RL exploration
expert guidance
Reward
State Space
hinder the policy im-
Figure 1: Our proposed method provides a suboptimal expert direction (red curve) for the agent, which prevents it from pointless exploration (brown curves) under sparse reward environments.
Our goal is to develop algorithm that leverage the imperfect demonstrations generated by suboptimal expert to boost online learning in sparse reward environment. We assume that there is a suboptimal expert policy ne but we have no access it. All we can use are its demonstrations, which have the form D = {r') }r where T' = (si; of................ si, di),7* Te- And we make the following reasonable and indispensable assumption concerning the caliber of the expert policy:
Assumption the initial learning stages, the agent can get higher advantage values than current policy π. if it acts according to the expert policy Te.
Eag (A,(8, - (s,
The above assumption implies that taking actions according to the expert will provide a higher advantage than taking actions according the current policy This is reasonable since the expert policy perform much better than an untrained policy in the initial learning stages.
To better illustrate the analysis and simplify the formulation below, we use the term "expert" to indicate the suboptimal expert unless using the term "optimal" as the prefix. The offline and the online policy is respectively denoted generated by the suboptimal expert is denoted as D, while the replay buffer consisting of trajectories sampled by the agent is denoted as B. The output of the descriminator is d S x A + (0, 1).
Tb and â. The demonstrations consisting of trajectories
The three key components in HYPO are the discriminator, the offline guider, and the online agent. The discriminator is used to control the learning objective of the offline guider by distinguishing whether the trajectories are generated by the expert or the agent. The offline guider performs BC task to provide the agent an adaptive guidance by learning from the expert and the agent dynamically. The online agent policy interacts with the environments and distills knowledge from the guider policy constantly to outperform the expert.

---

4.1 Semi-supervised Discriminator Learning
The discriminator GAIL mentioned in Subsection 3.2 suffers from the overfitting problem, since the well- discriminator may overfit to the suboptimal expert (Zolna et al. 2021). Concretely, as the agent policy improves, the behaviors of the agent become increasingly similar to those of the expert. However, due to the suboptimality of the expert, it is crucial to avoid overfitting to the suboptimal expert, as it can hinder the agent from converging to the optimal performance. To address this issue, we formulate the discriminator objective as a positive-unlabelet (PU) reward learning problem (Xu & Denil, 202T). which allows us train the discriminator by treating the agent trajectories as unlabeled mixture of the expert trajectories and the agent's trajectories. In this way, the guider can also learn from the agent since the agent's 's trajectories are not just the negative data, but the mixture of positive and negative data. The formulated discriminator objective is given follows:
min q d (a,a)~D
E log d(s.u))
E
[-log (1-d(s.
E
- log (1- -(, d)); (5)
(e,0)~B
(e,a)~D
where D denotes the offline demonstrations, B the replay buffer consisting of the trajectorics sampled by the agent, and 1) E [0, 1] the positive class prior, which is assumed to be known.
Furthermore, it difficult to distinguish between the expert's '' and agent's trajectories by only relying on the information of state-action pairs. To mitigate this issue, an additional signal log mi is added into the input of the discriminator, leading to the final objective of the discriminator as follows:
Ld 1
E (s,0)~D
log d(s,, a, lo,og mb)
+
E
[ log (1-d(s, d(ss, a, log (ogg)))]
(s.a)~
E (s,a)~
log
d(ss, a, log mo)
Specifically, when considering state-action pairs (s, a) originating from the agent's trajectories, (a(as) will assign large probabilities to the agent's actions within these corresponding states. Conversely, for state-action pairs (s, a) derived from the expert's trajectories, To (a(s) will assign small probabilities the expert's actions within these corresponding states.
Remark on 77. It is worth noting that, unlike the standard positive-unlabeled learning setup, the positive class prior n in our setting changes as the online policy learning progresses and the distribution of states in the replay buffer evolves. Specifically, 17 is set to increase as the learning progresses.
4.2 Adaptive Target for Offfine Imitation
The objective of BC task is to mimic the expert policy through matching the conditional distribution (-Is) over actions. Therefore, the BC methods cannot learn a policy that outperforms the expert without additional guidance. In HYPO, тn is able to use the information from the trajectories sampled by the online policy learn from the agent, which can assist тa to attain better performance than the original expert. Specifically, we use two adaptive functional weights FExpert (d) and GAgent (d)
determine the learning objective of Te dynamically:
log (ogr) F d(ss a, (, log + (e.0)~B
- log (o(a))) loggso(als) 9 d(s., log m.)
C.o (a.0)-D
(7)
where F and G, denoting FExpert (d) and GAgene(d) respectively, are the functional weights to determine the objective of Th. The functional weights are expected to force the T. approach the expert in the initial stage, and to increase L2 to ensure discriminator robust throughout the whole learning process. Since the performance of the online learning policy differs significantly, ranging from being random to reaching the oetimop the discriminator needs to be robust enough to handle these changes. Inspired by the adversarial robustness (Carlini et al., 2019) and the weighted discriminator (Xu et al. (2022), we use to maximize L. to improve the robustness of discriminator by minimizing the worst-case error. Consequently, the form of the functional weights of FExpert and G Agent can be derived as follows:
FExpert (d) = d(1 d)
1)
Aggent (d) = -1-- I-d'
(8)
5

---

where a is weight factor. Refer to Appendix B for the detailed derivation.
To understand the trend of the functional weight FExpert (d) intuitively, we visualize it as illus- trated in Figure 2 For transitions from D, the weight is insensitive to d when is small, since the proportion of positive samples in the (s, a) pairs generated by the agent is low. In this case, Ty can provide the online agent with guidance by learning towards the expert. As training pro- gresses, the proportion 7) of positive samples in agent trajectories gradually increases, thus begins to dominate the weight changes of Exxpert (d). In this case, Fixpert (d) becomes sensitive to d, and gets smaller since the d gets large when (s, a) comes from the expert, resulting in the learning towards the agent for Ty. As for the other weight of Gggnnt which will be large if the (s, a) sampled by the agent स is similar to that of the expert, changes insensitively when d remains low.
Increasing 17
The trend of weight FExpert (d). The deeper blue color means the lower weight value. is increasing during the training and d is various from 0.1 0.9 (clipped).
Figure 2:
4.3 Performance Improvement of Online Learning
useful reward signal is necessary for online RL algorithms learn efficiently. However, in the sparse reward environments, the agent cannot explore states that have non-zero reward without additional guidance. To this end, we use an offline guider to provide the agent with guidance, which can significantly boost the exploration of agent, especially in the initial stages of training. Specifically, the online agent policy á, which learns from the sparse environment rewards, constantly distills knowledge from the guider policy Te. This is implemented by adding another corrective objective on the original PPO (Schulman al. (017) policy update:
JHYPO F. min (0), clip(rr.))) 1, +,
where C' a decreasing coefficient of the KL-divergence. In Eq (9) we append a constraint of the KL-divergence between the guider policy n. and the agent policy т. In this way the agent can distill knowledge constantly from the guider. Besides, the decreasing coefficient of C can prevent the agent from the excessively conservative policy. Next, we theoretically analyze how the additional constraint can influence the agent. We consider the following two cases: case the agent's 's performance is worse than the expert, which corresponds to satisfaction of Assumption 1, and case 2) the agent's s performance better than the expert, which means Assumption is not satisfied.
In the initial stage of training, the offline guider policy can be approximately treated as the expert policy since the BC method is more efficient than policy gradient (PG) and the Fxxpert keeps large. which drives Ty to update towards the expert. Now the performance improvement guarantee of the online agent under the Assumption can be given as follows:
Case 1.
Proposition Let ã be policy that satisfies Assumption Then, for policy f.
Jn(#) J (1- (1-
(1-7)) 7) - 2D Tb),
(10)
where ER,* = max,, (s,
Proof. Refer the Appendix B.
is reasonable to assume that satisfies Assumption the initial stage of learning since the current policy a of agent is learning from scratch. Then, minimizing D (ft, ma) can get a non-negate negaiveee lower bound in Equation (T0), which means that we can accelerate the online learning of the agent by minimizing the KL-divergence between á and he- This process can be described as, the agent learns from the guider to attain the expert performance, which can constantly get non-zero rewards from the sparse environments.
the training progresses, the agent can achieve the expert performance efficiently. This can be realized in most of the previous imitation methods. However, want investigate that, how
Case 2.

---

Hopper
Walker
Football
Half Cheetah
Ant
AirSim
Figure 3: MuJoCo (left) a set of popular continuous control environments with tasks of varying difficulty. Google Research Football (top right) is novel RL environment where agents trained to play football an advance, physics based 3D simulation. AirSim (bottom right) is a new simulator built Unreal Engine that offers physically and visually realistic simulations for both of these goals.
the agent can surpass the expert and achieve the near-optimal performance in sparse reward tasks, which corresponds to do not satisfy the Assumption 1. If the guidance simply removed from the agent, in other words, the agent no longer distills knowledge from the guider, the bootstrap error and the uncertain reward signals policy to restrict the agent, like other LID methods, the agent policy will be excessively conservative, making it hard for the agent achieve the near-optimal performance. Therefore, we address this issue by changing the learning objective of guider from expert to the agent, which will reduce the KL-divergence between my and â. The policy improvement lower bound when Assumption is satisfied is given as follows:
reduce the performance of agent policy. If
the expert
Proposition 2. For policy â and any policy 7,
3Rmax 2(1 7)2
JR(T) - JR(
2D (t,
(11)
where Rmax max R(s, a)l-
Proof. Refer to the Appendix B.
The guider update towards the agent will reduce the KL-divergence item to improve the policy improvement bound in Eq (TT).
The mutual update of the guider policy and agent policy is the core feature behind the HYPO algorithm. In this way, the agent can leverage few suboptimal demonstrations for efficient exploration while avoding the negative impact of the low-quality data.
5 Experiments
In this section, we investigate whether HYPO can achieve near-optimal performance in extremely sparse reward environments by overcoming the restriction of imperfect demonstrations, and how the number and quality the trajectories can influence the performance of HYPO. To comprehensively our method, we first perform an exhaustive evaluation of HYPO in MuJoCo (Todorov al.)
2012) with sparse rewards and Google Research Football (GRF) (Kurach et al. 2020) with huge policy Unmanned Aerial Vehicle (UAV)
space and only sparse score reward. We also evaluate HYPO
task based on the Unreal Engine and AirSim (Shah et al. 201 8) to show the effectiveness of HYPO
addressing more challenging control tasks with high-fidelity. All the mentioned environments are shown in Figure

---

HYPO[ours)
LOGO
GAIL
POID
PPO
Expert
Demo
Performance (Scaled)
100
100
100
100
75
75
75
50
50
25
25
0
M
M
0
M
0
2
3
4
5
6
6 Walker2d
8
10
2 4
10
1
4
Hopper
HalfCheetah
Ant
Figure 4: MuJoCo simulation results. The r-axis is the number of samples. The y-axis is the average episode return, which is scaled to make the expert achieve 100 and random policy achieve
HYPO(ours)
LOGO
GAIL
POID
Hopper
Walker2d
HalfCheetah
Ant
Performance (Scaled)
100
100
100
100
75
75
75
50
50
25
25
25
0
10
20
50
5
10
20
50
5
10
20
50
10
20
50
Number trajectories in demonstrations
Porformance (Scaled)
100
100
100
75
75
50
50
50
50
25
25
25
10
20
30
40
10
20
30
40
10
20
30
40
10
20
30
40
Performance of trajectories demonstrations
Figure 5: Final performance the baseline algorithms that are trained with various number and performance of trajectories in demonstrations. HYPO is minimally affected by the quality of data.
5.1 MuJoCo Simulation
We first sparsify the built-in dense rewards on the MuJoCo platform for evaluating the methods in sparse-reward environments. Specifically, reward of +1 is provided only after the agent moves forward over specific distance. We compare HYPO to the following baselines: (1) Expert, which applies PPO using the original dense rewards to achieve the optimal return; (2) Demo, which is suboptimal expert the early stage of training; (3) PPO, which directly trains PPO with sparse rewards; (4) GAIL (Ho & Ermon 2016), which uses discriminator to provide demonstration- guided reward for training; (5) POID (Kang et al. 2018), which uses a weighted combination the environment reward and the demonstration-guui guidedd reward; and (6) LOGO (Rengarajan et al. (2022). which merges policy improvement step and a policy guidance step. All the demonstrations used for training generated by Demo. Refer Appendix C for more experimental details.
As shown in Figure 4 PPO fails to learn an useful policy all environments due the lack of guidance from demonstrations. GAIL can only achieve the level of the suboptimal expert policy since it can only mimic the policy that generates the demonstrations. POfD can attain higher return than GAIL in HalfCheetah but similar return in other tasks. LOGO can achieve relative better performance due its decaying trust region that constrains the policy to gradually get rid of the influence of suboptimal data. However, the conservative policy can still impede the learning efficiency during the later learning process. As expected, HYPO outperforms the above methods in all environments by a large margin, which fully validates the effectiveness of HYPO in learning

---

near-optimal performance in extremely sparse reward environments by overcoming the restriction of imperfect demonstrations. We also investigate the influence of the number and quality (in terms of cumulative return) of demonstrations on the final performance. As shown in Figure HYPO capable of achieving far better performance than other methods, even with small number of imperfect trajectories. Refer to Appendix F for more details about the performance of LOGO.
5.2 Google Research Football and AirSim Simulation
The GRF task to control single player to cooperate with teammates to break specific de- fensive line formation and score. This environ- ment provides two type of reward settings, i.e.,
Expert
HYPO
Demo
PPO
1.0
50
Avg Goal Differene
0.8
40
the sparse score reward, and the dense check- points reward. AirSim
high-fidelity environ-
0.6
30
Avg Return
ment where we need to control an UAV to fly the terminal while avoiding all the obstacles. This task also provides the sparse reward set-
0.4
20
ting for flying a specific distance and the dense reward setting for keeping a certain speed of for- ward flight. The expert baseline trained in the dense reward setting, while the imperfect demo data is generated by a partially trained expert. The results in Figure [6] show that HYPO can achieve near-optimal performance in these chal- lenging tasks with imperfect demonstrations.
0.2
10
0
Run to Score
vs with Keeper
Pass and Shoot with Keeper
AirSim Simulation
Figure 6: Results in GRF and AirSim. The left y-axis is for GRF and the right for AirSim.
6 Conclusion and Outlook
In this paper, we investigate how to accelerate agent online learning with imperfect demonstrations in sparse reward environments. We introduce HYPO, novel RL algorithm that can attain near optimal performance in sparse reward settings by avoiding the excessive conservative policy. The highlight of HYPO lies in its capability learn an offline guider and an online agent, while updating these two policies mutually to help the agent learn from the demonstrations and environments more efficiently. Experiments in various environments including MuJoCo, Google Research Football and AirSim UAV simulation demonstrate that HYPO can greatly promote the learning efficiency in sparse reward tasks with imperfect data. Our future work is to extend HYPO to multi-agent scenarios when agents need to learn coordinated policies with sparse rewards. Moreover, plan to investigate the potential of HYPO in more real-world applications, such as the autonomous driving, where sparse rewards are common challenges.
Acknowledgments and Disclosure of Funding
This work was supported by an SYSU-ByteDance Research Project.
References
Carlini, N., Athalye, A., Papernot, N., Brendel, W., Rauber, J., Tsipras, D., Goodfellow, I., Madry, A., and Kurakin, A. On evaluating adversarial robustness. arXiv preprint arXiv: Ntt.iv:9992.060000 2019.
Devidze, R., Radanovic. G., Kamalaruban, P., and Singla, A. Explicable reward design for rein- forcement learning agents. Advances in Neural Information Processing Systems, 34:201 10-2013, 2021.
Fu. J., Luo, K., and Levine, S. Learning robust rewards with adversarial inverse reinforcement learning, arXiv preprint arXiv:1710.1122: 2017.
Fujimoto, S., Meger, D., and Precup. Off-policy deep reinforcement learning without exploration. In International conference machine learning, pp. 2052-2062, PMLR, 2019.

---

Hester, T., Vecerik, M., Pietquin, O., Lanctot, M.. Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband. I., al. Deep q-learning from demonstrations. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Ho, J. and Ermon, S. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Kang, B., Jie, Z., and Feng, J. Policy optimization with demonstrations. In International conference on machine learning, pp. 2469-2478. PMLR, 2018.
Kostrikov, I,, Fergus, R., Tompson, J., and Nachum, O. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783. PMLR, 2021.
Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, Stabilizing off-policy q-learning via bootstrap- ping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Kurach, K., Raichuk, A., Stañczyk, P., Zajac, M., Bachem, O., Espeholt, L., Riquelme, C., Vincent, D., Michalski, M., Bousquet, O., et al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pp. 4501-4510, 2020.
Levine, S., Kumar, A., Tucker, G., and Fu, Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01644: 2020.
Nair, A., Gupta, A., Dalal, M., and Levine, S. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
Ng. A. Y., Russell, S., et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, pp. 2, 2000.
Rengarajan, D., Vaidya, G., Sarvesh, A., Kalathil, D., and Shakkottai, S. Reinforcement learning with sparse rewards using guidance from offline demonstration. arXiv preprint arXiv:2202.04628, 2022.
Ross, S., Gordon, G., and Bagnell, D. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-637-66 635.. JMLR Workshop and Conference Proceedings. 2011.
Schaal, S. Learning from demonstration. Advances in neural information processing systems, 9, 1996.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization 2017. 2017.
algorithms. arXiv preprint arXiv:1707.06347
Shah, S., Dey, D., Lovett, C., and Kapoor, A. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and Service Robotics: Results of the 11th International Conference, pp. 621-63366555 Springer, 2018.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
Todorov, E., Erez, T., and Tassa, Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, 5026-5033 IEEE, 2012.
Vecerík, M., Hester, T., Scholz, J., Wang, F., Pietquin, O., Piot, B., Heess, N. M. O., Rothörl, T., Lampe, T., and Riedmiller, M. A. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. ArXiv, abs/1707.0885 08817, 2017.
Wu, Y., Tucker, G., and Nachum, O. Behavior regularized offline reinforcement learning. arXiv preprint /9/ 11361,11111111111111111111111 2019.
Xu, D. and Denil, M. Positive-unlabeled reward learning. In Conference on Robot Learning, pp- 205-219. PMLR, 2021.
10

---

Xu, H., Zhan, X.. Yin, H., and Qin, H. Discriminator Discriminator-weighter offline imitation learning from suboptimal demonstrations. In International Conference on Machine Learning, pp. 24725-24742. PMLR, R, 2022.
Zolna, K., Reed, S., Novikov, A., Colmenarejo, S. G., Budden, D., Cabi, S., Denil, M., de Freitas, N., and Wang, Z. Task-relevant adversarial imitation learning. In Conference on Robot Learning, pp- 247-263 PMLR, 2021.


Reviews:
[{"review_id": "9UCkEVryWq", "reviewer": "Reviewer_8vGE", "summary": "The goal of this work is to develop an approach for RL in the sparse reward setting, wherein the RL agent is directed by a guidance policy that provides direction to the online learner.  While this approach has been used before, the main contribution of this work is on developing a guidance policy that utilizes both offline data gathered by a behavior policy and online data gathered by the learning agent.  KL-Regularization with respect to this guidance policy is used to guide the learning agent, with the level of regularization being reduced as the learning proceeds so that the policy learned can be optimal.   Some theoretical justification is provided, following in much the same manner as LOGO (Rengarajan et al. 2022) that shows that the guidance procedure will extract an advantage from the offline data if it is possible to do so.", "questions": "Please see negatives above.  I think this is an interesting idea, and would like to see your responses to the decay question in particular.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Interesting idea on how to bring in guidance using both offline and online data.\n\n+ The guidance policy should smoothly go from a pure BC of the data to being identical to the online policy.  This should help in reducing excessive dependence on the guidance policy as in LOGO.", "weaknesses": "- The KL regularization coefficient is indicated as decaying with time.  If the guidance policy truly converges to the online policy, the coefficient should not actually need decaying at all.   Indeed this is much the same as the excessive conservative behavior that the authors indicate is a problem with LOGO.\n\n- It is also unclear how exactly the regularization is undertaken or how the coefficient is decayed.  Is this tuned up by hand for each environment?  That would be an unfair comparison with the other approaches where the decay rate of guidance is implicitly chosen by the algorithm itself.\n\n- In some of the experiments (HalfCheetah), HYPO does better than the upper bound (PPO trained with a dense reward).  How is this possible?\n\nSmaller issues:\n- The analytical part is essentially identical to LOGO, although that is not a major contribution of the paper.\n\n- The second set of figures on page 5 have the x-axis missing the M (which I guess stands for million samples).  They can't possibly train in 40-50 samples", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and valuable feedback. We explain the concerns point by point below.\n\n**Q1: About the decaying of the KL regularization coefficient.**\n\nA: The primary purpose of the regularization term here is to facilitate knowledge distillation from the guidance policy to the online policy. We opt for a decaying approach of the regularization term to avoid introducing an overly conservative policy to the online agent. As the performance of the online policy surpasses that of the expert, the guidance policy converges to the online policy gradually, and thus there is no need to decay the coefficient anymore. Two aspects, however, require further clarification:\n\n1. First, HYPO does not induce an excessively conservative policy akin to that observed in LOGO. In LOGO, the guidance is provided through pre-trained policies using Generative Adversarial Imitation Learning (GAIL), indicating a fixed proficiency level of the guidance policy. This, in turn, imposes significant limitations on the subsequent enhancement of the online policy.\n2. Although the coefficient of the regularization term remains constant during the later stages, it remains a relatively low value, specifically 0.1 for all environments. This is informed by the necessity to avoid potential performance collapse due to unstable exploration under sparse rewards, as illustrated in Figure 3 (the PDF in author rebuttal).\n\n**Q2: About how exactly the regularization is undertaken or how the coefficient is decayed.**\n\nA: The regularization coefficient linearly decays from 1.0 to 0.1 across all environments in the paper,  with a step size of 2M. We do not manually adjust this coefficient for each environment, since we only need to ensure that the regularization term loss and actor loss are of the same magnitude, similar to the control of entropy loss coefficient.\n\n**Q3: About the results that HYPO does better than PPO trained with dense reward.**\n\nA: In certain experimental environments, we indeed observe that agents trained through imitation learning can outperform the expert policies trained under dense rewards. This result can be also observed in the following cases: 1) Figure 1 in the LOGO paper; 2) Figure 3 in the POfD paper; and 3) Figure 1 in the GAIL paper. We hypothesize that this result can be attributed to the following reasons:\n\n1. Mismatch in Reward Signals: During the training of HYPO, only the primary long term reward signal, i.e., forward movement, is employed. However, in the original HalfCheetah environment, additional dense rewards such as control cost are present. While in specific environments, these extra reward signals might aid agents in mitigating the challenges posed by sparse rewards, expediting convergence, there are still scenarios when these additional signals might not be consistent with the long term performance. \n\n2. Approximate Optimal Policies: Policies that converge under dense rewards may not necessarily represent optimal solutions. The convergence of Policy Gradient (PG) methods like PPO is influenced by several factors, such as exploratory aspects (entropy). Hence, different combinations of parameters and components might impact the convergence performance of the algorithm in various ways. While we have selected a set of reasonably suitable parameters and maintained consistency between the training of PPO and HYPO, HYPO incorporates some additional parameters compared to PPO. It is possible that, under the combination of these parameters, HYPO manages to explore more efficiently than PPO.\n\n\n**Q4:  About the analytical part compared with LOGO.**\n\nA: The difference of the theoretical analysis between HYPO and LOGO lies in the fact that the guidance policy imitates both the expert and the agent to provide guidance. Specifically, in the initial stages, the analysis is similar to LOGO since the online agent is learning from scratch and needs to get guidance from the demonstrations. Once the online policy surpasses the performance of the demonstration, the offline guider policy aligns itself with the online policy to enhance the policy improvement bound by decreasing $D_{KL}$ :\n\n$$\nJ_R(\\hat\\pi)-J_R(\\tilde\\pi)\\ge -\\frac{3R_{\\rm{max}}}{2(1-\\gamma)^2} \\sqrt{2D^{\\max}_{\\rm{KL}}(\\hat\\pi,\\tilde\\pi)}. \\tag1\n$$\n\nIn LOGO, the authors manually control the trust region $\\delta_k$ for policy updates:\n\n$$\nJ_R(\\pi_{k+1})-J_R(\\pi_k)\\ge-(\\sqrt{2\\delta}{\\gamma\\epsilon_{R,k}}+3R_{\\max}\\delta_k)/(1-\\gamma)^2, \\tag2\n$$\n\n which results in overly conservative policies and unstable convergence.\n\n**Q5: About the x-axis in Figure 5.**\n\nA: The x-axis in Figure 5 does not lack 'M'. The x-axis indicates \"the number of trajectories in demonstrations\", with each trajectory containing approximately 1k transitions (samples). These samples are indeed sufficient for HYPO's training. As the offline guider in HYPO does not interact with the environment and is responsible solely for providing guidance to the online policy, issues such as distribution shift (out of distributions) commonly seen in offline learning due to inadequate sample size can be readily addressed. This serves as one of the major advantages of HYPO – it requires only a **small amount** of **suboptimal** trajectories, facilitating easy deployment and efficient application."}, {"review_id": "nNYJTPLO0q", "reviewer": "Reviewer_8Gvz", "summary": "This paper presents HYPO, a Learning from demonstration (LfD) method to effectively learn policy from a limited amount of imperfect demonstrations. At the core of the method are a hybrid discriminator to tell positive/negative state-action pairs through positive-unlabeled reward learning, an offline imitation learning policy to adaptively guide the policy learning to learn from the mix of expert and agent demonstrations, and an online policy learning extended from PPO to distill the knowledge from the offline policy. The method is evaluated on three benchmark datasets and shown to consistently outperform baselines.", "questions": "1. Are the agent demonstration $\\mathcal{B}$ sampled from online agent or offline agent? From the context, it seems that they are sampled from online agent. However, in the discriminator loss function (Eq (6)), offline BC policy is used to provide the action probability $\\pi_b(a|s)$. Could the author explain the mismatch here? Also will the inclusion of $\\pi_b$ in Eq (6) result in a discriminator partially overfit to the $\\log \\pi_b$ and ignore $s, a$ (at least for the samples from $\\mathcal{B}$)?\n\n2. What is the exact schedule of $\\eta$ and the value of $\\alpha$? How is the algorithm sensitive to them? If so, is there empirically easy way to determine them?\n\n3. What is the insight to have both an offline imitation learning policy and an online RL+imitation policy? Can we skip the offline imitation learning policy and only keep the online policy learning by modifying online policy update reward (Eq(9)) to include the dynamic imitation loss of Eq. (7)?\n\n4. What is the secret sauce that the authors think is the key to the success of improving upon demonstrations? The learning of the discriminator and the offline policy is unware of which trajectory produced by the learned agent is positive or negative since it purely relies on a positive class prior $\\eta$. Is it possible that the discriminator learns a wrong quality assignment thus confuses the learning of the offline policy with wrong $\\mathcal{G}$?\n\n5. The author claims the learned policy is near-optimal for several times across the main text, which is a too strong claim without strict evidence and might mislead the reader and oversell the paper.\n\n6. How is the method's performance on more contact-rich manipulation tasks? Such as the [robosuite](https://robosuite.ai/) benchmark?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea of improving learned policy beyond imperfect demonstration via treating policy rollout as unlabelled mix of positive and negative demonstration is insightful.\n\n2. The method is tested on both simple and complex control tasks with low-dim and high-dim observation spaces. The learned policy can surpass the demonstration performance. The empirical performance of method over baselines is significant and convincing.\n\n3. The ablation study on the number of trajectories is interesting and shows the robustness of the approach even on a few imperfect expert trajectories.", "weaknesses": "1. The exposition of the paper can be improved. The method part is not clear.\n    \n    a) In section 4.1, the authors use the word **agent policy** to refer to both online agent policy and offline agent policy.\n\n    b) What is the discriminator's robust mean in section 4.2? and how $\\pi_b$ improves the discriminator's robustness is unclear (the corresponding part in Appendix is hard to follow).\n\n2. The necessity of the offline policy is not clear.\n\n3. More ablation study of the hyperparameters is necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and valuable feedback. We explain the concerns point by point below.\n\n**Q1: About the presentation and confusion of notations.**\n\nA: In the paper, the **guider** is trained using **offline imitation learning**, and thus, we use the term offline imitation policy (guider) to indicate it. The **agent** is trained using **online reinforcement learning**, and we use the online learning policy (agent) to indicate it. We will proofread the paper thoroughly in order to keep consistent of these terms.\n\n**Q2: About the  robustness of the discriminator.**\n\nA: The robustness of the discriminator refers to its ability to accurately differentiate between positive and negative samples even when there is a significant variation in the performance of the input samples. This is particularly relevant since the performance of the online policy is consistently improving, causing substantial disparities in the quality of the input samples. We make $\\pi_b$ challenge the discriminator $d$ by doing the opposite to minimizing $\\mathcal{L}_d$, and this can be seen as minimizing the worst-case error ([Carlini et al., 2019](https://arxiv.org/abs/1902.06705); [Fawzi et al., 2016](https://proceedings.neurips.cc/paper/2016/hash/7ce3284b743aefde80ffd9aec500e085-Abstract.html); [Goodfellow et al., 2015](https://arxiv.org/abs/1412.6572)).\n\n**Q3: About the necessity of the offline policy.**\nA: The offline policy is a crucial component of our method and plays a critical role in achieving the final performance:\n\n1. In the initial training stage, the offline policy emulates the behavior of the expert policy and guides the learning of the online policy. Without the guidance of the offline policy, the online policy would struggle to learn meaningful policies in environments with such sparse rewards.\n2. Throughout the training process, the offline policy dynamically selects imitation targets. As the performance of the online policy surpasses that of the expert, the offline policy adjusts to imitate the online policy to stabilize the online policy's updates, preventing the emergence of overly conservative policies.\n\n**Q4: More ablation study of the hyperparameters.**\n\nA: We conducted additional experiments to perform ablation studies on the hyperparameters. The results of these experiments can be observed in Figures 1-3 (PDF in author rebuttal).\n\n**Q5: About the $\\pi_b(a|s)$ in Equation 6.**\n\nA: First, $\\mathcal{B}$ is sampled from the online agent policy. Regardless of whether $(s,a)$ comes from $\\mathcal{B}$ or $\\mathcal{D}$, the discriminator consistently treats $\\pi_b(a|s)$ as an additional signal to aid in its training. Specifically, if $(s,a)$ coms from $\\mathcal{B}$ (online agent), the expected value of $\\pi_b(a|s)$ is higher due to the policy generating $(s,a)$ and $\\pi_b$ being closely aligned with each other, resulting in a small KL divergence between them. Conversely, if $(s,a)$ comes from $\\mathcal{D}$ (expert demonstrations), the expected value of $\\pi_b(a|s)$ is lower since $\\pi_b$ significantly deviates from the expert policy responsible for generating the demonstrations. Consequently, the input to the discriminator in Equation 6 includes $(s,a,\\pi_b)$. \n\n**Q6: About overfitting to $\\log\\pi_b$ in discriminator.**\n\nA: If the discriminator in HYPO overfits to $\\pi_b(a|s)$, it can indeed impact the accuracy of the discriminator. In order to demonstrate the role of $\\pi_b$, we have provided a comparison of the discriminator's performance during training. See the results in Figure 4 (PDF in author rebuttal).\n\n**Q7: About the hyperparameters $\\eta$ and $\\alpha$.**\n\nA:  Please refer to the author rebuttal and Figures 1,2 (PDF in author rebuttal) for details.\n\n**Q8: About the insight of using both an offline imitation learning policy and an online RL policy.**\n\nA: The key insight of our method is to harness the guidance from the offline imitation policy to aid the learning of the online policy while avoiding an excessive conservative impact and facilitating convergence. If we solely rely on online agent imitation learning without the guider  (offline imitation policy), the online policy can be excessively constrained by the expert demonstrations, leading to overly conservative updates and failure to converge to an approximate optimal level.\n\nRegarding the combination of updates in Equation 9 with Equation 7, we acknowledge it as a promising avenue that could be explored in future research. However, it appears unfeasible within the existing HYPO framework.\n\n**Q9: About the secret sauce of improving upon demonstrations.**\n\nA: The discriminator can learn to distinguish trajectories solely relying on the original loss function (3) in this paper. The PU-Learning and $\\eta$ are introduced to mitigate the overfitting problem, by re-weighting the losses for positive and unlabeled data. Therefore, the discriminator can learn to distinguish these trajectories correctly. The accuracy of the discriminator is shown in Figure 4 (PDF in author rebuttal).\n\n**Q10: About the claims of near-optimal performance.**\n\nA: Indeed, the usage of 'near-optimal' might appear too strong. The reason behind this claim in the paper was the observation that the convergence performance closely aligns with that achieved under dense rewards. We will revise this claim in the next version of the paper.\n\n**Q11: About the performance of HYPO on robosuite.**\n\nA: To evaluate the algorithm's performance in robosuite environments, we conducted additional experiments, and the results of which are illustrated in Table 1 (PDF in author rebuttal) and the last section in author rebuttal. "}, {"review_id": "L7FmVcEhxS", "reviewer": "Reviewer_74Fn", "summary": "The paper introduces a method (HYPO) for RL using LfD, specifically in the setting where available demonstrations are suboptimal and/or incomplete. The method maintains a discriminator trained to differentiate agent-generated $(s,a)$ pairs from those available as demonstrations. The method further trains two policies simultaneously – an offline policy $\\pi_b$ that is trained dynamically using behavior cloning on demonstration $(s,a)$ samples initially, and on those generated by the agent’s policy $\\pi_{\\theta}$ as the agent’s policy improves. $\\pi_{\\theta}$ is updated by minimizing a PPO-style, clipped-advantage loss function that also penalizes large deviations in $\\pi_{\\theta}$ from $\\pi_b$ via a KL penalty. Experiments on sparse-reward versions of Mujoco environments show HYPO outperforms competing methods; sensitivity analyses demonstrate robustness to number/quality of demonstrations.   ", "questions": "-\tHow is $r_t(\\theta)$ in Equation (9) computed? Specifically, is the probability ratio computed as $\\frac{\\pi_{\\theta}^{k}(a|s)}{\\pi_{\\theta}^{k-1}(a|s)}$ as is done in the original PPO implementation (Schulman et. al., 2017)? Or is it computed as $\\frac{\\pi_{\\theta}^{k}(a|s)}{\\pi_{b}(a|s)}$ (where $k$ is the iteration.) \n-\tDo the current experiments employ a variant of LOGO that uses a different loss function or methodology from the original implementation (Rengarajan et. al., 2022)?\n-\tIn Assumption 1, equation (4), if $\\pi$ is random policy, is it always guaranteed that $ A_{\\pi} (s,a_{e}) > A_{\\pi}(s,a)$? (given that advantage $A_{\\pi}(s,a)$ is computed under $\\pi$ using $Q_{\\pi}$ and $V_{\\pi}$ which, for a random policy may be arbitrarily poor.)\n-\t(Minor) In practice, does training all three (the discriminator, guider policy and agent policy) simultaneously require significant hyperparameter tuning (given that the estimates of all three are likely to be extremely suboptimal and noisy initially) ?\n", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "-\tThe idea of adaptive target BC to dynamically train the offline (guider) policy is novel and creative. Augmenting the PPO loss function in order to regularize the agent policy towards the guider policy is also an interesting contribution. Together, these ideas leverage available offline demonstration data in a useful way that enables faster policy learning. \n-\tAll ideas are expressed clearly via adequate examples, illustrations, and experiments. The sensitivity analysis is rightly focused on the central premise of the paper—offline demonstration data and its suboptimality.  \n", "weaknesses": "-\tIt has been emphasized that HYPO can learn from suboptimal and incomplete trajectories. While the sensitivity studies show adequate evidence for the former, it is not shown how HYPO can learn from incomplete demonstrations. A discussion of how HYPO’s formulation enables this and perhaps empirical verification could be helpful in this regard.\n-\tLOGO (Rengarajan et. al., 2022) considers a similar problem setting with suboptimal demonstrated behavior with sparse rewards. Their evaluations show that LOGO achieves expert-level performance on the Hopper and HalfCheetah environments. However, LOGO is shown to perform quite poorly in the results presented in the current paper leading to the feeling that the experiments here are perhaps skewed unfairly in favor of HYPO.\n-\tConfusing notation/lacking some consistency in annotating the guider/agent/demonstrator policies and the data samples that obtained from them. For example, it is easy to confuse the samples in $\\mathcal{B}$ as coming from the guider policy $\\pi_b$ (since both use ‘b’). Referring to them as demonstrator and demonstrations (rather than expert and expert demonstrations) might also help in easily differentiating between the two and make it easier to follow along.\n-\t(Minor) Having the algorithm in the main text of the paper (if it can be accommodated) would be helpful.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and valuable feedback. We explain the concerns point by point below.\n\n**Q1: About the incomplete demonstrations.**\n\nA: In our method,  \"incomplete\" indicates that expert trajectories lack reward signals. Specifically, for offline learning, complete trajectories used for imitation typically consist of $(s, a, r, s^\\prime)$ tuples. However, in HYPO, the trajectories used for learning lack the reward signal $r$ and consist only of $(s, a)$ pairs (the discriminator $d(s,a,\\log\\pi_b)$ in Eq.6 ).\n\n**Q2: About the performance of LOGO in this paper.**\n\nA: The experimental results of the LOGO algorithm presented in our paper were directly conducted using the publicly available LOGO [source code](https://github.com/DesikRengarajan/LOGO), without any modifications to its components. Our reproduced results align with the outcomes reported in their paper. The reason for the comparatively weaker performance of LOGO in the figures of our paper is attributed to the limitations imposed by the training samples. In the original LOGO paper, the x-axis reaches **1e7**, which consumes an enormous amount of samples. However, in our experiments, such as the Hopper environment, we only considered a maximum of 6M (x-axis is **1e6**) samples.\n\nFurthermore, although both LOGO and HYPO use suboptimal samples, LOGO's suboptimal samples reach over 60% of the performance of the optimal expert (e.g., in the Walker and HalfCheetah environments). In contrast, HYPO requires suboptimal samples that achieve just 20% or 10% of the optimal expert's performance, indicating that HYPO imposes less stringent requirements on expert trajectories. We will elaborate on this point further in the experimental section of the subsequent versions of the paper.\n\n**Q3: About the confusing notation.**\n\nA: We acknowledge the lack of uniform notation and consistency in certain aspects. We use the symbol $\\mathcal{B}$ to indicate the 'buffer' of the online agent. We will change it to another symbol in order to clear the confusion.\n\n**Q4: About the algorithm pseudo-code.**\n\nA:  Thanks for the comment. We will consider including the algorithm into the main body of the paper in the next version.\n\n**Q5: The computation of $r_t(\\theta)$ in Eq.9.**\n\nA: The computation of $r_t(\\theta)$ in Equation 9 is conducted using the importance weight $\\frac{\\pi^k_\\theta(a|s)}{\\pi^{k-1}_\\theta(a|s)}$, as is standard in the original PPO implementation. In HYPO, the online agent policy learns from the offline guider by exclusively distilling knowledge from the guider, without any additional specialized techniques. This simplicity allows HYPO to be conveniently implemented on PPO or other Policy Gradient (PG) methods. \n\nRegarding the latter form $\\frac{\\pi^k_\\theta(a|s)}{\\pi_b(a|s)}$, it could be employed if $\\pi_b$ interacts with the environment and the trajectories are inserted into the replay buffer of the online agent. This approach bears resemblance to the [IRAT methods](https://proceedings.mlr.press/v162/wang22ao.html) as proposed by Wang et al. in 2022.\n\n**Q6: The implementation of LOGO.**\n\nA: Please refer to the response of Q2 above.\n\n**Q7: About the guarantee of $A_\\pi(s,a_e) > A_\\pi(s,a)$ in Assumption 1.**\n\nA: Intuitively, Assumption.1 implies that taking action acording to $\\pi_e$ will provide a higher advantage than taking according to $\\pi$ **since $\\mathbb{E}_{a\\sim\\pi}[A_\\pi(s,a)]=0$**. If the action is taken according to $\\pi_e$ (e.g., $a_e$), the expectation $\\mathbb{E}_{a_e\\sim\\pi_e}[A_\\pi(s,a_e)]$ will be larger than 0 (the expectation is necessary, otherwise the assumption cannot be guaranteed).\n\n**Q8: About the hyperparameter tuning.**\n\nA: In fact, tuning the parameters of this method is particularly straightforward. In comparison to GAIL, our discriminator is no longer adversarial in nature, leading to significantly improved algorithm stability. Consequently, meticulous fine-tuning, as often required for GANs, is unnecessary. Although HYPO comprises three main components (the discriminator, the guider policy, the agent policy), the guider policy essentially functions akin to a Behavior Cloning (BC) task, involving minimal critical parameter adjustments. The parameters of the agent policy can be directly borrowed from the parameter settings in PPO, as we refrained from extensive parameter tuning specific to PPO.\n\nThe primary parameters that require adjustment within the algorithm are the aforementioned $\\eta$ (the positive class prior in Eq.6), $\\alpha$ (the weight factor in Eq.8), and $C$ (the KL coefficient in Eq.9). Estimating these three parameters is remarkably straightforward, and we achieved impressive results with almost optimal settings upon initial configuration. Specific details regarding these parameters can be found in the beginning portion of the Author Rebuttal."}, {"review_id": "fZs8y0Kehk", "reviewer": "Reviewer_2APk", "summary": "This work proposes a reinforcement learning algorithm to learn in a sparse reward environment with the help of data from a sub-optimal expert. The proposed algorithm, HYbrid Policy Optimization (HYPO) consists of three main parts, a discriminator, an offline guider, and an online agent. The primary idea is to use the sub-optimal expert data and discriminator loss, to train the offline guider.  The online agent interacts with the environment, and the offline guider is then used to guide the online agent using a BC loss.  ", "questions": "See weaknesses.", "limitations": "I would encourage the authors to include a section on the limitations of their work. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The idea of using an offline guider is interesting, and the authors do a good job in explaining their algorithm and substantiate their claims with experiments. ", "weaknesses": "While the paper has some interesting ideas, there are a few clarifications/weaknesses\n\n1. The inclusion of the offline guider as part of the input to the discriminator lacks clarity. It would be beneficial for the authors to conduct an ablation study to better understand this aspect.\n\n2. There appears to be a typo in equation 7,  it should likely be $\\pi_b$ instead of $\\pi$ (inferred from equation 6). Additionally, it is unclear whether the gradients flow through the discriminator (since $\\pi_b$ is an input to the discriminator) when computing the gradient of equation 7.\n\n3. How is the hyperparameter $\\eta$ varied? \n\nThe authors claim that the offline guider is able to use information from the trajectories sampled from the online policy to learn from the agent (line 202 - 204). Which helps it attain a better perform than the original expert. The authors further substantiate this claim using proposition 2 (lines 267-270) and the design of the offline guider's loss function.\nIf this is the case, \n\n4. Why is the value of C in equation 9 decayed? Shouldn't the training of the offline guider automatically bring $\\pi_b$ closer to $\\hat{\\pi}$, thereby reducing the KL term to 0?\n\n5. Isn't this decaying feature similar to LOGO (line 304-306), which the authors claim to result in conservative policies?\n\n6. How is the value of C value decayed? What would happen if C is set to 1 and not decayed?\n\n7. How is the KL divergence term calculated in equation 9? Is it an average KL?\n\n8. The theoretical contributions of the paper seem minimal, primarily consisting of minor adaptations or direct reworkings of results from PofD and LOGO.\n\nSuggestions that would further improve this work\n\n\n9. The algorithm seems sensitive to the hyperparameters $\\alpha$, $\\eta$, and $C$. I would highly recommend the authors to conduct a sensitivity analysis, and provide the values used, and the decay methodology employed in the paper. \n\n10. Inclusion of other baselines such as GAIL-PPO (use GAIL to train using sub-optimal data and then initialize the trained policy and perform PPO) and DAPG [1] will further substantiate the need for an offline guider. \n\n\n\n\nTypos\n\nLine 104: r is a function of state and action\n\n\n[1] Rajeswaran, Aravind, et al. \"Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.\" arXiv preprint arXiv:1709.10087 (2017).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and valuable feedback. We explain the concerns point by point below.\n\n**Q1: The ablation study of the input of discriminator.**\n\nA: We conducted additional ablation experiments to emphasize the significance of utilizing the offline guider as an additional signal to facilitate discriminator learning. Please refer to Figure 4 (PDF in author rebuttal) and the global author response for further details.\n\n**Q2: About $\\pi$ in Equation 7 and the gradient flow.**\n\nA: In Equation (7), $\\pi$ refers to $\\pi_b$. We are sorry for this confusion and will modify it in the next version. Regarding computing $d(s,a,\\log\\pi_b)$, we indeed apply gradient clipping (in Lines 137-138 of `\\HYPO\\hypo\\network\\disc.py` in the submitted code).\n\n**Q3: About the hyperparameter $\\eta$.**\n\nA: Please refer to the global author response and the Figure 1 (PDF in author rebuttal).\n\n**Q4: Why is the value of $C$ in Eq.9 decayed?**\n\nA: The primary purpose of decaying the coefficient $C$ is to gradually reduce the constraint imposed by the guider on the online agent. The online agent not only learns from the guidance of the guider but also interacts with the environment and learns from the rewards provided by the environment. When its performance is relatively poor or doesn't match the suboptimal expert's performance, it relies more on the guider's guidance. However, as the online agent learns and improves, surpassing the suboptimal expert, it increasingly requires exploration and learning from the environment. At this stage, if the guider continues to exert strong constraints, it could hinder the online agent's independent exploratory learning process, leading to excessively conservative strategies, similar to the results observed in the LOGO algorithm.\n\nWe conducted additional experiments to validate this aspect, and the details can be found in global author rebuttal and the Figure 3 (PDF in author rebuttal).\n\n**Q5: About the conservative policy and the difference with LOGO.**\n\nA: First, the process of coefficient decay allows the guider's constraint on the online agent to gradually weaken. Consequently, the online agent's policy evolves from being passively guided to actively exploring. However, in the case of LOGO, the decay affects the policy's trust region $\\delta$, leading to an increasingly conservative policy (smaller trust region means stronger constraint). This contrasts with HYPO's objective, which aims to achieve the opposite effect (i.e., achieving stronger constraint).\n\nFurthermore, HYPO is able to adopt this approach due to the dynamic improvement of the guider policy alongside the online agent. In LOGO, the pre-trained suboptimal expert policy remains static throughout, necessitating the constraint of the online agent's updates to prevent adverse effects stemming from the expert policy in the later stages of training.\n\n**Q6: About the hyperparameter $C$.**\n\nA: In all the experiments we have provided, the coefficient $C$ is decayed from 1.0 to 0.1. If $C$ remains constant without decay, the rate of improvement of the online agent would slow down, impeding the exploration efficiency of the online agent. This can be observed in the new added Figure 3 (PDF in author rebuttal).\n\n**Q7: How is the KL divergence term calculated in equation 9? Is it an average KL?**\n\nA: Yes, it indeed represents an average KL divergence. We initially record the action probability distribution of the guider policy and store it in the buffer of the online agent. When updating the actor of the online agent, we calculate the KL divergence between the computed online agent policy and the offline guider policy. The detailed calculation process can be found in the provided code, specifically in Lines 153-157 of `\\HYPO\\hypo\\algo\\hypo_ppo.py`.\n\n**Q8: About the theoretical contributions against to LOGO.**\n\nA: The primary distinction between the theoretical contributions of this paper and the LOGO approach lies in the fact that, in our work, after the online agent surpasses the suboptimal expert policy, the performance improvement lower bound is elevated through the learning of the guider policy, as opposed to the LOGO method which restricts updates via manually specified trust region parameters. Additionally, we provide theoretical analysis of the online agent policy's improvement, particularly in the later learning stages.\n\n**Q9: About the sensitivity analysis of the hyperparameters.**\n\nA: We have provided a comprehensive  sensitivity analysis of the hyperparameters. Please refer to the global author rebuttal and the PDF (PDF in author rebuttal).\n\n**Q10: About other baselines like GAIL-PPO and DAPG.**\n\nA: We conducted additional experiments and present the results in Figure 5 (PDF in author rebuttal).\n\n**Q11: About the limitations.**\n\nA: One of the limitations is the time complexity that we have mentioned in Appendix D. We will provide more discussion on this issue in the later version."}, {"review_id": "aHmKayf0Xs", "reviewer": "Reviewer_5BQ6", "summary": "This paper addresses the exploration challenge in RL by introducing imperfect demonstrations. These demonstrations are used to train an offline guider policy through imitation learning. Subsequently, the guider policy is employed to provide instructions during online training, thereby significantly enhancing the agent's exploration, particularly in the initial stages of training. Experimental results demonstrate that the proposed method outperforms certain baselines when faced with sparse rewards.", "questions": "Please refer to the weaknesses.", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The motivation is clearly articulated.\n2. The training approach for the guider policy is innovative.\n3. The experimental results demonstrate that the proposed method outperforms several baselines significantly.", "weaknesses": "1. This paper introduces a novel approach for training the guider policy. However, the experimental results fail to demonstrate its superiority. It would be beneficial to conduct additional ablation studies to thoroughly analyze the proposed method and provide more comprehensive insights.\n2. The comparison between the proposed method and baselines is limited to Mujoco experiments. It would be desirable to expand the comparisons to include other environments, allowing for a more comprehensive evaluation of the method's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful and valuable feedback. We explain the concerns point by point below.\n\n**Q1: About the superiority of HYPO and the ablation study.**\n\nA: We further elaborate on the superiority of our algorithm compared to other relevant methods and conduct more ablation experiments or sensitivity analysis to comprehensively evaluate the proposed method.\n\nAbout the superiority of HYPO: The most pronounced advantage of our algorithm is that, it can converge to near-optimal performance using only a **small amount** and **imperfect** (suboptimal & incomplete) demonstrations. While other related methods, such as LOGO, GAIL, POfD, require a large amount of demonstrations with high quality, and they are easy to suffer from the overly conservative policies, leading to a poor performance. These comparison results are provided on MuJoCo (e.g., Figure 5 in the main text). To provide a more comprehensive evaluation of our algorithm, we conduct more ablation study/sensitivity analysis. Please refer to the author rebuttal part and the figures in submitted PDF for more detailed results.\n\n**Q2: About more environments.**\n\nA: We have conducted more contact-rich manipulation tasks such as the [robosuite](https://robosuite.ai/) benchmark. Please refer to the author rebuttal and Table 1 (PDF in author rebuttal).\n\n"}]

Author rebuttal:
We would like to express our sincere gratitude for the thorough review and valuable feedback on our paper. The reviewers' insights and suggestions are quite helpful to improve the quality and clarity of our work. In this author rebuttal, we conduct additional experiments about the **ablation study**, **sensitivity analysis**, more **baselines** and more **environments**, which are the major common concerns of the reviewers. We provide brief analysis of these results below.

**1. Sensitivity analysis of the hyperparameters**

There are three main hyperparameters in our method HYPO: the positive class prior $\eta$, the weight factor $\alpha$, and the regularization coefficient $C$. During the development of our algorithm, we did not tune these hyperparameters much, as they are easy to estimate. For example, $\eta$ is set to 0.5 in most of the PU-Learning tasks, while we set it to [0.2, 0.8] due to the performance improving of the online agent policy. As for $\alpha$, we need to ensure $\mathcal{F}_{\text{Expert}}>0$ whith the domains of $\eta$ and $d$. Therefore, the value of $\alpha$ must be greater than 5.0. Finally $C$, which can be set like the entropy coefficient, needs to ensure that the losses are within the similar order of magnitude.

First, Figure 1 (PDF in author rebuttal) illustrates the impact of parameter $\eta$ on our method. In this figure $\eta$ indicates the initial value, and linearly increases to the end value of $1-\eta$. $\eta$ primarily affects the updates of the guider, since it determines the weight of the positive samples that come from the online agent. But the online agent performance is rarely affected by the value of $\eta$ (Figure 1a). Second, Figure 2 (PDF in author rebuttal) illustrates the impact of parameter $\alpha$ on HYPO. Similar to $\eta$, $\alpha$ also primarily affects the updates of the guider.  However, the offline imitation policy can dynamically adjust these two weights of $\mathcal{F}$ and $\mathcal{G}$, which means that bigger $\mathcal{F}$ can lead to bigger $\mathcal{G}$ (Figure 2b, 2c). In this way, the weight factor $\alpha$ rarely affect the learning of the guider. Therefore, the agent's performance is not sensitive to $\alpha$ (Figure 2a). Finally, Figure 3 (PDF in author rebuttal) illustrates the impact of parameter $C$ on HYPO. In this figure the $C_{min}$ indicates the end value of $C$ (all decay from 1.0). If $C$ has a large value, such as 0.5 or 1.0 (not decaying), the performance of the online agent can be affected due to the overly restrictions (Figure 3a). However, $C$ also can not be too small, such as 0. In this case, the KL between the offline guider and the online agent would be very large, leading to collapse in the agent's performance.

**2. The ablation study of the input in discriminator**

We compare the accuracy of distinguishing the expert data and the agent data to study the necessity of $\log \pi_b$ in the discriminator. As illustrated in Figure 4 (PDF in author rebuttal), introducing $\log \pi_b$ in the input of discriminator is beneficial for accurate discrimination.

**3. New baselines: GAIL-PPO and DAPG**

We add new baselines and test them in the MuJoCo environments (e.g., HalfCheetah, as shown in Figure 5 (PDF in author rebuttal)). These new baselines include: 1) **GAIL-PPO**, which trains GAIL to get a suboptimal performance and then initialize the trained policy and perform PPO; and 2) **DAPG**, which warm starts Natural Policy Gradient (NPG) algorithm using BC, and fine tunes it online using behavior data in a heuristic manner.

**4. New environment: robosuite**

In order to more comprehensively evaluate the performance of HYPO, we conduct additional experiments in [robosuite](https://robosuite.ai/) environment. We choose two easy tasks (Block Lifting, Door Opening) and a hard task (Two Arm Lifting) to test our method. First, we use PPO benchmark to train an expert in the dense reward setting (use the 'reward shaping' option). Then we use a partially trained expert that is still at a sub-optimal stage of learning to provide the demonstrations. Finally we implement HYPO, GAIL and POfD in these tasks without using 'reward shaping' to compare the performance, as shown in Table 1 (PDF in author rebuttal). 

Meta-review: This paper proposes an approach for RL in the sparse reward setting, wherein the RL agent is directed by a guidance policy that provides direction to the online learner. While this idea has been studied before, the main contribution of this work is on developing a guidance policy that utilizes both offline data gathered by a behavior policy and online data gathered by the learning agent. 


The review scores are 7, 7, 6, 5, 5 and the average score is 6.0. The reviewers are generally positive about the novelty of the algorithmic contribution, clarity of presentation, and technical soundness. The main concern expressed by multiple reviewers was about the similarity and the lack of detailed comparisons with the LOGO algorithm (Rengarajan et al. 2022). The authors have given detailed response these comments, and the reviewers have accepted their explanations. 

For the final version, please make sure that you include the comments and suggestion by the reviewers.

Thanks,

AC
Decision: Accept (poster)