SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hanlin Yang
Chao  $\mathbf{Y}\mathbf{u}^{*}$
Peng Sun
Siji Chen
yuchao3@mail.sysu.edu.cn
Exploration is one of the main challenges in Reinforcement Learning (RL), especially in environments with sparse rewards. Learning from Demonstrations (LfD) is a promising approach to solving this problem by leveraging expert demonstrations. However, expert demonstrations of high quality are usually costly or even impossible to collect in real-world applications. In this work, we propose a novel RL algorithm called HYbrid Policy Optimization (HYPO), which uses a small number of imperfect demonstrations to accelerate an agent's online learning process. The key idea is to train an offline guider policy using imitation learning in order to instruct an online agent policy to explore efficiently. Through mutual update of the guider policy and the agent policy, the agent can leverage suboptimal demonstrations for efficient exploration while avoiding the conservative policy caused by imperfect demonstrations. Empirical results show that HYPO significantly outperforms several baselines in various challenging tasks, such as MuJoCo with sparse rewards, Google Research Football, and the AirSim drone simulation.
Reinforcement Learning (RL) (Sutton & Barto, 2018) plays an important role in solving real-world control problems with large state-action spaces. In RL, an agent learns a decision-making policy through interaction with the environment, based on a reward function that indicates the agent's learning goal. However, pre-defining a reward function relies heavily on human knowledge that varies from one to another, and sometimes carefully hand-tuned reward function could lead to undesired or even hazardous policies (Devidze et al., 2021). Therefore, a more intuitive way of reward definition is to utilize the completeness of the task, e.g., the distance that a robot walks or scoring a goal in football games. While being unbiased towards the learning goal, learning with such sparse rewards could be challenging due to the inefficient exploration in the large problem space.
In recent years, there has been intensive research interest in Learning from Demonstrations (LfD) as a promising approach to addressing sparse reward tasks by leveraging the expert demonstrations. Methods like DDPGfD (Vecerik et al., 2017), DQfD (Hester et al., 2018) and AWAC (Nair et al., 2020) normally require that the demonstrations consist of complete trajectories including state, actions and rewards, and thus are not applicable in real-world applications when it is only possible to observe the state and action information in the demonstrations. Other methods based on Imitation Learning (IL), such as Generative Adversarial Imitation Learning (GAIL) (Ho & Ermon, 2016) and Adversarial Inverse Reinforcement Learning (AIRL) (Fu et al., 2017), usually require that the demonstrations be generated by an optimal expert, which are difficult to achieve in complex tasks. More recent approaches such as Policy Optimization with Demonstrations (POdfD) (Kang et al.,
*Corresponding author. Code is available at https://github.com/joenghl/HPO.
2018) and Learning Online with Guidance Offline (LOGO) (Rengarajan et al., 2022) can result in an excessively conservative policy (Nair et al., 2020), due to the negative impact of imperfect demonstrations. Specifically, when the agent performs better than the suboptimal expert policy, simply using the demonstration-guided reward or distilling knowledge (e.g., POfD and LOGO) from the suboptimal policy will cause an excessively conservative policy of the online learning agent.
In this paper, we propose HYbrid Policy Optimization (HYPO), a novel RL algorithm that is capable of achieving near-optimal performance in sparse reward environments using only a small number of imperfect demonstrations with low quality and incomplete trajectories. The basic idea is to learn an offline imitation policy (i.e., the guider) to guide the learning of the agent and an online learning policy (i.e., the agent) to interact with the environment. These two policies are updated mutually to help the agent learn from both the expert demonstrations and the environment more efficiently. The key insight of HYPO is that, the guider imitates the expert to guide the learning of the agent during the initial stage, and then the agent learns to outperform the expert through online interactions with the environment. We evaluate HYPO on a wide variety of control tasks, including MuJoCo with sparse rewards, Google Research Football and AirSim UAV simulation. Besides, we also investigate how the number and quality of the demonstrations can influence the performance of HYPO and other baseline methods. The results show that 1) HYPO can learn successful policies in these challenging tasks efficiently, and 2) HYPO is minimally affected by the number and quality of the demonstrations.
2.1 Offline RL
Offline RL (Levine et al., 2020) enables to learn policies using only offline data, without any online interaction with the environment. However, the related offline RL methods, such as BCQ (Fujimoto et al., 2019), BEAR (Kumar et al., 2019), BRAC (Wu et al., 2019), and Fisher-BRC (Kostrikov et al., 2021) still suffer from the issue of distributional shift, i.e., the policy is trained under one distribution but evaluated on a different distribution. A common way to mitigate this issue is to restrict the policy distribution to be similar to the policy that generates the offline data. However, this constraint can result in excessively conservative policies. In contrast, HYPO can avoid the distribution shift problem since the offline trained guider has no interaction with the environment, but is only used to guide the online agent for more efficient exploration. In addition, unlike the existing offline RL approaches that normally require a large number of data with complete trajectories, thus have limited applications when collecting a large number of perfect demonstrations is either time-consuming or even infeasible, HYPO is able to learn a near-optimal policy given only a small number of incomplete demonstrations generated by a suboptimal policy.
2.2 IL
IL methods are designed to mimic the behaviors of experts. As one of the most well-known IL algorithms, Behavior Cloning (BC) learns a policy by directly minimizing the discrepancy between the agent and the expert in the demonstration data. However, BC has been shown to suffer from issues such as compounding errors and the inability to handle the distributional shift Ross et al. (2011). As a recent BC-based method, DWBC Xu et al. (2022) trains a pure offline BC task to attain the expert performance by using a discriminator that distinguishes the optimal expert data and the suboptimal data. Different from DWBC, HYPO can learn policies that outperform the expert by using online interaction with the environments and offline guidance from the guider. Moreover, unlike BC-based methods that always require optimal demonstrations, HYPO is capable of learning with only imperfect demonstrations. As another type of IL, Inverse RL (IRL) (Ng et al., 2000) learns a reward function that explains the expert behavior and then uses this reward function to guide the agent's learning process. Some popular approaches use adversarial training to train a policy that is similar to the expert policy, but at the same time is robust to distributional shift, such as the GAIL (Ho & Ermon, 2016) and AIRL (Fu et al., 2017). However, unlike HYPO, all these methods assume that the expert actions be optimal.
2.3 LfD
The goal of LfD is to use demonstration data to aid agent online learning, so as to overcome the exploration problem in RL, especially in environments with sparse rewards Schaal (1996). DQfD (Hester et al., 2018) and DDPGfD (Vecerik et al., 2017) rely on complete offline expert demonstrations with associated rewards to accelerate the online learning process in discrete and continuous action spaces, respectively. POfD Kang et al. (2018) combines the benefits of both imitation learning and RL to learn policies from both demonstration data and online interactions by learning from a shaped reward averaged over the original environment reward and a demonstration-guided reward. However, as the environment reward is much sparser than the demonstration-guided reward, such a mixed reward may cause the learning objective to deviate from the goal of the optimal policy. Another LfD approach is LOGO (Rengarajan et al., 2022), which pre-trains an expert policy to guide the learning of agent, and uses a trust region controlled by a hyper-parameter to restrict agent learning. Unlike LOGO that the final policy may be excessively conservative due to the fixed suboptimal expert, HYPO enables the agent to outperform the expert through updating the guider and the agent mutually.
3.1 Markov Decision Process
We consider the standard Markov Decision Process (MDP) (Sutton & Barto, 2018) as the mathematical framework for modeling sequential decision-making problems, which is defined by a tuple  $\langle S, A, P, r, d_0, \gamma \rangle$ , where  $S$  is a finite set of states,  $A$  is a finite set of actions,  $P: S \times A \times S \to \mathbb{R}$  is the transition probability function,  $r: S \to \mathbb{R}$  is the reward function,  $d_0$  is the initial distribution, and  $\gamma \in [0,1)$  is the discount factor. An agent interacts with the environment based on policy  $\pi: S \times A \to [0,1]$ , which maps from state to distribution over actions. The performance of  $\pi$  is evaluated by its expected discounted reward  $J(\pi)$ :
$$
J (\pi) = \mathbb {E} _ {\pi} [ r (s, a) ] = \mathbb {E} _ {\left(s _ {0}, a _ {0}, s _ {1}, \dots ,\right)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r \left(s _ {t}, a _ {t}\right) \right], \tag {1}
$$
where  $(s_0, a_0, s_1, \ldots)$  is a trajectory generated from interaction with the environment, i.e.,  $s_0 \sim d_0$ ,  $a_t \sim \pi(\cdot | s_t)$  and  $s_{t+1} \sim P(\cdot | s_t, a_t)$ . The value function of a policy is defined as  $V_\pi(s) = \mathbb{E}_{\tau \sim \pi}[\sum_{t=0}^\infty \gamma^t r(s_t, a_t) | s_0 = s]$ , i.e., the expected cumulative discounted reward obtained by the policy  $\pi$  from the state  $s$ . Correspondingly, the action-value function is defined as  $Q_\pi(s, a) = \mathbb{E}_{\tau \sim \pi}[\sum_{t=0}^\infty \gamma^t r(s_t, a_t) | s_0 = s, a_0 = a]$ , and the advantage function is  $A_\pi(s, a) = Q_\pi(s, a) - V_\pi(s)$ . The objective of RL algorithm is to discover the optimal policy  $\pi^*$  that can maximize the expectation of discounted return  $J(\pi)$ , i.e.,  $\pi^* = \operatorname{argmax}_\pi J(\pi)$ .
3.2 Learning with Offline Demonstrations
BC and GAIL are well-known baseline methods for learning with offline demonstrations. The difference is that BC is a pure offline method without any interactions with the environment, while GAIL is an online learning method through generating and learning from the trajectories.
BC. BC is used for matching the distribution over actions, with its objective given by:
$$
\mathcal {L} _ {\pi} = \underset {(s, a) \sim \mathcal {D}} {\mathbb {E}} \left[ - \log \pi (a | s) \right], \tag {2}
$$
where  $\mathcal{D}$  is the offline demonstrations. Optimizing objective (2) will make the distribution of actions w.r.t.  $\pi_b$  match the one in the expert policy that generates the state-action pairs in  $\mathcal{D}$ .
GAIL. The discriminator aims to distinguish between the offline expert demonstrations and the trajectories sampled by an online agent. The objective of the discriminator in GAIL is given by:
$$
\mathcal {L} _ {d - \mathrm {G A I L}} = \min  _ {d} \underset {(s, a) \sim \mathcal {D}} {\mathbb {E}} \left[ - \log d (s, a) \right] + \underset {(s, a) \sim \mathcal {B}} {\mathbb {E}} \left[ - \log \left(1 - d (s, a)\right) \right], \tag {3}
$$
where  $\mathcal{D}$  is the offline demonstrations,  $\mathcal{B}$  in the replay buffer consists of trajectories sampled by the agent,  $d$  is the discriminative classifiers  $d: S \times \mathcal{A} \to (0,1)$ . Optimizing objective (3) will
make the learned discriminator assign 1 to all transitions from the offline demonstration  $\mathcal{D}$  and 0 to all transitions from  $\mathcal{B}$ . During training, the discriminator will assign a large reward to agent when  $d_{(s,a)\sim \mathcal{B}}(s,a)$  is close to 1, which is inspired by the generative adversarial training.
The discriminator can be interpreted as a new reward function  $r(s, a) = -\log \left(1 - d(s, a)\right)$  providing learning signal to the agent. In this way, the agent will receive larger reward if its trajectories are more like the expert, leading to against the discriminator.
4 Hybrid Policy Optimization with Imperfect Demonstrations
Figure 1: Our proposed method provides a suboptimal expert direction (red curve) for the agent, which prevents it from pointless exploration (brown curves) under sparse reward environments.
Before presenting detailed descriptions of our method, we use Figure 1 to illustrate the overall motivation. In the early learning stage, the agent usually suffers from pointless exploration (i.e., brown curves) due to the sparsity of the reward signal. In such stage, the agent is generally worse than the expert, and thus can gain beneficial guidance from the expert demonstrations to attain a relatively fair performance that is close to the expert. In the later learning stages, we expect the agent to surpass the expert through exploration in the environment, hence being too close to the expert can hinder the policy improvement. However, without constraining the deviation to the expert, the agent can still encounter performance collapse due to the pointless exploration caused by the sparse reward. We tackle these challenges by updating the two policies of the guider and agent mutually, to provide the agent with an appropriate guidance. In this way, the agent is able to outperform the expert to achieve a near-optimal performance.
Our goal is to develop an algorithm that can leverage the imperfect demonstrations generated by a suboptimal expert to boost online learning in sparse reward environment. We assume that there is a suboptimal expert policy  $\pi_e$  but we have no access to it. All we can use are its demonstrations, which have the form  $\mathcal{D} = \{\tau^i\}_{i=1}^n$ , where  $\tau^i = (s_1^i, a_1^i, \ldots, s_t^i, a_t^i)$ ,  $\tau^i \sim \pi_e$ . And we make the following reasonable and indispensable assumption concerning the caliber of the expert policy:
Assumption 1. In the initial learning stages, the agent can get higher advantage values than current policy  $\pi$  if it acts according to the expert policy  $\pi_{e}$ ,
$$
\mathbb {E} _ {a _ {e} \sim \pi_ {e}, a \sim \pi} \left[ A _ {\pi} (s, a _ {e}) - A _ {\pi} (s, a) \right] \geq \xi > 0, \forall s \in \mathcal {S} \tag {4}
$$
The above assumption implies that taking actions according to the expert will provide a higher advantage than taking actions according to the current policy  $\pi$ . This is reasonable since the expert policy performs much better than an untrained policy in the initial learning stages.
To better illustrate the analysis and simplify the formulation below, we use the term "expert" to indicate the suboptimal expert unless using the term "optimal" as the prefix. The offline and the online policy is respectively denoted as  $\pi_b$  and  $\hat{\pi}$ . The demonstrations consisting of trajectories generated by the suboptimal expert is denoted as  $\mathcal{D}$ , while the replay buffer consisting of trajectories sampled by the agent is denoted as  $\mathcal{B}$ . The output of the discriminator is  $d: S \times \mathcal{A} \to (0,1)$ .
The three key components in HYPO are the discriminator, the offline guider, and the online agent. The discriminator is used to control the learning objective of the offline guider by distinguishing whether the trajectories are generated by the expert or the agent. The offline guider performs a BC task to provide the agent an adaptive guidance by learning from the expert and the agent dynamically. The online agent policy interacts with the environments and distills knowledge from the guider policy constantly to outperform the expert.
4.1 Semi-supervised Discriminator Learning
The discriminator in GAIL mentioned in Subsection 3.2 suffers from the overfitting problem, since the well-learned discriminator may overfit to the suboptimal expert (Zolna et al., 2021). Concretely, as the agent policy  $\hat{\pi}$  improves, the behaviors of the agent become increasingly similar to those of the expert. However, due to the suboptimality of the expert, it is crucial to avoid overfitting to the suboptimal expert, as it can hinder the agent from converging to the optimal performance. To address this issue, we formulate the discriminator objective as a positive-unlabeled (PU) reward learning problem (Xu & Denil, 2021), which allows us to train the discriminator by treating the agent trajectories as an unlabeled mixture of the expert trajectories and the agent's trajectories. In this way, the guider can also learn from the agent since the agent's trajectories are not just the negative data, but the mixture of positive and negative data. The formulated discriminator objective is given as follows:
$$
\min  _ {d} \eta_ {(s, a) \sim \mathcal {D}} \left[ - \log d (s, a) \right] + \underset {(s, a) \sim \mathcal {B}} {\mathbb {E}} \left[ - \log (1 - d (s, a)) \right] - \eta_ {(s, a) \sim \mathcal {D}} \left[ - \log (1 - d (s, a)) \right], \tag {5}
$$
where  $\mathcal{D}$  denotes the offline demonstrations,  $\mathcal{B}$  the replay buffer consisting of the trajectories sampled by the agent, and  $\eta \in [0,1]$  the positive class prior, which is assumed to be known.
Furthermore, it is difficult to distinguish between the expert's and agent's trajectories by only relying on the information of state-action pairs. To mitigate this issue, an additional signal  $\log \pi_{b}$  is added into the input of the discriminator, leading to the final objective of the discriminator as follows:
$$
\begin{array}{l} \mathcal {L} _ {d} = \eta \underset {(s, a) \sim \mathcal {D}} {\mathbb {E}} \Big [ - \log d (s, a, \log \pi_ {b}) \Big ] + \underset {(s, a) \sim \mathcal {B}} {\mathbb {E}} \Big [ - \log \Big (1 - d (s, a, \log \pi_ {b}) \Big) \Big ] \\ - \eta_ {(s, a) \sim \mathcal {D}} \left[ - \log (1 - d (s, a, \log \pi_ {b})) \right]. \tag {6} \\ \end{array}
$$
Specifically, when considering state-action pairs  $(s,a)$  originating from the agent's trajectories,  $\pi_b(a|s)$  will assign large probabilities to the agent's actions within these corresponding states. Conversely, for state-action pairs  $(s,a)$  derived from the expert's trajectories,  $\pi_b(a|s)$  will assign small probabilities to the expert's actions within these corresponding states.
Remark on  $\eta$ . It is worth noting that, unlike the standard positive-unlabeled learning setup, the positive class prior  $\eta$  in our setting changes as the online policy learning progresses and the distribution of states in the replay buffer evolves. Specifically,  $\eta$  is set to increase as the learning progresses.
4.2 Adaptive Target for Offline Imitation
The objective of BC task is to mimic the expert policy through matching the conditional distribution  $\pi (\cdot |s)$  over actions. Therefore, the BC methods cannot learn a policy that outperforms the expert without additional guidance. In HYPO,  $\pi_{b}$  is able to use the information from the trajectories sampled by the online policy  $\hat{\pi}$  to learn from the agent, which can assist  $\pi_{b}$  to attain a better performance than the original expert. Specifically, we use two adaptive functional weights  $\mathcal{F}_{\mathrm{Expert}}(d)$  and  $\mathcal{G}_{\mathrm{Agent}}(d)$  to determine the learning objective of  $\pi_{b}$  dynamically:
$$
\mathcal {L} _ {\pi_ {b}} = \underset {(s, a) \sim \mathcal {D}} {\mathbb {E}} \left[ - \log \pi_ {b} (a | s) \cdot \mathcal {F} \left(d (s, a, \log \pi_ {b})\right) \right] + \underset {(s, a) \sim \mathcal {B}} {\mathbb {E}} \left[ - \log \pi_ {b} (a | s) \cdot \mathcal {G} \left(d (s, a, \log \pi_ {b})\right) \right], \tag {7}
$$
where  $\mathcal{F}$  and  $\mathcal{G}$ , denoting  $\mathcal{F}_{\mathrm{Expert}}(d)$  and  $\mathcal{G}_{\mathrm{Agent}}(d)$ , respectively, are the functional weights to determine the objective of  $\pi_b$ . The functional weights are expected to force the  $\pi_b$  to approach the expert in the initial stage, and to increase  $\mathcal{L}_d$  to ensure discriminator robust throughout the whole learning process. Since the performance of the online learning policy differs significantly, ranging from being random to reaching the near-optimal, the discriminator needs to be robust enough to handle these changes. Inspired by the adversarial robustness (Carlini et al., 2019) and the weighted discriminator (Xu et al., 2022), we use  $\pi_b$  to maximize  $\mathcal{L}_d$  to improve the robustness of discriminator by minimizing the worst-case error. Consequently, the form of the functional weights of  $\mathcal{F}_{\mathrm{Expert}}$  and  $\mathcal{G}_{\mathrm{Agent}}$  can be derived as follows:
$$
\mathcal {F} _ {\text {E x p e r t}} (d) = \alpha - \frac {\eta}{d (1 - d)}, \quad \mathcal {G} _ {\text {A g e n t}} (d) = \frac {1}{1 - d}, \tag {8}
$$
where  $\alpha$  is a weight factor. Refer to Appendix B for the detailed derivation.
To understand the trend of the functional weight  $\mathcal{F}_{\mathrm{Expert}}(d)$  intuitively, we visualize it as illustrated in Figure 2. For transitions from  $\mathcal{D}$ , the weight is insensitive to  $d$  when  $\eta$  is small, since the proportion of positive samples in the  $(s,a)$  pairs generated by the agent is low. In this case,  $\pi_b$  can provide the online agent with guidance by learning towards the expert. As training progresses, the proportion  $\eta$  of positive samples in agent trajectories gradually increases, thus  $d$  begins to dominate the weight changes of  $\mathcal{F}_{\mathrm{Expert}}(d)$ . In this case,  $\mathcal{F}_{\mathrm{Expert}}(d)$  becomes sensitive to  $d$ , and gets smaller since the  $d$  gets
Figure 2: The trend of weight  $\mathcal{F}_{\mathrm{Expert}}(d)$ . The deeper blue color means the lower weight value.  $\eta$  is increasing during the training and  $d$  is various from 0.1 to 0.9 (clipped).
large when  $(s, a)$  comes from the expert, resulting in the learning towards the agent for  $\pi_b$ . As for the other weight of  $\mathcal{G}_{\mathrm{Agent}}$ , which will be large if the  $(s, a)$  sampled by the agent  $\hat{\pi}$  is similar to that of the expert, changes insensitively when  $d$  remains low.
4.3 Performance Improvement of Online Learning
An useful reward signal is necessary for online RL algorithms to learn efficiently. However, in the sparse reward environments, the agent cannot explore states that have non-zero reward without additional guidance. To this end, we use an offline guider to provide the agent with guidance, which can significantly boost the exploration of agent, especially in the initial stages of training. Specifically, the online agent policy  $\hat{\pi}$ , which learns from the sparse environment rewards, constantly distills knowledge from the guider policy  $\pi_{b}$ . This is implemented by adding another corrective objective on the original PPO (Schulman et al., 2017) policy update:
$$
J _ {\hat {\pi}} ^ {\mathrm {H Y P O}} (\theta) = \mathbb {E} _ {t} \left[ \min  \left(r _ {t} (\theta) A _ {t}, \operatorname {c l i p} \left(r _ {t} (\theta), 1 - \epsilon , 1 + \epsilon\right) A _ {t}\right) - C D _ {\mathrm {K L}} (\hat {\pi} | | \pi_ {b}) \right], \tag {9}
$$