SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $C$  is a decreasing coefficient of the KL-divergence. In Eq (9), we append a constraint of the KL-divergence between the guider policy  $\pi_{b}$  and the agent policy  $\hat{\pi}$ . In this way the agent can distill knowledge constantly from the guider. Besides, the decreasing coefficient of  $C$  can prevent the agent from the excessively conservative policy. Next, we theoretically analyze how the additional constraint can influence the agent. We consider the following two cases: case 1) the agent's performance is worse than the expert, which corresponds to satisfaction of Assumption 1, and case 2) the agent's performance is better than the expert, which means Assumption 1 is not satisfied.
Case 1. In the initial stage of training, the offline guider policy  $\pi_{b}$  can be approximately treated as the expert policy since the BC method is more efficient than policy gradient (PG) and the  $\mathcal{F}_{\mathrm{Expert}}$  keeps large, which drives  $\pi_{b}$  to update towards the expert. Now the performance improvement guarantee of the online agent under the Assumption 1 can be given as follows:
Proposition 1. Let  $\tilde{\pi}$  be a policy that satisfies Assumption 1. Then, for policy  $\hat{\pi}$ ,
$$
J _ {R} (\hat {\pi}) - J _ {R} (\tilde {\pi}) \geq (1 - \gamma) ^ {- 1} \xi - (1 - \gamma) ^ {- 1} \epsilon_ {R, \tilde {\pi}} \sqrt {2 D _ {\mathrm {K L}} ^ {\hat {\pi}} (\hat {\pi} , \pi_ {b})}, \tag {10}
$$
where  $\epsilon_{R,\tilde{\pi}} = \max_{s,a}|A_{\tilde{R}}^{\tilde{\pi}}(s,a)|$
Proof. Refer to the Appendix B.
It is reasonable to assume that  $\hat{\pi}$  satisfies Assumption 1 in the initial stage of learning since the current policy  $\hat{\pi}$  of agent is learning from scratch. Then, minimizing  $D_{\mathrm{KL}}^{\hat{\pi}}(\hat{\pi},\pi_b)$  can get a non-negative lower bound in Equation (10), which means that we can accelerate the online learning of the agent by minimizing the KL-divergence between  $\hat{\pi}$  and  $\pi_b$ . This process can be described as, the agent learns from the guider to attain the expert performance, which can constantly get non-zero rewards from the sparse environments.
Case 2. As the training progresses, the agent can achieve the expert performance efficiently. This can be realized in most of the previous imitation methods. However, we want to investigate that, how
Figure 3: MuJoCo (left) is a set of popular continuous control environments with tasks of varying difficulty. Google Research Football (top right) is a novel RL environment where agents are trained to play football in an advance, physics based 3D simulation. AirSim (bottom right) is a new simulator built on Unreal Engine that offers physically and visually realistic simulations for both of these goals.
the agent can surpass the expert and achieve the near-optimal performance in sparse reward tasks, which corresponds to do not satisfy the Assumption 1. If the guidance is simply removed from the agent, in other words, the agent no longer distills knowledge from the guider, the bootstrap error and the uncertain reward signals can reduce the performance of agent policy. If we use the expert policy to restrict the agent, like other LfD methods, the agent policy will be excessively conservative, making it hard for the agent to achieve the near-optimal performance. Therefore, we address this issue by changing the learning objective of guider from expert to the agent, which will reduce the KL-divergence between  $\pi_b$  and  $\hat{\pi}$ . The policy improvement lower bound when Assumption 1 is not satisfied is given as follows:
Proposition 2. For policy  $\hat{\pi}$  and any policy  $\tilde{\pi}$ ,
$$
J _ {R} (\hat {\pi}) - J _ {R} (\tilde {\pi}) \geq - \frac {3 R _ {\max }}{2 (1 - \gamma) ^ {2}} \sqrt {2 D _ {\mathrm {K L}} ^ {\max } (\hat {\pi} , \tilde {\pi})}, \tag {11}
$$
where  $R_{\max} = \max_{s,a}|R(s,a)|$ .
The guider update towards the agent will reduce the KL-divergence item to improve the policy improvement bound in Eq (11).
The mutual update of the guider policy and agent policy is the core feature behind the HYPO algorithm. In this way, the agent can leverage a few suboptimal demonstrations for efficient exploration while avoiding the negative impact of the low-quality data.
In this section, we investigate whether HYPO can achieve near-optimal performance in extremely sparse reward environments by overcoming the restriction of imperfect demonstrations, and how the number and quality of the trajectories can influence the performance of HYPO. To comprehensively assess our method, we first perform an exhaustive evaluation of HYPO in MuJoCo (Todorov et al., 2012) with sparse rewards and Google Research Football (GRF) (Kurach et al., 2020) with huge policy space and only a sparse score reward. We also evaluate HYPO on an Unmanned Aerial Vehicle (UAV)  ${}^{2}$  task based on the Unreal Engine and AirSim (Shah et al., 2018) to show the effectiveness of HYPO in addressing more challenging control tasks with high-fidelity. All the mentioned environments are shown in Figure 3.
$^{2}$ https://github.com/sunghoonhong/AirsimDRL
Figure 4: MuJoCo simulation results. The  $x$ -axis is the number of samples. The  $y$ -axis is the average episode return, which is scaled to make the expert achieve 100 and a random policy achieve 0.
Figure 5: Final performance of the baseline algorithms that are trained with various number and performance of trajectories in demonstrations. HYPO is minimally affected by the quality of data.
5.1 MuJoCo Simulation
We first sparsify the built-in dense rewards on the MuJoCo platform for evaluating the methods in sparse-reward environments. Specifically, a reward of  $+1$  is provided only after the agent moves forward over a specific distance. We compare HYPO to the following baselines: (1) Expert, which applies PPO using the original dense rewards to achieve the optimal return; (2) Demo, which is a suboptimal expert at the early stage of training; (3) PPO, which directly trains PPO with sparse rewards; (4) GAIL (Ho & Ermon, 2016), which uses a discriminator to provide a demonstration-guided reward for training; (5) POfD (Kang et al., 2018), which uses a weighted combination of the environment reward and the demonstration-guided reward; and (6) LOGO (Rengarajan et al., 2022), which merges a policy improvement step and a policy guidance step. All the demonstrations used for training are generated by Demo. Refer to Appendix C for more experimental details.
As shown in Figure 4, PPO fails to learn an useful policy in all environments due to the lack of guidance from demonstrations. GAIL can only achieve the level of the suboptimal expert policy since it can only mimic the policy that generates the demonstrations. POfD can attain a higher return than GAIL in HalfCheetah but a similar return in other tasks. LOGO can achieve relative better performance due to its decaying trust region that constrains the policy to gradually get rid of the influence of suboptimal data. However, the conservative policy can still impede the learning efficiency during the later learning process. As expected, HYPO outperforms the above methods in all environments by a large margin, which fully validates the effectiveness of HYPO in learning
near-optimal performance in extremely sparse reward environments by overcoming the restriction of imperfect demonstrations. We also investigate the influence of the number and quality (in terms of cumulative return) of demonstrations on the final performance. As shown in Figure 5, HYPO is capable of achieving far better performance than other methods, even with a small number of imperfect trajectories. Refer to Appendix F for more details about the performance of LOGO.
5.2 Google Research Football and AirSim Simulation
The GRF task is to control a single player to cooperate with teammates to break a specific defensive line formation and score. This environment provides two type of reward settings, i.e., the sparse score reward, and the dense checkpoints reward. AirSim is a high-fidelity environment where we need to control an UAV to fly to the terminal while avoiding all the obstacles. This task also provides the sparse reward setting for flying a specific distance and the dense reward setting for keeping a certain speed of forward flight. The expert baseline is trained in the dense reward setting, while the imperfect demo data is generated by a partially trained expert. The results in Figure 6 show that HYPO can achieve near-optimal performance in these challenging tasks with imperfect demonstrations.
Figure 6: Results in GRF and AirSim. The left  $y$ -axis is for GRF and the right for AirSim.
6 Conclusion and Outlook
In this paper, we investigate how to accelerate agent online learning with imperfect demonstrations in sparse reward environments. We introduce HYPO, a novel RL algorithm that can attain near-optimal performance in sparse reward settings by avoiding the excessive conservative policy. The highlight of HYPO lies in its capability to learn an offline guider and an online agent, while updating these two policies mutually to help the agent learn from the demonstrations and environments more efficiently. Experiments in various environments including MuJoCo, Google Research Football and AirSim UAV simulation demonstrate that HYPO can greatly promote the learning efficiency in sparse reward tasks with imperfect data. Our future work is to extend HYPO to multi-agent scenarios when agents need to learn coordinated policies with sparse rewards. Moreover, we plan to investigate the potential of HYPO in more real-world applications, such as the autonomous driving, where sparse rewards are common challenges.
This work was supported by an SYSU-ByteDance Research Project.
Carlini, N., Athalye, A., Papernot, N., Brendel, W., Rauber, J., Tsipras, D., Goodfellow, I., Madry, A., and Kurakin, A. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705, 2019.
Devidze, R., Radanovic, G., Kamalaruban, P., and Singla, A. Explicable reward design for reinforcement learning agents. Advances in Neural Information Processing Systems, 34:20118-20131, 2021.
Fu, J., Luo, K., and Levine, S. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248, 2017.
Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Xu, H., Zhan, X., Yin, H., and Qin, H. Discriminator-weighted offline imitation learning from suboptimal demonstrations. In International Conference on Machine Learning, pp. 24725-24742. PMLR, 2022.
Zolna, K., Reed, S., Novikov, A., Colmenarejo, S. G., Budden, D., Cabi, S., Denil, M., de Freitas, N., and Wang, Z. Task-relevant adversarial imitation learning. In Conference on Robot Learning, pp. 247-263. PMLR, 2021.