SlowGuess/NeurIPS_2023_OCR huggingface dataset:
**Base RL Methods.** Our method can be implemented for both continuous and discrete action spaces by extending TD3 [10] and DQN [31] with PV loss and the balanced buffer. While TD3 uses a deterministic policy, DQN adopts epsilon-greedy exploration that makes the policy stochastic. We remove the action noise in DQN and simply follow the argmax rule to select actions. Therefore, our method enjoys deterministic novice policy in both cases. The primary reason is that according to the feedback of human subjects, stochastic novice makes human subjects experience excessive fatigue due to the difficulty in monitoring and correcting agents' noisy actions. This design choice makes our method more user-friendly, as shown in the user study in Sec. 5.3.
Balanced Buffers. The intervention gradually becomes sparse as the agent learns to reduce human intervention. However, those sparse intervention signals contain even more important information on how to behave under critical situations. Previous method [26] stores agent data and human data into one buffer and samples them uniformly. Abusing the notations, the ratio between the transitions from the agent's exploration and from human involvement is  $|\mathcal{B}_n|:|\mathcal{B}_h|$  in each SGD batch. The human demonstrations are overwhelmed by the amount of agent-generated trajectories,
Table 1: Comparison of different approaches in MetaDrive-Keyboard. The overall intervention rate is given besides the human data usage.
Method	Training	Testing
Human Data Usage	Total Data Usage	Total Safety Cost	Episodic Return	Episodic Safety Cost	Success Rate
SAC	-	1M	2.76K ± 0.95K	386.77 ±35.1	0.73 ±1.18	0.82 ±0.18
PPO	-	1M	24.34K ±3.56K	335.39 ±12.41	3.41 ±1.11	0.69±0.08
TD3	-	1M	1.74K ± 0.62K	318.12 ±21.9	0.47 ±0.08	0.70 ±0.09
SAC-Lag	-	1M	1.84K ± 0.49K	351.96 ±101.88	0.72 ±0.49	0.73 ±0.29
PPO-Lag	-	1M	11.64K ± 4.16K	299.99 ±49.46	1.18 ±0.83	0.51 ±0.17
CPO	-	1M	4.36K ±2.22K	194.06 ±108.86	1.71 ±1.02	0.21 ±0.29
Human Demo.	30K	-	39	347.523	0.39	0.97
BC	30K (1.0)	-	-	113.32 ±10.21	2.171 ±0.65	0.073 ±0.02
GAIL	30K (0.015)	2M	25.90K ± 8.15K	81.51 ±9.43	1.308 ±0.23	0.0 ±0.0
HG-Dagger	39.0K (0.76)	51K	56	116.393	1.979	0.045
IWR	35.8K (0.79)	45K	52	226.221	1.457	0.465
HACO	19.2K (0.48)	40K	130	143.287	1.645	0.139
PVP w/o TD	13.5K (0.34)	40.5K	70	252.447	1.277	0.220
PVP w/ Reward	12.8K (0.32)	40K	30	319.383	0.767	0.755
PVP (Ours)	14.6K (0.37)	40K	76.8 ±9.3	353.636 ±23.7	0.898 ±0.15	0.857 ±0.04
leading to inefficient learning of those critical human behaviors and even catastrophic forgetting. For example, the driving policy sometimes fails to master acceleration at the beginning of an episode, even though the human subject has already demonstrated the expected maneuver multiple times. This is because the demonstration of initial acceleration only lasts a short period of time and thus is scarce in the buffer. To address this issue, we balance the transitions coming from the human buffer and the novice buffer. In each training iteration, we sample two equally-sized batches  $b_{n}$  and  $b_{h}$  from  $\mathcal{B}_n$  and  $\mathcal{B}_h$  respectively, each has  $N / 2$  samples.  $N$  is the batch size for the policy update. By concatenating  $b_{n}$  and  $b_{h}$ , our method can balance the data from the human's demonstration and from the agent's exploration, and hence the ratio between two types of data in each SGD batch keeps  $1:1$ . Therefore, in the initial acceleration example above, the balanced buffer recalls the acceleration behavior, preventing catastrophic forgetting.
Tasks. We conduct experiments on various control tasks with different observation and action spaces. For continuous action space, we use three driving environments, MetaDrive safety benchmark [25], CARLA Town01 [8], and a customized driving environment built upon Grand Theft Auto V (GTA V), a popular video game. In these tasks, the agent needs to steer the target vehicle with low-level acceleration, braking, and steering, to reach its destination. Specifically, in MetaDrive safety environments, the agent needs to avoid any crash in the heavy-traffic scene with normal vehicles, obstacles, and parked vehicles. In MetaDrive, there exists a split of training and test environments, and we present the performance of the learned agent in a held-out test environment. To examine our method with different observation modalities, we use the sensory state vector in MetaDrive and GTA V and the bird-eye view image in CARLA as observation. For discrete action space, we use MiniGrid Two Room task [4], which involves agent exploration such as moving toward a door and opening the door before reaching the destination. The observation of MiniGrid is the semantic map of the agent's local neighborhood. Please refer to Appendix E for more information about the environment setup.
Evaluation Metrics. In MetaDrive safety benchmark, we report total safety cost as the number of crashes during training, which reflects the number of potential dangers exposed to the human subject during training. We also report episodic return, episodic safety cost, and success rate as the test performance of the agents. Episodic safety cost is the average number of crashes in one
Table 2: Results of different approaches in CARLA.
Method	Human Data	Total Data	Route Completion	Success Rate
PPO	-	1M	0.24 ± 0.013	0.0 ± 0.0
TD3	-	1M	0.11 ± 0.05	0.0 ± 0.0
BC	5K	-	0.42 ± 0.08	0.20 ± 0.10
HG-DAgger	6.8K	24K	0.64	0.47
IWR	5.7K	24K	0.69	0.60
HACO	4.8K	24K	0.52	0.40
PVP (Ours)	6.6K	24K	0.92 ± 0.05	0.73 ± 0.08
Figure 3: We visualize the action sequences generated by HACO and PVP agents in the same MetaDrive map who are trained to 40K steps. PVP has much smoother actions.
episode. The success rate is the ratio of episodes in which agents reach the destination to the total test episodes. In CARLA, we report route completion and success rate. Route completion is the ratio of the traveled distance to the length of the complete route. GTA V uses route completion and MiniGrid uses success rate to measure the performance. Except for total safety cost, the aforementioned metrics measure the test-time performance, which is tested when the agent runs independently without human involvement. For human-in-the-loop experiments, we also report the total number of human-involved transitions (human data usage) and the overall intervention rate, which is the ratio of human data usage to total data usage. These show how much effort humans make to teach the agents. We also design a user study to measure the experience of human subjects in Sec. 5.3.
Human Interfaces. To examine the generalizability of our method, we leverage multiple control devices: Xbox Wireless Controller (Gamepad), keyboard, and Logitech G29 Racing Wheel. We denote the MetaDrive tasks with three devices as MetaDrive-Gamepad/Keyboard/Wheel. As shown in Fig. 2, human subjects can takeover through control devices and monitor the training process through the visualization of environments on the screen. The Ethics statement is provided in Appendix A.
Experimental Details. We implement most of the code with Stable-Baselines3 [37]. Training results of various baselines in MetaDrive tasks are obtained from the open-source code by [26]. The RL baselines are repeated 5 times with different random seeds, while other human-in-the-loop methods are repeated fewer times due to limited human resources. In the training of the human-in-the-loop methods, a real human subject participates in each experiment and we do not use any simulated user input. During testing, there is no form of human involvement. For each experiment, we evaluate each checkpoint in the environment for multiple runs and report the average task-specified metrics as the performance of this checkpoint. We report the performance of the best checkpoint as the result of the experiment. We provide the standard deviation if the experiments are repeated multiple runs in tables and figures. All experiments with humans are conducted on a local computer with an Nvidia GeForce RTX 3080. The local computer can support real-time simulation and training. Hyper-parameters and other details are given in Appendix E and G.
Baselines. We test four native RL baselines: PPO [45], SAC [13], TD3 [10] and DQN [31]. We also test three safe RL baselines: Constraint Policy Optimization (CPO) [2], PPO-Lagrangian [48], SAC-Lagrangian [12]. In all baselines above, the reward function and cost function (for MetaDrive Safety Benchmark) are defined by the environment and can be accessed by the agents. We also test IL methods Behavior Cloning (BC) and GAIL [14]. Human-in-the-loop methods that learn from active human involvement are tested: Human-Gated DAgger (HG-DAgger) [17], Intervention Weighted Regression (IWR) [29] and Human-AI Copilot Optimization (HACO) [26].
5.2 Baseline Comparison
Comparing with RL Counterparts. Fig. 2 shows the curves of test-time performance. In MetaDriveGamepad, our method achieves 350 returns in 37K steps. This takes about one hour in the real-world HL experiment. TD3 baseline fails to achieve comparable results even after 300K steps of training. In CARLA, PVP agents learn to drive within 30 minutes with our method, while TD3 cannot solve the task. In GTA V, PVP can solve the task with 1.2K human data usage and 20K total data usage. The whole experiment takes only 16 minutes. TD3 instead utilizes 300K steps to achieve similar
Table 3: User study result. The maximum score for each item is 5.
HG-DAgger	IWR	HACO	PVP
Compliance	3.0 ± 0.8	4.0± 0.8	3.0 ± 0.2	4.8 ± 0.5
Performance	2.2 ± 1.0	3.7 ± 0.9	3.3 ± 0.9	4.8 ± 0.5
Stress	3.2 ± 0.9	4.5 ± 0.5	2.3 ± 0.9	4.7 ± 0.6
Figure 4: Evolution of proxy values.
performance. In MiniGrid tasks, our method successfully solves the tasks while vanilla DQN fails, showing that PVP can learn an exploratory solution and can be incorporated into discrete action space. We also show experiments on one easier and one harder MiniGrid environment in Appendix F.2, where PVP greatly improves learning efficiency.
Comparing with Human-in-the-loop Baselines. Table 1 suggests all tested HL methods achieve extremely low safety violations in training compared to vanilla RL and Safe RL methods, empirically supporting the preference alignment of the active human involvement, if we consider human preference is to avoid safety violation. Compared to other human-in-the-loop methods, our method costs the lowest human efforts in terms of human data usage and overall intervention rate, while greatly outperforming baselines in testing performance. Since MetaDrive has a training and test environment split, the result suggests PVP can learn high-quality agents with generalizability. Similar results are shown in CARLA in Table 2. Compared to RL baselines, HL methods achieve decent success rates and route completion rates even with only 24K environmental interactions. Compared to HL baselines, PVP achieves the best route completion rate.
Visualization. In Fig. 3, we visualize the action sequences of the agents trained by PVP and a human-agent shared control baseline HACO [26]. The angle and length of each arrow represent the steering and acceleration, respectively. The human subject's actions are marked with yellow. Compared to HACO method, PVP agent produces smoother actions, which explains its high user study scores shown in the next section.
5.3 User Study
We design a user study questionnaire to assess the experience of human subjects. Details are provided in the human subject research protocol in Appendix B. Three aspects are considered: (1) Compliance measures whether the behaviors of the agent satisfy human intents. For example, a highly compliant agent behaves like human such that the human subjects feel like they are completing objectives by themselves. (2) Performance is the subjective evaluation from human subjects on whether the agent can solve the primal task, e.g. driving to the destination in navigation tasks. This score should be low if the agent cannot learn a particular behavior or forgets it even though human subjects have taught the agent multiple times. (3) Stress gauges the cognitive cost human subjects pay to train the agent. A typical source of stress is the annoying oscillation and jitter the agent demonstrates. Unexpected behavior that requires human's instant reaction also creates stress. A lower score means more stress.
Table 3 shows our method is the most user-friendly method. On the one hand, we use a deterministic novice policy that greatly alleviates the jitter and unexpected behaviors, reducing stress. On the other hand, our method masters human behaviors and suppresses undesired actions with the balanced buffer and proxy value, improving the user experience in compliance and performance.
TD learning: As shown in Table 1 "PVP w/o TD", disabling TD learning via setting  $J^{\mathrm{TD}}(Q) = 0$  significantly damages the performance of PVP, suggesting that propagating information from human-involved states to other states is critical to the success of PVP.
PVP with reward: Both MetaDrive and CARLA results in Table 1 and 4 show that adding the environmental reward doesn't bring significant improvement in the learning performance, which might be caused by the fact that the native reward function might not be aligned with human preference.
Balanced buffer: We find that disabling balanced buffers (PVP w/o BB) makes the training unstable and leads to poor performance. This design avoids the catastrophic forgetting when the agent-generated data overwhelms the human demonstrations as in HACO [26].
Novice buffer: We find that PVP without the Novice buffer (PVP w/o NB) yields poor performance. The agent data stored in the novice buffer contains information on human preference and the forward dynamics of the environment. Thus, PVP does not discard the agent exploratory data as opposed to HG-DAgger [17].
Table 4: Ablation studies in CARLA.
Method	Human Data	Route Completion	Success Rate
HACO	4.8K	0.52	0.40
HACO w/o SP	5.1K	0.49	0.20
PVP w/o BB	2.8K	0.62	0.33
PVP w/o NB	4.2K	0.708	0.33
PVP w/ Rew.	4.4K	0.793	0.467
PVP w/ SP	12.3K	0.40	0.20
PVP w/ CQL	8.0K	0.622	0.266
PVP (Ours)	6.6K	0.92 ± 0.05	0.73 ± 0.08
Stochastic policy: We implement PVP based on Soft Actor-critic [13] so that the novice policy is now a stochastic policy. As shown in the "PVP w/ SP" in Table 4, introducing randomness in novice actions greatly reduces the performance. The human subjects report that the novice agents with stochastic policy oscillate frequently, making it hard to respond when the agents suddenly drive toward the side road. HACO [26] has similar human-AI shared control as PVP, but it adopts a stochastic policy. For comparison, we also implement HACO without a stochastic policy. "HACO w/o SP" suggests deterministic policy can not bring significant improvement to HACO.
Regularization on Q values: As discussed in Sec. 4.2, PVP objective can be interpreted as CQL with a newly introduced L2 regularization term on the Q values. We conduct the experiment to evaluate the performance of the vanilla CQL objective with other PVP designs in our reward-free online learning settings. As shown in "PVP w/ CQL" in Table 4, CQL objective yields worse performance. This experiment shows that the vanilla CQL doesn't work in this human active involvement setting. As shown in Fig. 4, the proxy value in the vanilla CQL method has a much larger magnitude which makes the values of behavior actions (the actions applied to the environment) and agent actions hard to distinguish. PVP has smoother proxy values with a clear margin between behavior and novice Q. CQL does not set a bound for the proxy value, thus proxy values in those extreme human actions are reinforced without a bound, making the novice policy rapidly learn those extreme actions, whereas PVP has bounded proxy values, leading to more stable training and better overall performance.
Learning through active human involvement is a promising approach enabling safe and efficient policy learning. In this work, we propose Proxy Value Propagation (PVP) that can effectively learn from the intervention and the corrective feedback from active human involvement. PVP can be seamlessly integrated into existing value-based RL methods and achieves highly efficient reward-free policy learning, without offline pretraining and reward engineering. Human-in-the-loop experiments show the proposed method achieves superior performance and better user experience across diverse environments with different action spaces and human control devices, showing that the learning from active human involvement is a efficient policy learning method aligning human preference.
Limitations. (1) We only apply our method to two value-based RL methods. Advanced techniques such as exploration encouraging [33] and prioritized replay buffer [44] can be added to further improve the result. (2) Our method is not applicable to tasks where humans can not provide demonstrations. (3) We assume that human always demonstrates desired actions. We will show in Appendix D that suboptimal human behaviors will damage learning. In this case, we can define a sparse cost function in the training environment and utilize constrained optimization [2] to penalize bad demonstrations. (4) We assume that human subjects are available and attentive throughout the entire training. While our method is proven to be effective even under heavy traffic environments, we plan to further enhance its sample efficiency. We will achieve this goal by conducting offline RL training and policy evaluation in the background or passively involving human subjects whenever the model is uncertain about the environment.
Acknowledgment: This work was supported by the National Science Foundation under Grant No. 2235012. The human experiment in this study is approved through the IRB#23-000116 at UCLA.
[54] Yunkun Xu, Zhenyu Liu, Guifang Duan, Jiangcheng Zhu, Xiaolong Bai, and Jianrong Tan. Look before you leap: Safe model-based reinforcement learning with human intervention. In Conference on Robot Learning, pages 332-341. PMLR, 2022.
[55] Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data in offline reinforcement learning. arXiv preprint arXiv:2202.01741, 2022.
[56] Jiakai Zhang and Kyunghyun Cho. Query-efficient imitation learning for end-to-end simulated driving. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
A Ethics Statement
Human subjects get paid to participate in the experiments. They can pause or stop the experiment if any discomfort happens. No human subjects are injured because all tasks we test are in virtual simulation. Each experiment will not last longer than one hour and subjects will rest at least three hours after one experiment. During training and data processing, no personal information is revealed in the collected dataset or the trained agents. We have obtained IRB approval to conduct this project.
B Human Subject Research Protocol
Recruiting and Requirement. For our study, we recruit 5 human subjects. All of them are college students and have the age from 20 to 30 years. Furthermore, every participant are required to have a valid driver's license and have experience in playing video game. Participation in our study is entirely voluntary. We ensure transparency by informing all subjects about the nature of the experiments and how their demonstrations would be used. Every subject provide written consent, confirming they are fully aware and in agreement. Additionally, the study is conducted with the IRB approval.
Onboarding Period. Participants are required to undergo a practice session, during which they drive under complete control to get a sense of the control devices (wheel, gamepad and keyboard), the environment interface, the dynamics of each environment and how an episode will fail or success. Each subject gets familiar with all the control devices and all the environments, which is indicating by performing at least 10 successful episodes, before they participate in the main experiments.
Main Experiment. During the initial stages of the formal experiment, subjects are advised to retain full control of the agent for the first one or two episodes. Subsequently, they may begin to let the agents taking control and intervene as necessary. The objective in all driving experiments is twofold: firstly, to safely navigate the vehicle to its designated destination, and secondly, to ensure the vehicle's operation aligns with traffic regulations and human preferences.
Subjects are encouraged to perform intervention whenever they perceive the vehicle might be in a dangerous situation, in violation of traffic rules, or in whatever scenario the human subjects feel they wouldn't behave in the way the novice policies do.
To ensure data integrity and counter potential proficiency biases, the order of experiments with different control devices, tasks and training algorithms is randomized for each subject. By doing this we mitigate the bias that a subject might become more familiar with the task when experimenting different algorithms.
User Study Questionnaire. We design a user study questionnaire to assess the experience of human subjects. The questionnaire is provided in Appendix B. Three aspects are considered:
- Compliance measures whether the behaviors of the agent satisfy human intents. For example, a highly compliant agent behaves like human such that the human subjects feel like they are completing objectives by themselves.
- Performance is the subjective evaluation from human subjects on whether the agent can solve the primal task, e.g. driving to the destination in navigation tasks. This score should be low if the agent cannot learn a particular behavior or forget it even though human subjects have taught the agent multiple times.
- Stress gauges the cognitive cost human subjects pay to train the agent. A typical source of stress is the annoying oscillation and jitter the agent demonstrates. Unexpected behavior that requires human's instant reaction also creates stress. A lower score means more stress.
The same questions are repeated for each algorithm the human subject experimenting on.
Compliance: Generally, do you think the agent trained with this method complies with your intention?
The higher score the better.
$(+)$  Good: Agent drives as you so that you don't even need to take over.
(-) Bad: Sudden unexpected behavior makes you mad.
Choices: 1, 2, 3, 4, 5
Performance: Do you think the agent trained with this method learns fast and performs well in terms of solving the task? The higher score the better.
$(+)$  Good: The agent learns fast so I don't need to take over too much in the later period.
(-) Bad: The agent forgets what it learns so I have to re-teach it.
(-) Bad: The agent never learns a specific behavior like accelerating or turning even though I have taught it so many times.
Stress: Do you think training with this agent is tired or stressed? The lower score the more fatigue and stress. A higher score means you are more relaxed.
Tiredness might come from many sources: Oscillating trajectory, unexpected behaviors, degrading performance that you have to re-teach, etc.
It is possible that your agent is not performing well but you don't feel tired training it. On the other hand, it is possible that your agent has good performance but still causes fatigue due to unexpected behaviors.
C Demo Video
Please find our demo video in the supplementary material. This video shows the footage of human experiments and the comparisons between agents learned by the baselines and the proposed method. The video contains three sections:
1. The first section shows how we learn the driving policy in CARLA task within 20 minutes. We also compare the behavior of agents learned from PVP and TD3 baseline.
2. In the second section, we show the footage of MetaDrive human experiment where the human subject uses a gamepad as the control device. We present the behavior comparison between PVP and TD3 baseline.
3. In the third section, we show the applicability of our method to other tasks. PVP performs well in GTA V and can drive smoothly on the highway. In the discrete control tasks, the behavior comparison between PVP and DQN baseline in MiniGrid Empty Room and Four Room are provided.
D Preference Alignment