SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhenghao Peng $^{\S}$ , Wenjie Mo $^{\S}$ , Chenda Duan $^{\S}$ , Quanyi Li $^{\dagger}$ , Bolei Zhou $^{\S}$ $^{\S}$ University of California, Los Angeles,  $^{\dagger}$ University of Edinburgh
Learning from active human involvement enables the human subject to actively intervene and demonstrate to the AI agent during training. The interaction and corrective feedback from human brings safety and AI alignment to the learning process. In this work, we propose a new reward-free active human involvement method called Proxy Value Propagation for policy optimization. Our key insight is that a proxy value function can be designed to express human intents, wherein state-action pairs in the human demonstration are labeled with high values, while those agents' actions that are intervened receive low values. Through the TD-learning framework, labeled values of demonstrated state-action pairs are further propagated to other unlabeled data generated from agents' exploration. The proxy value function thus induces a policy that faithfully emulates human behaviors. Human-in-the-loop experiments show the generality and efficiency of our method. With minimal modification to existing reinforcement learning algorithms, our method can learn to solve continuous and discrete control tasks with various human control devices, including the challenging task of driving in Grand Theft Auto V. Demo video and code are available at: https://metadriverse.github.io/pvp.
Reinforcement learning (RL) has been successfully applied in many domains, ranging from board game Go [46], strategy game StarCraft II [42], autonomous driving [18], and even nuclear fusion [7]. Existing RL methods assume the manually designed reward functions can fully express human intents and preferences. However, the resulting agents might exhibit biased, misguided, or undesired behaviors due to faulty reward functions [23, 40, 20]. Moreover, the poor sample efficiency as well as the safety concern due to the trial-and-error exploration prevent the real-world deployment of RL.
Human-in-the-loop methods are promising to achieve alignment, learning efficiency, and safety. Human-in-the-loop policy learning relies on human subjects to oversee the learning process of the autonomous agents, thus it can better align the learned behaviors with the preferences of humans compared with handcrafted reward functions. Different forms of human involvement in human-in-the-loop policy learning have been studied over the years. Human subjects can advise actions upon the requests of the robots [28] or provide preference-based feedback to assess the relative value of the collected trajectories [53, 5, 38, 52, 36, 11, 34]. These methods learn from passive human involvement, where the human subjects do not provide real-time feedback and intervention during data collection. For safety-critical tasks such as autonomous driving, safety is undoubtedly the first priority in human preference and the passive involvement methods yield unbounded risks in such settings. An increasing body of works focuses on active human involvement, where human subjects actively intervene and provide demonstrations during the execution time [17, 47, 29, 26]. With online correction and demonstration from human subjects, AI alignment and training-time safety of the system can be substantially enhanced.
In this work, we focus on learning from active human involvement and develop a simple yet effective method that can turn a common value-based RL method into a reward-free human-in-the-loop method with minimal modification. Our key insight is that we can learn a proxy value function from active human involvement, such that the proxy values encode human intents and guide policy learning to emulate human behaviors. Specifically, we propose the Proxy Value Propagation (PVP) method which labels high Q values to human actions and low Q values to agent actions that are intervened by the human subjects. The proxy values are then propagated to unlabeled state-action pairs in the agent's exploration through TD-learning. Value-based RL methods soon learn policies that align with human intents because of the value-maximization nature. Experiments show that PVP can be successfully applied to both continuous and discrete action spaces, and achieve higher learning efficiency compared to baselines in various tasks, including driving in Grand Theft Auto V (GTA V). It is also compatible with different forms of human control devices, including gamepad, driving wheel, and keyboard. We summarize our main contributions as follows:
1) We propose a simple yet effective method, Proxy Value Propagation, that can be integrated into existing RL algorithms to learn from active human involvement. Our method is reward-free and can be generalized across various task settings and human control devices.
2) The experiments show that the proposed PVP method enables superior performance and high learning efficiency in various tasks from the MiniGrid, MetaDrive, CARLA, to GTA V environment. User study further shows that PVP achieves better performance and is more user-friendly compared to other human-in-the-loop baselines.
AI alignment is one of the major issues in learning trustworthy intelligent agents for real-world applications. It is difficult to represent various human preferences into a scalar reward function in existing Reinforcement Learning (RL) methods [40, 6]. Meanwhile, the manually designed reward function, which might be misaligned with human preferences, often leads to undesired behaviors [23, 20]. As a promising complement to RL, Human-in-the-loop Learning (HL) can overcome costly reward engineering and convey human intents to the learning process directly through human involvement. Compared to imitation learning (IL) [14, 9], where the agent learns directly from high-quality human demonstration, HL methods benefit from interactive human involvement and feedback during the training, mitigating the possible distributional shift that usually happens when learning from offline data [39].
Preference-based RL. A large body of work focuses on learning human preference via ranking pair of trajectories generated by the learning agent [5, 11, 38, 52, 41, 36, 22, 51]. InstructGPT [34] aligns language models by first supervised learning in demonstration and then finetuning by the reward learned from human preference feedback. Preference learning can be applied to the tasks that human can not conduct, such as moving a six-legged Ant robot by assigning exact torque at each joint [5]. For those tasks that human can demonstrate, these methods do not fully utilize real-time feedback from human subjects during agent-environment interaction.
HL with Passive Human Involvement. Different from preference-based RL, human subjects can provide direct feedback to the learning agent during training through passive human involvement. Some works learn policy from human-provided evaluative feedback, a Boolean flagging correct or wrong actions [19, 3, 32]. This is similar to the intervention in our framework. However, in [32], humans provide high-level instructions, e.g. pointing to the left/right, while in PVP humans provide intervention and low-level demonstrations. The other line of work allows the neural policy to operate the robot and the human subjects can provide demonstration upon the requests from the learning agents [28, 30, 16]. The expert policy will intervene when uncertainty is huge, where the agent uncertainty is estimated by the variance of actions [30]. These methods reduce the cost of human resources but have potential risks to human subjects since they do not fully control the system. For example, when human subjects use these algorithms to train autopilot AI, they are exposed to significant risks if they are in a self-driving cars due to unpredictable agent behaviors.
Learning from Active Human Involvement. For safety-critical tasks such as autonomous driving, the safety of both the controlled vehicles and the human subjects is the top priority. There are many works that allow human subjects to proactively involve the agent-environment interactions based on their own judgment to ensure safety, which we call active human involvement. Human subjects
can terminate the episode if a near-accidental situation happens and such intervention policy can be learned [56, 1, 43, 35, 54, 50]. Recent studies explore active human involvement methods through intervention and demonstration in the human-agent shared autonomy [27, 30, 17, 47, 26, 16, 54]. However, previous methods do not fully utilize the power of human involvement. COACH [27] treats human labels as indications of advantage instead of simply as reward. Compared to COACH, our method accepts not only the feedback (the intervention signal) but also the human demonstration. Our method does not consider the time delay of human subjects explicitly as COACH does. Interactive imitation learning method (HG-DAgger) [17] does not leverage data collected by agents, while Intervention Weighted Regression (IWR) [29] does not suppress undesired actions likely intervened by human. Meanwhile, Expert Intervention Learning (EIL) [47] and IWR [29] focus on optimizing actions step-wise without considering the temporal correlation between steps. These drawbacks harm learning efficiency and thus incur more human involvement. Moreover, previous methods lack experiments to demonstrate the generalizability to different task settings and human control devices.
Policy learning aims at finding a policy to solve the sequential decision-making problem, which is usually modeled by a Markov decision process (MDP). MDP is defined by the tuple  $M = \langle S, \mathcal{A}, \mathcal{P}, r, \gamma, d_0 \rangle$  consisting of a state space  $S$ , an action space  $\mathcal{A}$ , a state transition function  $\mathcal{P}: S \times \mathcal{A} \to S$ , a reward function  $r: S \times \mathcal{A} \to [R_{\min}, R_{\max}]$ , a discount factor  $\gamma \in (0,1)$ , and an initial state distribution  $d_0: S \to [0,1]$ . The goal of conventional reinforcement learning is to learn a novice policy  $\pi_n(a|s): S \times \mathcal{A} \to [0,1]$  that can maximize the expected cumulative return:  $\pi_n = \arg \max_{\pi_n} \mathbb{E}_{\tau \sim P_{\pi_n}}[\sum_{t=0}^{T} \gamma^t r(s_t, a_t)]$ , wherein  $\tau = (s_0, a_0, \dots, s_T, a_T)$  is the trajectory sampled from trajectory distribution  $P_{\pi_n}$  induced by  $\pi_n$ ,  $d_0$  and  $\mathcal{P}$ . Here  $\pi_n$  defines a stochastic policy, while deterministic policy can be denoted as  $\mu_n(s): S \to \mathcal{A}$  and its action distribution is a Dirac delta distribution  $\pi_n(a|s) = \delta(a - \mu_n(s))$ .
The reward function imposes an assumption that the reward can fully reflect the intentions of the users and incentivize desired behaviors. However, this assumption may not always hold and the learned agent may obtain biased behaviors or figure out the loophole to finish the task [23, 40]. Revisiting the primal goal when developing learning systems, we find the reward is not a necessity since what we really want to achieve is the realization of human preference in the learned behaviors and, as suggested by [40], the ultimate source of information about human preferences are human behaviors.
Imitation Learning (IL) methods directly learn  $\pi_{n}$  from human behaviors. Assuming a human expert has a human policy  $\pi_h(a_h|s): S \times \mathcal{A} \to [0,1]$ , which outputs human action  $a_{h} \in \mathcal{A}$ . Note that human action shares the same action space as novice action. IL learns from the trajectories generated by human policy  $\tau_{h} \sim P_{\pi_{h}}$  and optimizes the novice policy to close the gap between  $\tau_{n} \sim P_{\pi_{n}}$  and  $\tau_{h}$ . Instead of generating an offline dataset and training novice policy against it [14, 9], we can incorporate a human subject into the loop of training for providing online data. This can mitigate the distributional shift since the data generated with human-in-the-loop has closer state distribution to that of the novice policy [39]. This can be modeled by introducing an intervention policy  $I(\cdot | s, a_{n})$  to describe human subjects' intervention behaviors. In earlier methods such as DAgger [39], the intervention policy is a Bernoulli distribution and the control authority switches back and forth between the novice and the expert. It is unrealistic to invite a real human subject to be involved in such training. Later studies allow the human subjects to intervene and take full control [49, 43, 26, 54], which we call such setting as learning from active human involvement. During training, a human subject accompanies the novice policy and can intervene with the agent by taking over the control to demonstrate desired behaviors. The intervention policy can be considered as a deterministic policy denoted by  $I(s, a_{n}): S \times \mathcal{A} \to \{0,1\}$  where  $a_{n} \sim \pi_{n}(\cdot | s)$  is agent's action. With notations above, the behavior policy  $\pi_{b}$  that generates actions during training is:
$$
\pi_ {b} (a | s) = \left(1 - I \left(s, \mu_ {n} (s)\right)\right) \delta \left(a - \mu_ {n} (s)\right) + I \left(s, \mu_ {n} (s)\right) \pi_ {h} (a | s). \tag {1}
$$
With such a model of active human involvement, we can now formulate our objectives.
Task-specified metrics. Our primal goal is to find novice agents whose behaviors are well-aligned with human preferences. In this work, we inform the human subjects of the primal goal of the tasks, e.g. navigating to the destination in driving tasks. They are also aware of how task-specified metrics, such as success rate and route completion provided by the test environments, are computed.
These metrics serve as a proxy for human preferences in evaluating trained agents' performance. Unlike prior work where these metrics were used as rewards, our learning agent cannot access them. The only supervision sources in our method are human interventions,  $I(s,a)$ , and demonstrations,  $a_{h} \sim \pi_{h}(\cdot |s)$ .
Preference Alignment. In our method, humans can intervene at any time. Most interventions occur in near-accidental situations or when agents are performing poorly. Conversely, lack of intervention indicates alignment with human preferences. Hence, another goal is to develop a novice policy that minimizes human interventions during shared control. In the next section, we will discuss our insights and how we build a concise, general, and efficient learning method to achieve these objectives.
We propose the Proxy Value Propagation (PVP) method which can transform a value-based RL method into an efficient reward-free human-in-the-loop policy optimization method that learns from active human involvement. PVP is compatible with various task settings, such as continuous and discrete action spaces, as well as various human control devices. In this section, we first summarize the basic workflow of value-based RL before introducing the motivation and the design of PVP. We then describe the implementation details.
Value-based RL: The proposed human-in-the-loop method results from the minimum modification of existing reinforcement learning methods. Thus, we briefly introduce the background of related methods. Value-based RL optimizes the value function and policy iteratively. On the value function side, we denote the state-action value and state value of policy  $\pi$  as  $Q(s,a) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^{t} r\left(s_{t}, a_{t}\right)\right]$  and  $V(s) = \mathbb{E}_{a \sim \pi(\cdot|s)} Q(s,a)$ , respectively. A neural network is commonly used to estimate the value function with Bellman backup:  $Q(s,a) \gets r(s,a) + \gamma \max_{a'} Q(s',a')$ , where  $s'$  is the next state. To learn the value network  $Q_{\theta}$  parameterized by  $\theta$ , stochastic gradient descent on the temporal difference (TD) loss is conducted  $J^{\mathrm{TD}}(\theta) = \mathbb{E}_{(s,a,s')} |Q_{\theta}(s,a) - (r(s,a) + \gamma \max_{a'} Q_{\hat{\theta}}(s',a'))|^{2}$ , where  $Q_{\hat{\theta}}$  can be a delay-updated target network. In this work, we adopt the TD learning in the reward-free setting. Remove the reward in the TD loss, the TD loss becomes:
$$
J ^ {\mathrm {T D}} (\theta) = \underset {(s, a, s ^ {\prime})} {\mathbb {E}} | Q _ {\theta} (s, a) - \gamma \max  _ {a ^ {\prime}} Q _ {\hat {\theta}} \left(s ^ {\prime}, a ^ {\prime}\right) | ^ {2}. \tag {2}
$$
On the policy side, based on the learned value function, the deterministic policy  $\mu_{n}$  parameterized by  $\phi$  can be learned by maximizing the Q values:  $J(\phi) = \mathbb{E}_sQ(s,\mu_n(s;\phi))$ . The optimal policy is expected to maximize Q values:
$$
\mu_ {n} (s) = \arg \max  _ {a} Q (s, a). \tag {3}
$$
4.1 Proxy Value Propagation
We illustrate the active human involvement of PVP in Fig. 1. During training, the human subject supervises the agent-environment interactions (Fig. 1 A). Those exploratory transitions by the agent are stored in the Novice Buffer  $\mathcal{B}_n = \{(s,a_n,s')\}$ . At any time, the human subject can intervene the free exploration of the agent by pressing a button in the control device (Fig. 1 B). While pressing the button, the human takes over the control and provides a demonstration of how to behave. During human involvement, both human and novice actions will be recorded into the Human Buffer  $\mathcal{B}_h = \{(s,a_n,a_h,s')\}$ . Concurrently with the human-agent shared control, our method keeps updating the novice policy by the novel Proxy Value Propagation mechanism (Fig. 1 C), which will be discussed later.
In the shared human-agent control, human intervention serves as a distinct indicator of suboptimal agent performance, which could result from the agent executing perilous actions or exhibiting ineffective behaviors. Thus, the optimal policy learned by the agent should (1) strive to approximate the behaviors demonstrated by the human subjects and (2) avoid performing actions that are intervened by humans.
The key insight of this work is that we can manipulate the Q values to induce desired behaviors, given that value-based RL has the nature to seek value-maximizing policy as Eq. 3. As shown in Fig. 1 C, for emulating human behavior and minimizing intervention, we sample data  $(s,a_{n},a_{h})$  from the
(A) Human oversees agent's exploration
(B) Human intervenes and provides demonstration
(C) Label proxy values and propagate through buffers
Figure 1: Illustration of Proxy Value Propagation. (A) Human oversees the agent's trial-and-error exploration with the environment. When the human subject does not intervene, the transitions will be recorded into the novice buffer  $\mathcal{B}_n$ . (B) When the human intervenes, both novice action  $a_n$  and human action  $a_h$  will be recorded into the human buffer  $\mathcal{B}_h$  but only the human action will be applied to the environment. (C) In training, we use the human buffer to compute proxy value loss and propagate the human intent knowledge to all transitions via TD loss without access to the reward.
human buffer and label the Q value of the human action  $a_h$  with  $+1$  and the novice action  $a_n$  with  $-1$ . This is achieved by fitting the Q network directly with PV loss:
$$
J ^ {\mathrm {P V}} (\theta) = \underset {(s, a _ {n}, a _ {h})} {\mathbb {E}} \left[ | Q _ {\theta} (s, a _ {h}) - 1 | ^ {2} + | Q _ {\theta} (s, a _ {n}) + 1 | ^ {2} \right] I (s, a _ {n}). \tag {4}
$$
The transitions in the novice buffer are not intervened by the human subject, meaning they are aligned with human preferences. Meanwhile, those transitions also contain information of the forward dynamics [24, 55]. To exploit the information contained in these transitions, instead of discarding these data as in [17], we propagate the proxy values to these states via TD learning in Eq. 2 and use those transitions together with those human-involved transitions for the policy learning. The final value loss is evaluated as follows:
$$
\begin{array}{l} J (\theta) = J ^ {\mathrm {P V}} (\theta) + J ^ {\mathrm {T D}} (\theta) = \underset {(s, a _ {n}, a _ {h}) \sim \mathcal {B} _ {h}} {\mathbb {E}} [ | Q _ {\theta} (s, a _ {h}) - 1 | ^ {2} + | Q _ {\theta} (s, a _ {n}) + 1 | ^ {2} ] I (s, a _ {n}) \\ + \underset {(s, a, s ^ {\prime}) \sim \mathcal {B} _ {h} \bigcup \mathcal {B} _ {n}} {\mathbb {E}} | Q _ {\theta} (s, a) - \gamma \max  _ {a ^ {\prime}} Q _ {\hat {\theta}} \left(s ^ {\prime}, a ^ {\prime}\right) | ^ {2} \tag {5} \\ \end{array}
$$
Then we follow the policy update process outlined in the base RL methods.
Connection to CQL. The proposed PVP method can be interpreted as adopting the Conservative Q-Learning (CQL) [21] objective for reward-free and online learning settings. It augments the CQL objective with an extra L2 regularization term imposed on the Q-values for human-involved transitions. In our online learning setting, Eq. 5 can be reformulated as:
$$
J (\theta) = \underset {\mathcal {B} _ {h}, I (s, a _ {n}) = 1} {\mathbb {E}} [ \underbrace {Q _ {\theta} ^ {2} (s , a _ {n}) + Q _ {\theta} ^ {2} (s , a _ {h})} _ {\text {L 2 R e g u l a r i z a t i o n o n Q}} + 2 + \underbrace {2 \left(Q _ {\theta} (s , a _ {n}) - Q _ {\theta} (s , a _ {h})\right) ] + \mathrm {T D} \text {l o s s .}} _ {\text {C Q L L o s s}}. \tag {6}
$$
CQL was originally proposed to mitigate the problem of overestimated Q-values in offline RL settings. These overestimations often lead to suboptimal policies due to the optimistic selection of actions with misleadingly high values. In our work, we deal with human actions and novice actions sampled from two different distributions, where overestimation might also occur. However, unlike CQL, PVP does not have access to a reward function, meaning the Q-values are not grounded in an estimation of true values. The additional L2 regularizer therefore serves to impose constraints on the Q-values, helping to prevent unbounded growth and potential overfitting. In Sec. 5.4, we compare the learned proxy Q-values under both CQL and PVP objectives. Our results indicate that human and agent actions are more distinguishable when learned through PVP.
Figure 2: Evaluation of PVP under four different environments with human control devices. For each environment, we plot the test-time performance curve of the agent trained by the proposed PVP and the RL counterpart TD3. The x-coordinate is the total number of environment interactions, which indicates the time steps the training agent (in RL method) or the human-agent system (in our method) experiences during training. Compared to the RL counterpart, the proposed method achieves much higher performance with superior learning efficiency.
Alternative to Reward Assignment. On the other hand, a more straightforward idea than PVP is to assign a reward of  $+1$  to human actions and  $-1$  to agent actions during intervention. Unfortunately, it is not practical since the Bellman backup is conducted on the transition triplet  $(s, a, s')$ , where one has to use future states' values to estimate current values. Therefore, the reward must correspond to the action from the behavior policy, the action  $a$  causes the transitions from  $s$  to  $s'$ . In our context, during involvement, the action  $a = a_h$  must come from human policy as the human subject is taking control. Though we can assign  $+1$  reward and compute value target in those human-involving transitions, we have no way to assign  $-1$  to the agent's actions because we don't know the next states caused by those actions and thus we can't compute the value target. It is also not practical to query the environment to get the next state  $s'' \sim \mathcal{P}(s, a_n)$  as the  $a_n$  is a potentially danger or undesired action and replaying it in the real-world environment is not feasible. In our preliminary experiment, we find the policy fails to learn anything regardless of the amount of human involvement provided. This is because the reward will be  $+1$  for all the human-involving transitions and the learning agent will find a pitfall to maximize its rewards: it always demonstrates undesired behaviors so that humans will always take control, which yields a  $+1$  reward.