SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here we provide a conceptual framework to describe the compliance of human intention. First, we introduce a ground-truth indicator  $C: \mathcal{S} \times \mathcal{A} \to \{0,1\}$  of the intention violation, denoting whether the action is undesired.  $C$  is not revealed to the learning algorithm.
$$
C (s, a) = \left\{ \begin{array}{l l} 1, & \text {i f} a \text {v i o l a t e s h u m a n i n t e n t i o n} \\ 0, & \text {o t h e r w i s e .} \end{array} \right. \tag {7}
$$
We will derive the upper bound of the discounted occurrence of intent violation, a measure of training time human intent compliance:
$$
S _ {\pi_ {b}} = S _ {\pi_ {b}} \left(s _ {0}\right) = \underset {\tau \sim P _ {\pi_ {b}}} {\mathbb {E}} \sum_ {t} \gamma^ {t} C \left(s _ {t}, a _ {t}\right), \tag {8}
$$
where  $P_{\pi_b}$  denotes the probability distribution of trajectories deduced by the behavior policy  $\pi_b$
During training, a human subject shares control with the learning agent. The agent's policy is a deterministic policy  $\mu_{n}(s)$ , the human's policy is a stochastic policy  $\pi_h(a|s)$ . The human subject intervenes  $I(s,a) = \text{True}$  under certain state and agent's action  $a_{n}$ . The mixed behavior policy  $\pi_{b}$  that produces the real actions to the environment is denoted as:
$$
\pi_ {b} (a | s) = \left(1 - I \left(s, \mu_ {n} (s)\right)\right) \delta \left(a - \mu_ {n} (s)\right) + I \left(s, \mu_ {n} (s)\right) \pi_ {h} (a | s), \tag {9}
$$
where we use Dirac delta distribution to represent the deterministic novice policy.
Two important assumptions on the human subject are introduced:
Assumption D.1 (Error rate of human policy). During human-AI shared control, the probability that the human subject produces an undesired action is bounded by a small value  $\epsilon < 1$ :
$$
\underset {s \sim P _ {\pi_ {b}}, a \sim \pi_ {h} (\cdot | s)} {\mathbb {E}} C (s, a) \leq \epsilon . \tag {10}
$$
Assumption D.2 (Error rate of intervention policy). During human-AI shared control, the probability that the human subject does not intervene when the action is undesired is bounded by a small value  $\kappa < 1$ :
$$
\underset {s \sim P _ {\pi_ {b}}} {\mathbb {E}} (1 - I (s, \mu (s))) C (s, \mu (s)) \leq \kappa . \tag {11}
$$
We introduce the following theorem and give the proof as follows.
Theorem D.3 (Upper bound of intent violation). The discounted occurrence of intent violation  $S_{\pi_b}$  of the behavior policy  $\pi_b$  is bounded by the error rate of the human action  $\epsilon$ , the error rate of the human intervention  $\kappa$  and the intervention rate  $\psi = \mathbb{E}_{s \sim P_{\pi_b}} I(s, a_n)$ :
$$
S _ {\pi_ {b}} \leq \frac {1}{1 - \gamma} (\kappa + \epsilon \psi). \tag {12}
$$
Proof. Consider Eq. 9, we have:
$$
\begin{array}{l} \underset {s \sim P _ {\pi_ {b}}, a \sim \pi_ {b} (\cdot | s)} {\mathbb {E}} C (s, a) = \underset {s \sim P _ {\pi_ {b}}} {\mathbb {E}} \left\{\left[ 1 - I (s, \mu_ {n} (s)) \right] C (s, \mu_ {n} (s)) + I (s, \mu_ {n} (s)) \underset {a \sim \pi_ {h} (\cdot | s)} {\mathbb {E}} C (s, a) \right\} \\ \leq \kappa + \epsilon \underset {s \sim P _ {\pi_ {b}}} {\mathbb {E}} I (s, \mu_ {n} (s)) = \kappa + \epsilon \psi \tag {13} \\ \end{array}
$$
The upper bound of  $S_{\pi_b}$ :
$$
S _ {\pi_ {b}} = \mathbb {E} _ {\tau \sim P _ {\pi_ {b}}} \sum_ {t = 0} \gamma^ {t} C \left(s _ {t}, a _ {t}\right) \leq \sum_ {t = 0} \gamma^ {t} (\kappa + \epsilon \psi) = \frac {1}{1 - \gamma} (\kappa + \epsilon \psi) \tag {14}
$$
E Environment Details
Figure 5: MetaDrive Safety benchmark.
Figure 6: CARLA Town01.
Figure 7: GTA V Training Environment.
Figure 8: MiniGrid (Four Room).
Table 5: Summary of the experiment environments.
Environment	Human Input Device	Observation Format	Action Space	Training & Test Set Split
MetaDrive	Gamepad, Keyboard, Wheel	State Vector	Continuous	Yes
CARLA	Wheel	Bird-eye View Image	Continuous	No
GTA V	Keyboard	State Vector	Continuous	Yes
MiniGrid	Keyboard	Semantic Map	Discrete	No
To avoid the potential risks of employing human subjects in physical experiments, we benchmark different approaches in four virtual simulated environments. We conduct experiments on various tasks with different observation and action spaces and human input devices. Table 5 summarizes the differences.
Continuous action space environments. For continuous action space, we use MetaDrive Safety Benchmark [25], CARLA Town01 environment [8] and a customized policy learning environment built upon Grand Theft Auto V (GTA V). In these tasks, the agent needs to steer the target car with low-level acceleration, brake and steering and move toward the destination.
MetaDrive Safety Benchmark preserves the capacity to evaluate the safety and generalizability in unseen environments since it uses procedural generation to synthesize an unlimited number of driving maps for the split of training and test sets, which is useful to benchmark the generalization capability of different approaches in the context of safe driving. We train agents in the training set, which contains 50 different scenes, and roll out the learning agents in the test set, which contains another 50 unique scenes. At each episode, the scene (road network) and the spawn location of traffic vehicles and ego vehicle are randomized. We use sensory state vector in MetaDrive as the observation for agents and thus apply MLP network architecture. When running pure RL methods in MetaDrive Safety Benchmark, a -1 penalty will be added to the reward when a crash happens. This is for a fair comparison with the safe RL methods who have access to the cost function directly. Specifically, we follow the default reward scheme in MetaDrive. The reward function in MetaDrive safety benchmark is composed of four parts as follows:
$$
R = c _ {\text {d i s p}} R _ {\text {d i s p}} + c _ {\text {s p e e d}} R _ {\text {s p e e d}} + c _ {\text {c o l l i s i o n}} R _ {\text {c o l l i s i o n}} + R _ {\text {t e r m}}. \tag {15}
$$
1. The displacement reward:  $R_{disp} = d_t - d_{t-1}$ , wherein the  $d_t$  and  $d_{t-1}$  denotes the longitudinal movement in meters of the target vehicle in Frenet coordinates of the target trajectory between two consecutive time steps. If the agent drives in the wrong way then the displacement reward will be multiplied by  $-1$ . The displacement reward provides a dense reward to encourage the agent to move forward. We set  $c_{disp} = 1$ .
2. The speed reward:  $R_{speed} = v_t / v_{max}$ , where  $v_t, v_{max}$  denotes current speed and maximum allow speed in current road in  $km/h$ , respectively. If the agent drives in wrong way then the speed reward will be multiplied by  $-1$ . We set  $c_{speed} = 0.1$ .
3. The collision reward:  $R_{collision} = 1$  if a collision with a vehicle, human, or object happens. Otherwise, it is 0. The coefficient  $c_{collision} = 5$ .
4. The terminal reward:  $R_{term}$  is non-zero only at the last time step. At that step, we set  $R_{disp} = R_{speed} = R_{collision} = 0$  and assign  $R_{term}$  according to the terminal state.  $R_{term}$  is set to  $+10$  if the vehicle reaches the destination (successes) and  $-5$  if the vehicle drives out of the road.
For measuring the safety, collision to vehicles, obstacles, sidewalk raises a cost  $+1$  at each time step. The sum of cost generated in one episode is the episodic cost.
In CARLA, we train and test agents in the Town01 environment. There exist many predefined routes in the town with different spawn locations and destinations. The length and spawn point of each route is randomized for each episode. We use the bird-eye view image in CARLA as observation and thus CNN is used as the feature extractor. In CARLA environment the reward function follows the reward function in MetaDrive safety benchmark.
In GTA V, we manually pick start and end coordinates in the world map to form multiple routes in different scenes. We split those scenes to the training and test sets. The training set contains two scenes with straight roads, turns, and medium traffic. The test set contains one different scene. For each episode, the traffic condition is randomized by the game engine. In GTA V environment the reward function follows the reward function in MetaDrive safety benchmark. The terminations include arriving at the destination (success), crashing with objects or vehicles for more than five frames (failure), and timeout (failure). The discrete keyboard input will be translated into continuous steering and acceleration signals for controls. Our customized human-in-the-loop compatible policy learning environment builds upon GTA V with a full set of well-defined observation, reward and termination conditions. The environment will be open-sourced and available to the community<sup>1</sup>.
For these tasks, the reward function is composed of two parts: a sparse termination reward (+10 when reaching the destination) and a dense moving reward (the distance moving toward the destination within one step).
Discrete action space environment. For discrete action space, we test on MiniGrid Two Room task [4]. MiniGrid Two Room is a task requiring heavy exploration since the agent needs to move toward a door and open the door before reaching the destination. The spawn locations, the destinations, door locations and the geometry of each room are randomized. The observation of MiniGrid is the semantic map of agent's local neighborhood. MiniGrid tasks only support using the keyboard as the input device. Only in the MiniGrid task, we render the agent's action in the environment so that the human can decide whether to take over or return back based on both the current state and agent's action. But this is not feasible in other tasks since other tasks require real-time responses from humans and there is not enough time for humans to observe agent's actions even if we plot those actions in the visualization interface. Following the default reward function as in original repository, in MiniGrid environment a sparse reward is used. When the agent reach the goal,  $+1$  reward is given and otherwise the reward is always 0.
Real-time experiment. Note that the local computer we use for human-in-the-loop experiments, which has an Nvidia GeForce RTX 3080 gpu, can support real-time simulation and training. In MetaDrive and CARLA, the physics simulation is run at  $10\mathrm{Hz}$  in the virtual world and in GTA V the frequency is  $30\mathrm{Hz}$ . That is, each environmental step will cause the virtual world advancing 0.1 / 0.033 seconds. After each environment interaction, PVP updates its policy once, inferring and backpropagating one SGD batch. Our experience suggests that the local computer can effortlessly support concurrent running of the simulation with human-agent shared control as well as the background policy update at the frequency in wall-time higher than the simulation frequency in the virtual world. That is, our training can run at frequency higher than  $10\mathrm{Hz} / 30\mathrm{Hz}$  so that the time-ellipse is actually faster in the real world than in the virtual world. We will limit the FPS to the system frequency so that the human subjects experience realistic time-ellipse.
Control devices. CARLA tasks use a Wheel, GTA V tasks use a keyboard, and MiniGrid tasks use a keyboard (provide discrete control signals). In all devices, a button is configured to indicate intervention. There is another button in the devices that activates an emergency stop. If any discomfort happens, human subjects can pause or stop the experiment immediately.
<sup>1</sup>The customized environment builds upon prior efforts on the communication between GTA V engine and Python interface: https://github.com/aitorzip/DeepGTAV, https://github.com/gdpinchina/DeeperGTAV, https://github.com/aitorzip/VPilot
F Extra Experimental Results
F.1 Impact of Control Devices
Table 6: The impact of different human input devices in MetaDrive benchmark.
Input Device	Method	Training	Testing
Human Data Usage	Total Data Usage	Total Safety Cost	Episodic Return	Episodic Safety Cost	Success Rate
Wheel	HACO	21.2K (0.53)	40K	42	250.039	1.453	0.355
PVP	10.3K (0.26)	40K	12	336.657	1.543	0.808
Gamepad	HACO	28.4K (0.71)	40K	55	71.37	1.97	0.0
PVP	7.4K (0.19)	40K	21	356.99	1.31	0.920
Keyboard	HACO	19.2K (0.48)	40K	130	143.28	1.645	0.139
PVP	14.6K (0.37)	40K	76	353.636	0.898	0.857
Table 6 presents the experiment results with different input devices in MetaDrive benchmark. In all settings, PVP agents outperform baseline method, showing the generalizability of PVP on different control devices. We observe that HACO [26] has performance discrepancy with different input devices. When using Gamepad, human subjects tend to push and pull the stick to the limits, producing extreme values. Extreme actions are particularly harmful to previous method as it does not incorporate the regularization terms on Q function to bound the Q values. When using the keyboard, the human subjects press arrow keys to indicate increasing/decreasing current steering/acceleration values for an increment. Therefore there will be fewer extreme values happening than using a Gamepad, which explains why the baseline HACO performs better with the keyboard compared to Gamepad. Due to less extreme values, when using Steering Wheel, HACO achieves good performance.
A case study in a toy environment. We retrieve the agents stored during the training of HACO and PVP in CARLA task. We test them in a straight road in CARLA town and plot their actuating signals in Fig. 9. In this task, the steering should be always close to zero. However, we find that as the training iterations increase, the HACO agents gradually demonstrate unstable steering and their steering is deviating. In human-AI shared control, such unstable behaviors force human subjects to involve frequently. In contrast, the PVP policies learn a much better solution in lane keeping.
Figure 9: Control signals in a straight road in CARLA.
Visualization of action sequences in training. In Fig. 10, we present the visualization of the trajectories during human-robot shared control. Comparing the visualization of HACO and PVP, we find that PVP generates smoother trajectories. Stable and smooth agent actions greatly improve human subjects' experience and relieve their stress during human-robot shared control. We can also find that as the training goes, PVP requires less human involvement. These results explain the performance of PVP and is aligned with the behavior shown in the supplementary video.
HACO @ 20K steps
PVP @ 20K steps
HACO @ 30K steps
PVP @ 30K steps
HACO @ 40K steps
Figure 10: In MetaDrive task, we use the top-down view to plot the trajectories of human-agent shared control. We use dense arrows to represent the actions that are applied to the environments. The arrow starts at the position of the car at that time step and its direction is the steering angle, projected into ego car's local coordination. The length of the arrow represents the acceleration. We use green and yellow arrows to denote agent's actions and human's actions, respectively.
PVP @ 40K steps
F.2 Extra Results in MiniGrid
(a) MiniGrid-Empty-Random-6x6-v0
(b) MiniGrid-TwoRooms-v0
Figure 11: MiniGrid results.
(c) MiniGrid-FourRooms-v0
In Fig. 11, we present the extra results in two additional MiniGrid environments. PVP achieves superior performance compared to RL baseline. Note that we use a CNN without recurrent module as the feature extractor. The performance of PVP can be further improved if we utilize the neural architecture with memory capability.
G Hyper-parameters
In MetaDrive safety benchmark [25] task, the observation is a state vector. There exists a split of training and test environments in MetaDrive. We present the result of the learned agent performing in the test environment.
In CARLA [8], the observation is the bird-eye view image in [84, 84, 5] shape, where 5 is the number of semantic channels. We train and evaluate the agents in the same NoCrashTown01 environment.
In GTA V, the observation is a state vector containing 2D LiDAR scanning for 240 total sampling points with max range  $50\mathrm{m}$  [15], vehicle state variables (speed, throttle, steering, heading), and navigation state variables (distance to road borders, distance to the next navigation point, collision to objects). There exists a split of training and test environments in GTA V. We present the result of the learned agent performing in the test environment.
In MiniGrid tasks [4] MiniGrid-Empty-Random-6x6-v0 (Empty Room), MiniGrid-MultiRoom-N2-S4-v0 (Two Room) and MiniGrid-MultiRoom-N4-S5-v0 (Four Room), the observation is the top-down view semantic map in shape [7, 7, 3].
In MetaDrive and GTA V, we use a MLP with two hidden layers, each has 256 units and ReLU activation, as the network architecture for the value network and policy network.
For CARLA task, since the input image has the same size of [84, 84] pixels, we use the same 5-layers CNN architecture with [16, 32, 64, 128, 256] filters in each layer. The corresponding kernel-size is [[4, 4], [3, 3], [3, 3], [3, 3], [4, 4]], and strides [3, 2, 2, 2, 4]. We use ReLU as activation functions between each layer.
For MiniGrid tasks, we use a 3-layer CNN architecture with [16, 16, 32] filters in each layer. All three layers have kernel-size 2 and there is a max-pooling layer between the first two layers. We use ReLU as activation functions between each layer.
Table 7: PVP (MetaDrive)
Hyper-parameter	Value
Discounted Factor γ	0.99
τ for Target Network Update	0.005
Learning Rate	0.0001
Steps before Learning Start	100
Steps per Iteration	1
Gradient Steps per Iteration	1
Train Batch Size	100
Q Value Bound	1
Table 8: PVP (CARLA)
Hyper-parameter	Value
Discounted Factor γ	0.99
τ for Target Network Update	0.005
Learning Rate	0.0001
Steps before Learning Start	100
Steps per Iteration	1
Gradient Steps per Iteration	1
Train Batch Size	128
Q Value Bound	1
Table 9: PVP (GTA V)
Hyper-parameter	Value
Discounted Factor γ	0.99
τ for Target Network Update	0.005
Learning Rate	0.0001
Steps before Learning Start	100
Steps per Iteration	1
Gradient Steps per Iteration	1
Train Batch Size	100
Q Value Bound	1
Table 10: PVP (MiniGrid)
Hyper-parameter	Value
Discounted Factor γ	0.99
τ for Target Network Update	0.005
Learning Rate	0.0001
Steps before Learning Start	50
Steps per Iteration	1
Gradient Steps per Iteration	32
Target Network Update Interval	1
Train Batch Size	256
Q Value Bound	1
Exploration Reducing Fraction	0
Random Action Probability Initial Value	0
Random Action Probability Final Value	0
Table 11: HACO (MetaDrive)
Hyper-parameter	Value
Discounted Factor γ	0.99
τ for Target Network Update	0.005
Learning Rate Actor	0.0003
Learning Rate Critic	0.0003
Learning Rate Entropy	0.0003
Steps before Learning Start	100
Steps per Iteration	1
Gradient Steps per Iteration	1
Target Network Update Interval	1
Train Batch Size	128
CQL Loss Temperature	1.0
Table 12: HACO (Carla)
Hyper-parameter	Value
Discounted Factor γ	0.99
τ for Target Network Update	0.005
Learning Rate Actor	0.0003
Learning Rate Critic	0.0003
Learning Rate Entropy	0.0003
Steps before Learning Start	100
Steps per Iteration	1
Gradient Steps per Iteration	1
Target Network Update Interval	1
Train Batch Size	128
CQL Loss Temperature	1.0
Table 13: TD3 (MetaDrive)
Hyper-parameter	Value
Discounted Factor γ	0.99
τ for Target Network Update	0.005
Learning Rate	0.0001
Steps before Learning Start	10000
Steps per Iteration	1
Gradient Steps per Iteration	1
Train Batch Size	100
Table 14: TD3 (Carla)
Hyper-parameter	Value
Discounted Factor γ	0.99
τ for Target Network Update	0.005
Learning Rate	0.0001
Steps before Learning Start	10000
Steps per Iteration	1
Gradient Steps per Iteration	1
Train Batch Size	100
Table 15: TD3 (GTA V)
Hyper-parameter	Value
Discounted Factor γ	0.99
τ for Target Network Update	0.005
Learning Rate	0.0001
Steps before Learning Start	10000
Steps per Iteration	1
Gradient Steps per Iteration	1
Train Batch Size	100
Table 16: DQN (MiniGrid)
Hyper-parameter	Value
Discounted Factor γ	0.99
τ for Target Network Update	0.005
Learning Rate	0.0001
Steps before Learning Start	50
Steps per Iteration	1
Gradient Steps per Iteration	32
Target Network Update Interval	1
Train Batch Size	256
Exploration Reducing Fraction	0.3
Random Action Probability Initial Value	0
Random Action Probability Final Value	0.05