Human-AI Shared Control via Policy Dissection
Authors: Quanyi Li
Zhenghao Peng
Haibin Wu
Lan Feng
Bolei Zhou
Venue: NeurIPS 2022 Accept (2022)
Keywords: Human-AI interaction
Interpetability
Decision and Control

Abstract: Human-AI shared control allows human to interact and collaborate with autonomous agents to accomplish control tasks in complex environments. Previous Reinforcement Learning (RL) methods attempted goal-conditioned designs to achieve human-controllable policies at the cost of redesigning the reward function and training paradigm. Inspired by the neuroscience approach to investigate the motor cortex in primates, we develop a simple yet effective frequency-based approach called Policy Dissection to align the intermediate representation of the learned neural controller with the kinematic attributes of the agent behavior. Without modifying the neural controller or retraining the model, the proposed approach can convert a given RL-trained policy into a human-controllable policy. We evaluate the proposed approach on many RL tasks such as autonomous driving and locomotion. The experiments show that human-AI shared control system achieved by Policy Dissection in driving task can substantially improve the performance and safety in unseen traffic scenes. With human in the inference loop, the locomotion robots also exhibit versatile controllable motion skills even though they are only trained to move forward. Our results suggest the promising direction of implementing human-AI shared autonomy through interpreting the learned representation of the autonomous agents. Code and demo videos are available at https://metadriverse.github.io/policydissect

Human-AI Shared Control via Policy Dissection
Quanyi L Zhenghao Peng", Haibin Wu', Lan Feng', Bolei Zhou Centre for Perceptual and Interactive Intelligence, 'ETH Zurich, FUniversity of Edinburgh, & UUniversity California, Los Angeles
Abstract
Human-AI shared control allows human to interact and collaborate with au- tonomous agents to accomplish control tasks in complex environments. Previ- ous Reinforcement Learning (RL) methods attempted goal-conditioned designs to achieve human-controllable policies at the cost of redesigning the reward func- tion and training paradigm. Inspired by the neuroscience approach to investigate the motor cortex in primates, we develop simple yet effective frequency-bass based approach called Policy Dissection to align the intermediate representation the learned neural controller with the kinematic attributes of the agent behavior. With- out modifying the neural controller or retraining the model, the proposed approach convert given RL-trained policy into uate the proposed approach on many RL tasks such as autonomous driving and locomotion. The experiments show that human-Al shared control system achieved by Policy Dissection in driving task can substantially improve the performance and safety in unseen traffic scenes. With human in the inference loop, the loco- motion robots also exhibit versatile controllable motion skills even though they are only trained move forward. Our results suggest the promising direction of implementing human-Al shared autonomy through interpreting the learned representation of the autonomous agents. Code and demo videos are available https: /ate:s//mmtarrrr/ verse. ....ggiio github. 1o/1io///// htopoooccyisseeeeecccc...cccc
policy. We eval-
Introduction
In recent years, autonomous agents trained from deep Reinforcement Learning (RL) have achieved huge success a range of applications from robot control [68, 56, 37, 72], autonomous driving [25, 7, 29], to the power system in smart building [44]. Despite the capability of discovering feasible policy under unknown system dynamics in a model-free setting [66]. the learning-based agents lack sufficient generalizability [14] in unseen environments, which hinders their real-world deployments. Furthermore, it remains difficult to understand the internal mechanism and the decision-making of the neural network [30, 78], especially when abnormal behaviors happen [10, 77, 49]. Thus the absence of generalizability, transparency and controllability limits the application of the end-to-end neural controllers in safety-critical systems.
One potential direction to make the neural controller safer and more trustworthy is incorporating human into the inference loop, which we refer to human-Al shared control. With human involve- ment, the shared autonomy systems can achieve substantial improvement over the performance and safety [54, 15, 55], even new tasks and unseen environments. Human-Al shared control has been implemented in various forms, such as enforcing intervention [28, 51] or providing high-level commands [15] to the AI models. Previous works mainly explore goal-conditioned reinforcement learning (GCRL) as a form of human-Al shared control, where a high-level goal is fed as an input the policy during training [60, 15, 25, 57J. Thus the behavior of the learned agent can be controlled by varying the input goal [47, 38, 79, 37]. However, training goal-con cond--onttio policy requires additional modifications the observation design, the reward function and the training
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Pybullet-A1 (Top-down View)
Ant (Top-down View)
Move Left
Move Right
(Side View)
Move down
Move
8pin
MetaDrive
Stop
BipedalWelker (Side View)
Jump IsaacGym-Casse Cassie (Side View)
Lane Change
Lane Change
Tp Toe
Crouch
Back Fiip
Figure I: Examples of human-AT shared control enabled by Policy Dissection. Each autonomous agent trained to solve its primal task without changing the environment like the reward function and observation. After training, the stimulation-evooer map is then generated by Policy Dissection as the interface for human to steer the agent. In various environments, behaviors such as jumping, spinning, redirection, backflipping can be triggered by human subject. These examples demonstrate that Policy Dissection is a general method for making a pretrained neural policy controllable.
process. In the case of bipedal robot for example, difficult design reward functions for each behavior like crouching, backflipping, forward jumping. Also, initial state distribution and goal selection criteria should be meticulously designed for successful GCRL training [50, 57J. On the
other hand, the human control only happens at the goal level, still lacking in the interpretability of low-level neural policy.
In this work, we develop minimalist approach called Policy Dissection, to enable human-Ar AI shared control on autonomous agents trained on wide range of tasks. Policy Dissection does not impose assumptions on the agent's ' training scheme like specially-designes reward function, and it requires neither retraining the controller's 's network nor modifying the environment. Policy Dissection achieves human-Al shared control by firstly dissecting the internal representations of the learned policy and aligning them with specific kinematic attributes, and then modulating the activation the internal representations to elicit the desired motor skills.
Policy Dissection inspired by the neuroscience studies motor cortex of primates, where exerting electrical stimulation on different areas of motor cortex can elicit meaningful body movements [21] Thus, stimulation-evoker map can be built to reveal the relationship between the evoked body movements and the motor neurons located in different areas of motor cortex [21, 20, 34J. In order to replicate such diimulation-evoker map inside controller powered by artificial neural network, the proposed Policy Dissection conducts frequency analysis and matching strategy for associating the kinematic attribute and the unit activation. The procedure of building a stimulation-evoker map by Policy Dissection is illustrated in Fig. 2. Concretely, we first roll out the given well-trained policy for many episodes and collect a set of time series of the activities of all units, and the variation in kinematics and dynamics. Afterwards, for each kinematic attribute we identify one unit whose activity align best with the variation of this kinematic attribute following the principle that there the lowest frequency discrepancy between both the time series. After finishing the alignment for

---

Controller
Deceleration
Frequency matching
Human-AI Shared Control
Legged Robol
Nol Mo po
Slimulation-evcker Map
w.tooodrr o 9
Spin
Move Up
Horling
Figure 2: Overview of Policy Dissection. Taking the Ant robot example, method first identifies the connection between motor primitives and kinematic attributes (painted in the same colors). After that, human can activate units evoke desired behaviors by stimulating one more motor primitives. For example, stimulating motor primitive related to yaw rate makes the Ant spin, while activating motor primitive related to velocities in different axes brings moving-up deceleration.
all kinematic attributes, the identified units are then called motor primitives' whose activation can change corresponding kinematic attributes. Since behavior be described by the change of one or more kinematic attributes, the stimulation-evoker map can be built for recording the relationship of motor-primitives' activation and evoked behaviors. For example, the moving up behavior of the Gym-Ant shown in Fig. 2 can be described by turning upward (increasing yaw) and increasing upward speed (vy), and thus we can activate corresponding two motor primitives to elicit the desired movement. In conclusion, Policy Dissection constructs an interpretable control interface on top of the trained agent for human to interact with and evoke certain behaviors of the agent, thus achieving human-Al shared control.
We evaluate the proposed method several RL agents ranging from locomotion robots to autonomous driving agents simulation environments. Experimental results suggest that meaningful motor primitives emerge in the internal representation. As shown in Fig. novel behaviors can be deliberately evoked by activating units related to certain kinematic attributes, even though they are the necessary behaviors solve the primal task. In the quantitative evaluation, we use the human- Al shared control system enabled by Policy Dissection to improve the generalization in test-time unseen environment and achieve zero-shot task transfer. an autonomous driving task, we first train baseline agents under mild traffic conditions and evaluate the driving policy on new test environments containing unseen accidental scenes. Different from the poor performance of the baseline agent in these new environments, the human-Al AI shared control system achieves superior performance and safety guarantee, with 95% success rate and almost zero cost. the quadrupedal locomotion task, robot with only proprioceptive state input can avoid obstacles with shared control, even though its primal task is moving forward. These results show that the proposed Policy Dissection not only helps understand the learned representations of the neural controller, but also brings the intriguing potential application of human-Al shared control.
2 Related Work
Human-AI Shared Control. Existing methods of human-Al shared control can be roughly divided into two categories. The first category is to have human in the training loop and then the well- trained policy can be executed without human in test time [80]. By incorporating human into the training, previous works successfully improve the performance on visual control tasks such as Atari game [1, 58, 70]. Robotic control tasks also benefit from human feedback [28, 43, 64, 51, 71]. The other category is to have human in both training and test time to accurately accomplish human- assistive tasks. A series of works study this problem based on the Atari Game [54, 59, 8. 31]. In addition, the human-Al shared control system is built on autonomous vehicle [19], robotic arms [27] and in multi-agent setting [31]. The reported results various tasks indicate the effectiveness and efficiency of training and coordination with human-assistive AI. Unlike prior works, our method
MMotor primitive is used by researchers in biological motor control to indicate "building blocks of movement generation" [52, 32]
3

---

Layer
Layer
Policy Rollout
Heading
Heiht
V,
V.
Kinematic Attribule Tracking
Neural Activity Recording
Figure 3: We first roll out the trained policy and record the neural activities and track kinematic attributes, like yaw and velocity. After frequency matching. kinematic attributes are associated with certain units, which are further called motor primitives. The curves of kinematic attributes and the aligned motor primitive are painted in the same colors. For clarity, we only show the result of one recorded episode and proportion of units, and the curves of units are sorted by their amplitude.
allows human to collaborate with Al in test-time, while eschewing the need for any training-tinn human involvement. In experiments, shared control systems implemented several robotic control tasks to show the effectiveness of our method.
Neural Network Interpretability. There is a growing interest in the AI community to understand what is learned inside the neural network. Previous interpretability methods mostly focus on explain- ing the networks trained for visual tasks [81, 62, 1 11, 82, 40]. Various empirical studies have found that meaningful and interpretable structures emerge in the internal representation when trained for the primal tasks. For example, object detectors emerge in scene classifier [6], and disentangled layout and attribute controllers emerge from image generation model [74]. Varying the disentangled hidden features of the deep generative model can edit the content of the generated images. [5, 63, 73]. On the other hand, visuomotor control tasks can also acquire visual understanding to explain the behavior chosen by agent with visual input through saliency map [65, 3, 53, 12, 26, 22].
Neuroscience Inspired Control. The complex behaviors shown by recent autonomous agents prompt researchers to develop new analysis methods. As a source of inspiration, neuroscience provides useful tools [48, 34], and effective procedures [24, 18, 46] for understanding animal brains which are also like black-box systems. Recent works try to develop neural network- -baed policies from the perspective of neuroscience [17, 4, 76, 13J. closely related work on understanding learned agent, Merel et al. [46] use RSA [35] with CKA [33] similarity index to reveal that behaviors spanning multiple timescales are encoded distinct neural layers. Apart from providing the neuroscientific tools, many concepts from neuroscience, such motor primitive also inspire researchers to design generalizable controllers [69, 67, 45].
Policy Dissection Method
Policy Dissection aims at deriving human-controllable policy from a well-trained RI. agent. has two unique features enable human-AI shared control: First, it operates on well-trained RL agent without re-training or modifying the original reward/environment and the policy architecture. We only need to roll out the agent for several episodes and log its neural activity and the change of kinematic attributes. Based on the collected data, kinematic attributes are aligned with certain units which are further termed as motor primitives. Thus our method non-intrusive to the primal task and trained model, and applicable to wide range of environments. Second, activating certain motor primitives can elicit generalizable motor skills that even the agent is not trained to perform. As shown in Fig. 1, the bipedal Cassie robot can be steered to tip toe, crouch and back flip even if it is only trained move forward in the primal task. In the following subsections, we will introduce the four essential steps of the Policy Dissection method in detail.

---

3.1 Monitoring Neural Activity and Kinematics
Since multi-laye perceptron (MLP) is the most common network architecture used in many RL tasks, we assume the neural controller is an MLP with Z units, which has L hidden layers and all layers have the same number of / hidden units? We denote neuron by i, as g/d Z. where 1= I..... I. is the index of layer and i = I,.,, I is the neuron index in each layer, and its output at time step is zt Let J be the total number of kinematic attributes. The agent's 's kinematic attributes are represented by S = ] = I,....J. J, and the measurement of each kinematic attribute s² at time step is sy. We define the motor primitive as the unit m² E Z that can steer the agent and cause change of the corresponding kinematic attribute. We assume that each kinematic attribute s has corresponding motor primitive m² to drive the attribute to a desired state, and thus there are also J motor primitives, denoted by .M- = {m²² j. I,...JJ JJ Our goal is discover the motor primitive m² by associating the units with certain kinematic attribute s).
As shown in Fig. 3, we take the Ant robot an example. Policy Dissection starts by rolling out the pretrained agent for several episodes and tracking the activation of all units as well as the kinematics, such as yaw rate and velocity. We record the 1 L time series {z' = I Ls ....]]/= measuring the neural activities and J time series of kinematic attributes {s² |0 "2,...]] for further analysis.
22
3.2 Associating Units with Kinematic Attributes
Observing the neural activity and kinematic measurement recorded in Fig. 3, we can find that several repeated kinematic patterns appear in different frequencies when executing the policy, which reminiscent of animal behavioral research [21, 20, 34]. suggests that the motor skills has its predominant frequency component [46], controlling fast kinematics and slow kinematics. Since neural activation signals also encode distinct frequency information as shown in Fig. 3, this resemblance inspires us to make the following hypothesize:
For one kinematic attribute, the unit with minimal frequency discrepancy is the motor primitive.
To this end, we process time series of neural activity and kinematic attributes in frequency domain, where the difference of predominant frequency between two signals can be clearly revealed and the phase discrepancy of two time series are eliminated. As a consequence, this allows us to analyze the recorded data and associate kinematic attributes with units whose activity best matches the variation of target kinematic attributes. Concretely, all time series {x} including the neural activities {z/+)) and change of kinematic attributes {s/} are transformed into the frequency domain through Discrete-time Short-time Fourier transform (STFT):
(TFT(d,w(x)+ = Σ T. W( - die
-iot jut.
(1)
STFT splits the time series x = [x1, T2. into sequence of temporal window and d E [1, +oo] denotes the time step of the center of the temporal window. STFT then conducts Fourier transform in each temporal window. w E [0, +oo] denotes the frequency. To reduce spectral leakage and reserve the frequency information as much as possible, the window function W(t) used in Policy Dissection is blackman window. Moreover, the window length is 64 with hop length 16. We apply STFT instead of naive Fourier transform since it can retain temporal information by breaking the time series into several overlapping chunks of frames. This feature is important since in the following frequency matching we focus on the similarity of resulting spectrograms as well as the temporal correlation of two signals. We use the spectrogram to characterize measurement:
SG(d, w|x) = [s)FT(d,
After obtaining the spectrograms for all kinematic attributes and neural sreiiities,,requuennn discrep- ancy can be computed for each neuron-kinematies pair (z ($):
Dis ". .s))
Σ Σ (( SG(d, w|s))l
(3)
"To keep the notation concise, assume all hidden layers of MLP have the same number of units 1. MLPs with varying number of hidden units per layer still come the same conclusion.
5

---

For each kinematic attribute, frequeney matching is applied to select the unit with minimal average discrepancy across N collected episodes as the motor primitive responsible for triggering the change of desired kinematic attributes:
m argmin
Σ N
Dis(z!),s))
3.3 Building Stimulation-evoked Map
A behavior &ar can be described by changing a subset of kinematic attributes S C S, which can be achieved by activating a set of corresponding motor primitives M C ... Therefore, these movement generation building blocks, motor primitives, are associated with certain behaviors, inducing the Cimulation-evokel map {[M* b²)) }, k 1, .... K, where K is the number of discovered behaviors. Taking Cassie's 's back-flip shown in Fig. 1 as an example, this behavior can be described by increasing height 2. pitch and knee force. Therefore, we activate three corresponding motor primitives selected by Eq. 4 with output calculated by Eq. 8 to make the robot back flip. In practice, it is possible that more motor primitives are required. Again, taking the back-flip as an example, the force for both knee may be different after activating corresponding units, and thus giving the robot rolling velocity, which can be further eliminated by activating roll related motor primitive.
3.4 Steering Agent via Stimulation-evoked Map
Now the last thing for evoking behavior b²t is to determine the output value for motor primitives S At time step t, activating the motor primitive m² related to kinematic attribute s stands for applying its derivative š to the agent. For example, activating the speed related units equals to applying the derivative of speed, acceleration, to the agent. Therefore, changing the kinematic attribute s at time step to by activating the motor primitive m² with a unit output ed can be formulated as:
=
s + Lo
f(v))dt.
(5)
where 7 = 41 - lo is the activation period of the motor primitive m², constant wd is the time-invariant output of m² and f(v) a mapping from motor primitive output value v² $².
For steering the agent, our goal is to understand some properties of f(v), especially the correlation between f(v) and so that we amplify or reduce the s by choosing u or u Taking the Ant robot of Fig. as an example, activating the unit controlling y-axis movement with positive value can drive the Ant move left, and negative value makes the Ant move right. We will illustrate the approach to probe the correlation in the following subsection.
Identifying Correlation Coefficient. We compute the correlation coefficient through spectral phase discrepancy. we first find the predominant frequency component for each signal:
argmax Σ
SG w mm) SG(d,
At each window of STFT, we can compute the phase discrepancy of the transforms of m and s' at the predominant frequency w". The average phase discrepancy at all temporal windows is computed
across N collected episodes:
p' D.N D.N
D.NI Σ (="mF"((;""'m'') (STFF
where D is the number of temporal windows and & is the function retrieving the phase. We then normalize p to get correlation coefficient p between m and s as p² most important thing that whether the si will be amplified or reduced by overwriting the output of m² with a positive or negative value. After that, vd is finally determined by
12p'l/TT p tells us the
d
(8)
where ai is magnitude coefficient determining the scale of ز and thus 3ú We usually determine the oi by observing the performance after trying evoking the desired behavior with different e², while it can also be determined by an external PID controller, which takes the error & - si between target state å and current state sy as input and outputs a automatically.

---

Training
x50
Test
x20
MetaDrive SafetyEnv
PybutettA11 Erv
Isaadyym
Figure 4: Examples of training and test scenes used in the generalization experiment. In MetaDrive SafetyEnv, the held-out 20 test scenes are more complex compared to 50 training scenarios. For quadrupedal robot locomotion task, the robot is trained to traverse open but bumpy terrains and tested on the terrain full of obstacles where obstacle avoidance ability needed to reach the destination. In the Cassie environment, the agent is trained to move forward. However, crouching, jumping and sidestepping are required to reach the destination the test environment.
Experiments
We first apply Policy Dissection to an autonomous driving task study the learning dynamics in Sec. 4.2. We demonstrate that the emergent pivotal motor primitives are highly correlated with the success of policy learning. With the trained driving policy, we show a novel application of the shared autonomy in Sec. 4.3: improving test-time generalization, where Human-AL Al shared control can be an alternative to overcoming the generalization gap between training and unseen test-time environment. The experiment demonstrates that our method finds not only the correlation but causality between unit output and kinematics. In Sec. 4.4, we conduct experiments on quadrupedal robot and demonstrate that quadrupedal robot trained to traverse complex terrain with only proprioceptive states can transfer to accomplish the obstacle avoidance task with a bit human effort, even if does not have any information of the surrounding. For qualitatively evaluating our method, we provide demonstration in Sec. 4.5, where a bipedal Cassie robot trained for moving forward can solve challenging parkour environment with the help of human subject. In Sec. 4.6, we investigate the coarseness of the goal-conditioned control enabled by Policy Dissection through a command tracking experiment, and compare it with the state-of- the-art conliconni controller in IsaacGym [42, 57].
4.1 Experimental Setting
The experiments conducted on Cassie and ANYmal in IsaacGym [57], Ant and Walker in Mu- JoCo [68] and the BipedalWalker [9] follow the general setting. Other experiments conducted on MetaDrive and Pybullet-Al environments are introduced as follows:
MetaDrive. We train agents to accomplish autonomous driving task the safety environment of MetaDrive [39]. In this task, the goal is maneuver the target vehicle to its destination with low-level continuous control actions. An episode terminates only when the vehicle arrives at the destination, drives out of road, or collides with the sidewalk. The observation of autonomous vehicle consists of map, navigation, lidar, and state information such as speed, heading and side distance. More training environment details can be found in appendix.
Pybullet-A1. The legged robots locomotion experiments are conducted on the Pybullet-Unitree Al [16] robot. We follow the same environment setting used by Locotransformer robot [75], including terrain shape, obstacle distributions, sensors, reward definition, and termination condition. Two agents are trained. One "insensible" agent with only proprioceptive state is trained to walk uneven terrain, while the other agent with Locotransformer is directly trained to avoid obstacles. The shared control system will be built the "insensible" agent, where human subjects help the agent sidestep obstacles.
Training Set and Test Set. As shown Fig. 4, for both autonomous driving and robot locomotion tasks, we prepare training set and held-out test set. For driving task, the test set is used to benchmark the test-time generalizability of human-Al shared control system and Al-only control system. We train autonomous driving policies in 50 different environments where the traffic condition
7

---

1.0
Human Involvement
Episodic Return
Episodic Cost
Success Rate
0.8
Method
Human
405.57 1 94.79
357.21 58.
0.47 = 0.34
0.95 = 0.03
0.6
DajPiiniivve 1, Spead)
Trannin Success Rate
SACW/OH
263.85 + 34.74
5.75 +: 0.25
0.75 : 0.12
0.4
SAC#
26.52= 1,58
342.0:1 6.25
0.42. 0.32
0,9::00
0.2
SAC"
2.85 +: 0.22
353,19 +: 13,62
0.80 0.14
0.95 4 0.03
PPO#/OH
234.81 4: 23:45
3.37 2.01
0.75 4: 0.03
PPO
50
100
150
200
30.58 0.18
352,99-: 6.59
0.32+ 0.12
0.924 0.02
PPO Training Epoch
PPO
2.55 + 0.35
389.67 + 11:20
0.05 0.08
0.95 = 0.02
Die/Prinitive Side Distancc))
Rele-PPOR*
4.65 0.3
352.98 7.35
0.35 1: 0.1
1: 0.07
0.8
Tra ning Out Road Rate
Deep Deep-PPO "Pp
4.88 + 1.17
334,09 +: 291
0.48 +: 0.002
0.9 + 0.02
Table 1: Improvements on the performance and the safety on test enviromments with human-Al Al shared control. H indicates "human", H and H" indicates two takeover methods. H requires human to provide steering and throttle the whole takeover period, while H' only asks to activate motor primitives the start of takeover. We also do ablation study on the activation function and number of hidden layers of MLP. These results show our method is general and robust to the policy architecture and RL methods.
0.2
100
200
PPO Training Epoch
Figure 5: Learning Dynamics
mild (averagely cars per map) and all traffic vehicles and the target vehicle can drive smoothly towards their destinations. No obstacles are in these environments. In testing time, the trained agents will be evaluated in the other 20 unseen maps with higher traffic density (averagely 6 cars per map). Besides, obstacles like trafiic cones and breakdown vehicles will scatter randomly on the road. In this case, the target vehicle needs to be maneuvered delicately. We compare the performance and safety of the trained policy in two scenarios: runs in the test set solely, and runs with the human-Al shared control system built from Policy Dissection.
The primal task for the legged robot is to move toward one direction as fast as possible. We design the test environment be a forest-like environment where tree obstacles are randomly scattered. Therefore, though the overall goal is still to move forward, the agent needs extra effort to sidestep obstacles in the test environment.
Evaluation Metrics. For driving task, we evaluate agents on all the 20 test environments, and report the ratio of episodes where the agent arrives the destination as the success rate. Also, Out of Road rate can be calculated if the termination is caused by driving out of road. For benchmarking the safety of agents, the collision to vehicles, obstacles, sidewalk and buildings raises a cost +1. In addition, human attention cost is a huge concern of human-Al shared control system. We quantify this using human involvement, which is measured by the number of environment steps when the human subject presses keys for activating units, or controls throttle, brake, and steering wheel. We average the sum of reward, cost and human involvement on 20 environments as episodic return, episodic cost and human involvement. The above evaluation process will be repeated 5 times for each agent, and we report the average value and std for all metrics.
For the locomotion task, we use moving distance to measure the performance besides reward, since it directly reflects the goal. Moving distance is the displacement of robot on specific direction, i.e. x-axis. The evaluation is also repeated for 5 times on the exclusive test environment.
Human Interface. We use keyboard interface in all human-Al shared control experiments. When a specific key is pressed for evoking a behavior, related motor primitives will be activated and output v. The behavior termination can be triggered by ways: 1. Press another key and change to another behavior 2. The length of activation period T = t1 to reaches a predefined length. Once a behavior is terminated, the related motor primitives' output will not be overwritten. In addition, there is one key mapping to the default behavior that no motor primitives are activated.
All experiments without human are repeated 5 times with different random seeds. N = 5 episodes are used for applying Policy Dissection. For experiments involving human subject, we repeat each experiment with 5 human subjects. For driving task, all human subjects have driving license. Human subject can pause the experiments if any discomfort happens. No human subjects were harmed the experiments. Human subjects have signed the experiment agreement and get compensation.
8

---

4.2 Policy Dissection for Understanding Learning Dynamics
In this section, we verify the correlation between motor primitives and relevant kinematic attributes. We first train PPO [61] agent on the MetaDrive's 's training environments and then conduct Policy Dissection on the trained policy and find two motor primitives in which Primitive 1 is associated with Speed and Primitive 2 associated with Side Distance. We then plot the evolution of the frequency discrepancy between the identified motor primitives and the kinematics alongside with the macroscopic measure of the learning progress in Fig. 5. Note that the frequency discrepancies d [di, ... dr] is normalized to [0, 1] by d d-min(d)
miild))
The upper plot in Fig. 5 shows Primitive gradually emerges to control speed, indicated by the decreasing discrepancy. With strong negative correlation, the success rate rapidly increases when the discrepancy of speed primitive also rapidly decreases between epoch 20-30. Similarly, in the lower plot in Fig. 5, the decreasing discrepancy between Primitive 2 and side distance implies that the internal unit of the agent is gradually specialized to control side distance. There exists a strong correlation between the discrepancy and the out-of-road rate, which can only be reduced by improving the ability to keep side distance. We thus conclude that the pivotal primitives emerge during training.
4.3 Human-AI Shared Control for Test-time Generalization
Using the motor primitives identified above, we can determine the activation value of motor primitive for steering associated kinematics according to Eq. 8. We apply the same process to dissect the policy trained from SAC [23] with the same network structure and build a stimulation-evoke shared control. We then invite human subject collaborate with the SAC and PPO agents on the test environments with different takeover methods. As shown in Tab. 1, the average human involvement, episodic cost, and success rate in the test environments are reported across policies with without human involvement. Human involvement greatly improves performance and safety when deploying the policy in unseen environments where delicate maneuvers such as side-passing and yielding are evoked by human, if necessary, to help finish the task.
map for
Different from the original PPO policy which is MLP with 2 hidden layers, 256 units per layer and "tanh" activation function, we also repeat the same experiment for Deep-PPO agent which has 6 hidden layers, and Relu- PPO which has "Relu" activation function. The results reported in 1 show our method is invariant to the network architectures and RL algorithms.
4.4 Human-AI Shared Control for Task Transfer
In addition to the improved test-time generalization, the timulation-evoker map enables RL agent to ac- complish new tasks with the help of human. Recent quadrupedal robots trained by RL exhibit promis- ing ability to walk on bumpy terrain via end-to-end control [36]. However, the agent only takes propri- oceptive states input without the observation of surroundings, and thus it can not avoid obstacles on its own. As shown in Fig. 4, train the agent only to walk on uneven terrain but test it in new environment full of obstacles. By applying our method, motor primitives related to yaw rate and speed control can be found and serve the interface for human steer the "insensible" quadrupedal robot to avoid obstacles. This forms the "/ttt" method. We also include "State+Vision" method [75], which trains a visuomotor policy for legged robot directly in the test environment, enabling it to sidestep obstacles. As shown in Table 2, with human involvement, Policy Dissection successfully transforms the "insensible" policy that is not fit to this task and greatly improves the performance, achieving comparable performance to the agent specially designed for this task.
Method
Reward
Moving Distance
State State
18.23 + 10.55
6,55 + 0.04
413.93 a: 93.33
16.94 4: 3.08
State+Vision
445.47 190.85
18.64 + 6.45
Table 2: Test on Obstacle Avoidance Task.
4.5 Qualitative Result: Parkour Demo
As shown in the Fig. 4, the training objective of the bipedal Cassie robot is to move forward. After training. Policy Dissection can discover several skills in the learned representation, as shown in Fig. 6. Furthermore, these skills can be triggered and controlled by human subject for solving the challenging parkour environment. See the demo video for detail.

---

Slop
Crouch
Backflip
Redirection
Tip Toe
Jump
Parkour
Figure 6: Policy Dissection can discover several skills such as crouching, jumping, backflipping in the learned representation of the bipedal Cassie robot, which is originally trained to move forward. These skills can be combined by human subjects and overcome challenging parkour environment.
4.6 Experiment on the Control Precision
We conduct an experiment evaluate the control precision enabled by our method. We train soal-conditioned policy for mal-C robot IsaacGym [57]. The trained robot taking raw rate as input can track target yaw with PID con- troller, and thus serves as the baseline. For a fair comparison, we directly dissect this policy and identify the yaw primitive. Likewise, PID controller that takes the yaw error as input is used to automatically determine the output yd for yaw primi- tive, enabling tracking target yaw. Note that for the primitive activation control the explicit heading command in input is set to 0. Heading tracking results in Fig. 7 show that the control achieved by primitive activation has comparable precision to the goal-conditioned control (see demo video for more detail).
Coal-conditon
Primiive Activttio
Heedin
Thme
Figure 7: Heading tracking error from the goal-condition control and the primitive activation control.
Conclusion and Discussion
Inspired by the neuroscience approach to study the motor cortex in primates, we present a simple yet effective method called Policy Dissection to align internal neural representations and kinematic attributes pretrained neural policy. The resulting the agent and evoke its certain behaviors. Policy Dissection enables human-Al shared control systems on top of the models trained from standard RL algorithms, without any re-training or modification. quantitatively evaluate the human-Al AI shared control system in autonomous driving and locomotion tasks, and the result suggests the substantial improvement of the test-time generalization for task transfer. We further provide interactive demonstrations in various environments to show the generality of our method and its application for human-Al collaboration.
ietiulattioo-vvookee map allows human steer
Limitations and societal impact. One limitation of Policy Dissection is that we only evaluate this method in continuous control tasks with the state vector as input. A wider range of tasks and neural network architecture such as discrete control tasks and CNNs can be adopted in future research. One potential negative social impact is that the safery issue remains be solved when deploying shared control systems in real-world applications. As shown in the autonomous driving experiment, our method derived from empirical study has no theoretical guarantee on the safety of human subjects. The unsuccessful collaboration conflict between human and AI may lead to potential risks.
10

---

References
[1] O. Amir, E. Kamar, A. Kolobov, and B. Grosz. Interactive teaching strategies for agent training. Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, 2016.
[2] Andrychowicz, F. Wolski, A. Ray, Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. arXiv preprint arXiv: 2017.
[3] A. Atrey, K. Clary, and D. Jensen. Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning. arXiv preprint arXiv:1912.05745, 2019.
[4] A. Banino, C. Barry, B. Uria, C. Blundell, T. Lillicrap. P. Mirowski, A. Pritzel, M. J. Chadwick, T. Degris. J. Modayil, et al. Vector-based navigation using grid-like representations in artificial agents. Nature, 557 (7705):429-) 433, 2018.
[5] D. Bau, J. Zhu. Strobelt, Zhou, B. Tenenbaum, W. Freeman, and A. Torralba. Gan dissection: Visualizing and understanding generative adversarial networks. arXiv preprint arXiv: 1,,597 2018.
[6] D. Bau, J.-Y. Zhu, H. Strobelt, A. Lapedriza, B. Zhou, and A. Torralba. Understanding the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences, 117(48)330071-000 -30078, 2020.
[7] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp. P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al. End end learning for self-driving cars, arXiv preprint arXiv: arXiv: 1604.0034330 16311 2016.
[8] Bragg and E. Brunskill. Fake it till you make it: Learning-compatille performance support. In Uncertainty in Artificial Intelligence, pages 915-924. PMLR, 2020.
[9] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.
[10] Z. Cao, E. Biyik, W. Wang, A. Raventos, A. Gaidon, G. Rosman, and D. Sadigh. Reinforcement learning based control of imitative policies for near-accident driving. In Robotics: Science and Systems, 2020.
[11] S. Carter and M. Nielsen. Using artificial intelligence to augment human intelligence. Distill, 2(12):e9, 2017.
[12] X. Chen, Z. Wang, Fan, B. Jin, Mardziel, C. Jon-Wong, and A. Datta. Reconstructing actions to explain deep reinforcement learning, arXiv preprint arXiv:2009.0850/ 2020.
[13] J. Cho, D.-.. Lee, and Y-G. Yoon. Inducing functions through reinforcement learning without task specification. arXiv preprint arXiv:2111.116f1 2021.
[14] K. Cobbe, O. Klimov, C. Hesse, T. Kim, and J. Schulman. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning, pages 1282-1289. PMLR, 2019.
[15] F. Codevilla, M. Müller, A. López. V. Koltun, and A. Dosovitskiy. End-to-end driving via conditional imitation learning. In 2018 IEEE International Conference on Roboticy and Automation (ICRA), pages 4693-4700. IEEE, 2018.
[16] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. PyBullet, 2016.
[17] C. J. Cueva and X.-X. Wei. Emergence of grid-like representations by training recurrent neural networks to perform spatial localization. arXiv preprint arXiv:1803.00777, 2018
[18] A. K. Dhawale, S. B. Wolff, R. Ko, and B. P. Ölveczky. The basal ganglia control Icarned motor sequenees independently of motor cortex. BioRxiv, page 827261, 2019.
[19] L. Fridman. centere autonomous vehicle systems: Principles of effective shared autonomy. arXiv preprint arXiv:1810.01833 2018.
[20] M. Graziano. The intelligent movement machine: An ethological perspective on the primate motor system. Oxford University Press, 2008
[21] M. S. Graziano, C. S. Taylor, and T. Moore. Complex movements evoked by microstimulation of precentral cortex. Neuron, 34(5)8411885 851, 2002.

---

[22] S. Guo, R. Zhang, B. Liu, Y. Zhu, D. Ballard. M. Hayhoe, and P. Stone. Machine versus human attention in deep reinforcement learning tasks, Advances in Neural Information Processing Systems, 34, 2021.
[23] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with stochastic actor In International Conference Machine Learning, pages 1861-1870, 2018
[24] R. H. Hahnloser, A. A. Kozhevnikov, and M. S. Fee. An ultra-sparse code underliesthe generation of neural sequences a songbird. Nature, 419(6902))65-70, 2002.
[25] J. Hawke, R. Shen, C. Gurau, S. Sharma, D. Reda, N. Nikolov, P. Mazur, S. Micklethwaite, N. Griffiths. A. Shah, al. Urban driving with conditional imitation learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA). pages 251 251-257. JEEE, 2020.
[26] A. A. Ismail, M. Gunady, H. C. Bravo, and Feizi Benchmarking deep learning interpretability in time series predictions. arXiv preprint arXiv:2010.13924 2020.
[27] Jeon, D. Losey, and D. Sadigh. Shared autonomy with learned latent actions. arXiv preprint arXiv:2005.032/0 2020.
[28] M. Kelly, C. Sidrane, K. Driggs-Caman paggs-Campbell, and M. Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. In 2019 International Conference on Robolics and Automation (ICRA), pages 8077-8083 IEEE, 2019.
[29] A. Kendall, J. Hawke, D. Janz, P. Mazur, D. Reda, J.-M. Allen, V.-D Lam, A. Bewley, and A. Shah. Learning to drive in day. In 2019 International Conference Robotics and Automation (ICRA), pages 8248 8254. IEEE, 2019.
[30] J. Kim and J. Canny. Interpretable learning for self-driving cars by visualizing causal attention. In Proceedings of the IEEE international conference on computer vision, pages 2942-2950, 2017.
[31] P. Knott, M. Carroll, S. Devlin, K. Ciosek, K. Hofmann, A. D. Dragan, and R. Shah. Evaluating the robustness of collaborative agents. arXiv preprint arXiv:2101.0550/ 2021.
[32] J. Konezak. On the notion of motor primitives in humans and robots. Proceedings of the Fifth International Workshop on Epigenetic Robotics, 2005.
[33] S. Korblithh M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519-3529. PMLR, 2019.
[34] N. Kriegeskorte and J. Diedrichsen. Peeling the onion of brain representations. Annual review of neuooscience, 42:407-432, 2019.
[35] N. Kricgeskorte, M. Mur, and P. Bandettini. Representational similarity analysis connecting the branches of systems neuroscience. Frontiers Systems Neuroscience, 2, 2008. ISSN 1662-5137 doi: 10.3389/ ceurr.06.004.2000
[36] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged rohots. arXiv preprint arXiv:2107.044 2021.
[37] Lee, Hwangbo, L. Wellhausen, Koltun, and M. Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47)cabb55986 2020.
[38] A. Levy, Platt, and K. Sacnko. Hierarchical reinforeement learning with hindsight. arXiv preprint 1805::80050888 2018
[39] Q. Li, Z. Peng. Z. Xue, Q. Zhang, and B. Zhou. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. arXiv preprint arXiv:2109. 2021.
[40] X.-H. Li, Cao, Y. Shi, Bai, Gao, L. Qiu, C. Wang. Y. Gao, S. Zhang, X. Xue, A survey of data-driven and knowledge-awee aware explainable ai. JEEE Transactions on Knowledge and Data Engineering, 2020.
[41] A. Loquercio, E. Kaufmann, R. Ranftl, M. Müller, V. Koltun, and D. Scaramuzza. Learning high-speed flight in the wild. Science Robotics, 6(59).cabg5810 2021.
[42] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gpu- based physics simulation for robot learning. arXiv preprint 210: 10470, 2021.
12

---

[43] A. Mandlekar, D. Xu, R. Martún-Martin, Y. Zhu, L. Fei-Fei, and S. Savarese. Human-in-the--lopp imitation learning using remote teleoperation. arXiv preprint arXiv:2012.06733, 2020.
[44] K. Mason and Grijalva, A review of reinforcement learning for autonomous building energy management. Computers Electrical Engineering, 78:300-312, 2019.
[45] J. Merel, L. Hasenclever, A. Calashov, A. Ahuja, V. Pham, G. Wayne, Y. W. Teh, and N. Heess. Neural probabilistic motor primitives for humanoid control. arXiv preprint arXiv:1811.11711 2018.
[46] J. Merel, D. Aldarondo, J. Marshall, Y. Tassa, G. Wayne, and B. Ölveezky. Deep neurocthology of virtual rodent. arXiv preprint arXiv: I. .09451, 2019.
[47] O. Nachum, Gu, Lee, and Levine. Data-efficient hierarchical reinforcement learning. arXiv preprint 1rXiv:8805.0820000 2018.
[48] H. Nili, C. Wingfield, A. Walther, L. Su, W. Marslen-Wilson, and N. Kriegeskorte. A toolbox for representational similarity analysis. PLoS computational biology, 10(4)e1003353, 2014.
[49] D. Omeiza, H. Webb, M. Jirotka, and L. Kunze. Explanations in autonomous driving: survey, arXiv preprint a/Xiv:2103.05/4. 2021.
[50] X. B. Peng. P. Abbeel, Levine, and M. Van de Panne. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions On Gruphics (TOG), 37(4):1-14, 2018.
[51] Z. Peng, Q Li, C. Liu, and B. Zhou. Safe driving via expert guided policy optimization. In Sth Annual Conference on Robot Learning, 2021.
[52] J. Peters and S. Schaal. Learning motor primitives with reinforcement learning. 11th Joint Symposium on Neural Computation, University of Southern California. 2004.
[53] N. Puri, S. Verma, P. Gupta, D. Kayastha, S. Deshmukh, B. Krishnamurthy, and S. Singh. Explain your move: Understanding agent actions using specific and relevant feature attribution. arXiv preprint arXiv: arXiv:1912.122 1211912111,1 2019.
[54] S. Reddy, A. D. Dragan, and S. Levine. Shared autonomy via deep reinforcement learning. arXiv preprint arXiv: 1802.01744, 2018
[55] S. Reddy, A. Dragan, S. Levine, Legg, and J. Leike. Learning human objectives by evaluating hypothetical behavior. In International Conference on Machine Learning, pages 8020-8029. PMLR, 2020.
[56] F. Richter, R. K. Orosco, and M. C. Yip. Open-sourced reinforeement learning environments for surgical robotics. arXiv preprint arXiv: arXiv: 1903,02090 2019.
[57] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. Conference on Robot Learning, pages 91-100. PMLR, 2022.
[58] W. Saunders, G. Sastry, A. Stuhlmueller, and Evans. Trial without error: Towards safe reinforcement learning via human intervention. arXiv preprint arXivv: 2017.
[59] C. Schaff and M. R. Walter. Residual policy learning for shared autonomy, arXiv preprint 2000000000 2020.
[60] T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In International conference on machine learning, pages 1312-1320. PMLR, 2015.
[61] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint (707.v:7007.0344 2017.
[62] R. R. Selvaraju, A. Das. R. Vedantam, M. Cogswell, D. Parikh, and D. Batra. Grad-cam: Why did you say that? visual explanations from deep networks via gradient- based localization. CoRR, abs/1610.02391. 2016. URL http:// http///axvvv../. www.a.www/aa..a.........e.owoeee abs/ 023991
[63] Y. Shen and B. Zhou. Closed-form factorization latent semantics in gans. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1532-1540, 2021.
[64] J. Spencer, S. Choudhury, M. Barnes, M. Schmittle, M. Chiang. P. Ramadge, and S. Srinivasa. Learning from interventions. In Robotics: Science and Systems (RSS), 2020.
13

---

[65] F. P. Such, V. Madhavan, R. Liu, R. Wang, P. S. Castro, Y. Li, J. Zhi. L. Schubert, M. G. Bellemare.
J. Clune, et al. An atari model for aualyzing, visualizing, and comparing deep reinforcement learning agents, arXiv preprint arrXiv1882..01 2018.
[66] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[67] P. S. Thomas and A. G. Barto. Motor primitive discovery. In 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL), pages 1-8. IEEE, 2012.
[68] F. Todorov, T. Frez, and Y. Tassa. Mujoce: A physics engine for model-based control. In 2012 JEEF/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
[69] E. Ugur, E. Şahin, and F. Oztop. Self-discovery of motor primitives and learning grasp affordances. In 2012 IEEE/RSJ International Conference Intelligent Roboty and Systems, pages 3260-3267. IEEE, 2012.
[70] G. Warnell, N. Waytowich, V. Lawhern, and P. Stone. Deep tamer: Interactive agent shaping in high- dimensional state spaces. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[71] J. Wu, 7. Huang, C. Huang, 7. Hu, Hang. Y. Xing, and C. Lv. Human-in-the-loog deep reinforcement learning with application to autonomous driving, arXiv preprint arXiv:2104.072444 2021.
[72] Z. Xie, G. Berseth, P. Clary, Hurst, and M. de Panne. Feedback control for cassie with deep reinforcement learning. In 2018 IEEE/RS/ International Conferencee on Intelligent Robots and Systems (IROS), pages 1241-1246. IEEE, 2018.
[73] Y. Xu, Y. Shen, Zhu, Yang, and Zhou. Generative hierarchical features from synthesizing images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4432-4442, 2021.
[74] C. Yang, Y. Shen, and B. Zhou. Semantic hierarchy emerges in deep generative representations for scene synthesis. International Journal of Computer Vision, 129(5):1451-14)) 2021.
[75] R. Yang, M. Zhang, N. Hansen, H. Xu, and X. Wang. Learning vision-guided quadrupedal locomotion end-to-end with cross-modal transformers. arXiv preprint arXiv:2107.03399, 2021.
[76] N. Yoshida, T. Daikoku, Y. Nagai, and Y. Kuniyoshi. Embodiment perspective of reward definition for behavioural homeostasis. In Deep RL Workshop NeurlPS 2021, 2021.
[77] É. Zablocki, H. Ben- Youne,, P. Pérez, and M. Cord. Explainability of vision-hased hased autonomous driving systems: Review and challenges. arXiv preprint arXiv:2 101.05307, 2021.
[78] Q.-s. Zhang and S.-c. Zhu. Visual interprctability for deep learning: a survey, Frontiers of Information Technology & Electromic Engineering, (92279733,, 2018.
[79] R. Zhang, F. Torabi, I.. Guan, D. H. Ballard, and P. Stone. Leveraging human guidance for deep reinforcement learning tasks. arXiv preprint arXiv:
2019.
[80] R. Zhang, F. Torabi, G. Warnell, and P. Stone. Recent advances in leveraging human guidance for sequential decision-making tasks. Autonomous Agents and Multi-Agent Systems, 35(2):1-39, 2021.
[81] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2921-2929, 2016.
[82] B. Zhou, D. Bau, A. Oliva, and A. Toralla, Interpreting deep visual representations via network dissection. IEFE transactions on pattern analysis and machine intelligence, 41(9):213--2145 2018.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section 5, Limitations
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See supplemen- tal material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Hyperparameters are in appendix, and experimental settings are in Section 4
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 4. Experimental Setting
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you eite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [Yes]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [Yes] See Section Experimental Setting
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [Yes] See Section 4. Experimental Setting
15


Reviews:
[{"review_id": "oM7BRxY53v8", "reviewer": "Reviewer_g9wy", "summary": "The paper proposes a method to dissect a policy trained for a simple task to extract \"motor primitives\" (or set of neurons) that provides different behaviors or attributes. The authors then use these attributes to have humans control the agent for more complex tasks or to create different behaviors by combination. The paper showcases the algorithm in 2 domains: Driving and quadruped locomotion (in simulation). The authors first train for driving (or walking) forward, without any explicit goal conditioning. Then, the authors dissect the policy for different behaviors by replaying trained policy and doing frequency matching. Next, humans use the provided interface for obstacle avoidance. In addition, the authors show that they can obtain very different behaviors by combining these motor primitives in different domains such as 2D walker. As an example, they obtain front flipping behavior by activating jumping and roll rate which are discovered using policy dissection.", "questions": "Please see the weaknesses mentioned above. Could you address if it is possible to implement the suggested directions above? Otherwise, why not?", "limitations": "The limitations of the method is explained well, but does not contain the weaknesses (or potential directions) that I explained above. I don't think that the paper contains any major societal impact.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The method proposed by the authors is very interesting for interpretability and discovery of skills in Reinforcement Learning. As the authors state, instead of training for complex behaviors, the authors can train for simple ones and discover different skills from replay. I think the front flipping is a very good example. It is very hard to design reward functions for all these different skills and train a network that can perform all of them, but the authors can find these skills and combine them without reward engineering which is very interesting, especially from locomotion perspective where the training are usually goal conditioned not skill-rewarded.\n\nAnother interesting side-effect of the approach is from robotics perspective. Although the authors are working on simulation, not real robots, the proposed method allows discovery of skills with small amount of replay data. This can be a very interesting approach to the well known sim-to-real problem, where trained policies are not behave the same way in the real world. The proposed method can be used to discover skills on the hardware although it does not match the simulated behavior, this would lead to motor skills that work on the real robot.\n\nIn my opionion, the apprach has some weaknesses as well.\n1- The authors use the approach to have humans control the robot to solve slightly more complex task. On the other hand, the obstacle avoidance problem can already be solved with goal conditioned policies. So the results does not fully exploit the method to its full capacity. Instead, the authors could try to achieve a problem that can't be solved with simple training. One example I can think of is to have a parkour setup that requires jumping, walking low. \n\n2-The authors could automate the higher level controls instead of collaborating with humans. The proposed schema is perfect for a hierarchical reinforcement learning approach. Once skills are discovered, the authors could retrain with the higher level controls provided to the humans with additional sensory information that did not exist in the initial training. This could very well train in very short time and handle the complex scenarios.\n\n3-The authors could autmoate the discovery of the skills. The literature contains many examples of such approaches based on maximizing entropy during training. Instead of using human designed motor primitives, learning could provide more diverse behaviors as well.\n\n4-The discovery is based on replay of the very simple behavior. If I understand correctly, the matching is based on behaviors seen in very small timeframes (i.e. it turns slightly left and then slightly right to go straight). But, if the initially trained policy was great, the behavior could very well just go straight which would lead to failure to match (again, if I understand the matching process correctly). Adding disturbances (at the neuron level) to the replay would provide much more than the initial data set where the agent is just going straight. Of course this might require some more work and careful design of the disturbing process.\n\nDespite these weaknesses, I think the method has a very good potential and very interesting to read and the provided experiments is considerably enough to show some of this potential as well. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RZioKYKuSi_", "reviewer": "Reviewer_m4BT", "summary": "In this paper, the authors propose a novel method to modify the behavior of pretrained neural network policies to obtain the desired control effects without changing the weights of the networks unchanged. Instead, the authors analyze the activation patterns of all neuron units in the network, and create associations between the neuron activation and desired kinematic properties (such as angular velocity, translational velocity, etc) in the frequency domain through Fourier transforms. For each kinematic attribute, the proposed method identifies one evoking neuron; So during the online policy roll out the users can directly stimulate a certain desired behavior of the robot by activating the corresponding neuron, even though the policy is never trained for this new behavior/task. The authors demonstrate in a set of experiments that, with their method and human-in-the-loop, they can improve the performance of many safety environments. ", "questions": "Other questions and suggestions:\n\nThe Figure 6 is a bit confusing. It does not make sense to have discrepancy distance and training success rate curves share the same y-axis.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of the paper:\n\nIt introduces a new concept to do zero-shot generalization of existing policies without any re-training.\n\nProvides a new insight on the interpretability of neural network policies, i.e. from the frequency domain to associate neuron activations with behavior. \n\nDemonstrates that, even though a policy may not be trained with certain tasks, some behaviors might have already been seen during the training phases and somehow are “memorized” through the network structure. \n\nThe weaknesses of the paper:\n\nThe motivation of the paper is to make neural controllers safer by incorporating humans in the loop. Thus, I assume that control accuracy and robustness of the NN policy is very important. However, by “hacking” into a NN policy that is not trained with human inputs (e.g. steering, throttle cars), I am not convinced if the behavior, such as velocity tracking accuracy, etc,  can be as good as goal/human input conditioned policies. \n\nThe authors assume that for each kinematic attribute, there is one neuron that is responsible for. This assumption is not quite sound to me, as the activation of a neuron will have chain effects for other neurons down the propagation path. And the other neurons might be responsible for other kinematic behaviors according to the same assumption. \n\nAlso, I am not sure how this method will scale with truly deep neural networks with many hidden layers and neurons, as there are many more activation paths for the network.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EjxcQ5CcHXr", "reviewer": "Reviewer_XzPE", "summary": "  This paper proposes a framework where a human can intervene to steer the behaviour of an RL agent. This is done by allowing the human to select from a \"stimulation-evoked map\". ", "questions": "* How are the motor primitives to be stimulated found/labelled? For example, how is the spinning primitive added? Are these predefined or learnt?", "limitations": "Agent isn't able to learn from interventions", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* I think the paper has a big problem with clarity, the setting is very unclear.\n\n* Experiments don't compare against a baseline\n\n* Requires full attention from the human and system isn't able to learn from the human interventions\n\n\n\nApologies for not discussing this more but this is not my area and I found the paper really unclear, so I struggle to write a proper review.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1NtWWTUOtRd", "reviewer": "Reviewer_cucK", "summary": "This paper introduces a method termed 'policy dissection', which finds a mapping between neurons in a trained continuous control policy and high-level kinematic motions of interest (eg. turn left, right, stop, go forward). This mapping is then used to allow for human intervention of modification of the trained policy. The core contributions are a method for identifying this mapping (using a frequency-based analysis technique), and to show that this can be exploited to allow policy intervention (by amplifying units in a direction that correlates with an increase in the desired kinematic motion) and high level policy modification by humans.", "questions": "I have a fair idea of how the mapping between neurons and kinematic primitives of interest is identified, but I am less clear about how the reverse is implemented, and how a user keypress is turned into a unit amplification. Please could you explain this in more detail? Eg. How much do you amplify, is this temporary and lasting only a few seconds, or a permanent change.\n\nI would value hearing whether the perceived coarseness of the intervention control is due to the relatively naive mapping (only really rough correlations) or whether this could be improved through a better human intervention control policy?\n\nMinor comments:\n\nThis work reminded me of control loop pairing work (relative gain array) where the best input output pairings are selected for multivariate control systems. I would value hearing your perspective on future work and what is needed to get finer control over motion primitives from the network? Does this need better mappings, or better eg. hierarchical policies?\n\nCan you be more specific about the metadrive tasks used to populate Table 1, and the different takeover methods.", "limitations": "Limitations are discussed (neurons that do not align with any primitives), but I would value more discussion on the coarseness of the method - the video does make human take-over appear quite crude. I would also value a discussion of competing approaches, eg. policies that are explicitly trained to allow for kinematic intervention, and motivation for the proposed approach over this.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- This is a creative and interesting idea, which I have not come across before. I enjoyed reading this paper.\n- I think this work is significant from the perspective of trying to understand the implicit representations or primitives captured by neural control policies, and because it introduces a method for humans to controllably modify a learned policy to achieve a goal.\n- A thorough set of experiments are provided on a number of experimental settings.\n\nWeaknesses:\n\n- The paper could do with some smoothing, and improvements to the writing. There are numerous typos and grammatical errors, but these did not get in the way of the idea being communicated. \n- Although the proposed approach does allow for some level of human intervention and modification, this appears to be very imprecise and it is unlikely that the proposed approach allows more than very coarse control.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept