Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a method to dissect a policy trained for a simple task to extract "motor primitives" (or set of neurons) that provides different behaviors or attributes. The authors then use these attributes to have humans control the agent for more complex tasks or to create different behaviors by combination. The paper showcases the algorithm in 2 domains: Driving and quadruped locomotion (in simulation). The authors first train for driving (or walking) forward, without any explicit goal conditioning. Then, the authors dissect the policy for different behaviors by replaying trained policy and doing frequency matching. Next, humans use the provided interface for obstacle avoidance. In addition, the authors show that they can obtain very different behaviors by combining these motor primitives in different domains such as 2D walker. As an example, they obtain front flipping behavior by activating jumping and roll rate which are discovered using policy dissection.
strengths_and_weaknesses: The method proposed by the authors is very interesting for interpretability and discovery of skills in Reinforcement Learning. As the authors state, instead of training for complex behaviors, the authors can train for simple ones and discover different skills from replay. I think the front flipping is a very good example. It is very hard to design reward functions for all these different skills and train a network that can perform all of them, but the authors can find these skills and combine them without reward engineering which is very interesting, especially from locomotion perspective where the training are usually goal conditioned not skill-rewarded.

Another interesting side-effect of the approach is from robotics perspective. Although the authors are working on simulation, not real robots, the proposed method allows discovery of skills with small amount of replay data. This can be a very interesting approach to the well known sim-to-real problem, where trained policies are not behave the same way in the real world. The proposed method can be used to discover skills on the hardware although it does not match the simulated behavior, this would lead to motor skills that work on the real robot.

In my opionion, the apprach has some weaknesses as well.
1- The authors use the approach to have humans control the robot to solve slightly more complex task. On the other hand, the obstacle avoidance problem can already be solved with goal conditioned policies. So the results does not fully exploit the method to its full capacity. Instead, the authors could try to achieve a problem that can't be solved with simple training. One example I can think of is to have a parkour setup that requires jumping, walking low. 

2-The authors could automate the higher level controls instead of collaborating with humans. The proposed schema is perfect for a hierarchical reinforcement learning approach. Once skills are discovered, the authors could retrain with the higher level controls provided to the humans with additional sensory information that did not exist in the initial training. This could very well train in very short time and handle the complex scenarios.

3-The authors could autmoate the discovery of the skills. The literature contains many examples of such approaches based on maximizing entropy during training. Instead of using human designed motor primitives, learning could provide more diverse behaviors as well.

4-The discovery is based on replay of the very simple behavior. If I understand correctly, the matching is based on behaviors seen in very small timeframes (i.e. it turns slightly left and then slightly right to go straight). But, if the initially trained policy was great, the behavior could very well just go straight which would lead to failure to match (again, if I understand the matching process correctly). Adding disturbances (at the neuron level) to the replay would provide much more than the initial data set where the agent is just going straight. Of course this might require some more work and careful design of the disturbing process.

Despite these weaknesses, I think the method has a very good potential and very interesting to read and the provided experiments is considerably enough to show some of this potential as well. 
questions: Please see the weaknesses mentioned above. Could you address if it is possible to implement the suggested directions above? Otherwise, why not?
limitations: The limitations of the method is explained well, but does not contain the weaknesses (or potential directions) that I explained above. I don't think that the paper contains any major societal impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: In this paper, the authors propose a novel method to modify the behavior of pretrained neural network policies to obtain the desired control effects without changing the weights of the networks unchanged. Instead, the authors analyze the activation patterns of all neuron units in the network, and create associations between the neuron activation and desired kinematic properties (such as angular velocity, translational velocity, etc) in the frequency domain through Fourier transforms. For each kinematic attribute, the proposed method identifies one evoking neuron; So during the online policy roll out the users can directly stimulate a certain desired behavior of the robot by activating the corresponding neuron, even though the policy is never trained for this new behavior/task. The authors demonstrate in a set of experiments that, with their method and human-in-the-loop, they can improve the performance of many safety environments. 
strengths_and_weaknesses: The strengths of the paper:

It introduces a new concept to do zero-shot generalization of existing policies without any re-training.

Provides a new insight on the interpretability of neural network policies, i.e. from the frequency domain to associate neuron activations with behavior. 

Demonstrates that, even though a policy may not be trained with certain tasks, some behaviors might have already been seen during the training phases and somehow are “memorized” through the network structure. 

The weaknesses of the paper:

The motivation of the paper is to make neural controllers safer by incorporating humans in the loop. Thus, I assume that control accuracy and robustness of the NN policy is very important. However, by “hacking” into a NN policy that is not trained with human inputs (e.g. steering, throttle cars), I am not convinced if the behavior, such as velocity tracking accuracy, etc,  can be as good as goal/human input conditioned policies. 

The authors assume that for each kinematic attribute, there is one neuron that is responsible for. This assumption is not quite sound to me, as the activation of a neuron will have chain effects for other neurons down the propagation path. And the other neurons might be responsible for other kinematic behaviors according to the same assumption. 

Also, I am not sure how this method will scale with truly deep neural networks with many hidden layers and neurons, as there are many more activation paths for the network.

questions: Other questions and suggestions:

The Figure 6 is a bit confusing. It does not make sense to have discrepancy distance and training success rate curves share the same y-axis.
limitations: N/A
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary:   This paper proposes a framework where a human can intervene to steer the behaviour of an RL agent. This is done by allowing the human to select from a "stimulation-evoked map". 
strengths_and_weaknesses: * I think the paper has a big problem with clarity, the setting is very unclear.

* Experiments don't compare against a baseline

* Requires full attention from the human and system isn't able to learn from the human interventions



Apologies for not discussing this more but this is not my area and I found the paper really unclear, so I struggle to write a proper review.
questions: * How are the motor primitives to be stimulated found/labelled? For example, how is the spinning primitive added? Are these predefined or learnt?
limitations: Agent isn't able to learn from interventions
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces a method termed 'policy dissection', which finds a mapping between neurons in a trained continuous control policy and high-level kinematic motions of interest (eg. turn left, right, stop, go forward). This mapping is then used to allow for human intervention of modification of the trained policy. The core contributions are a method for identifying this mapping (using a frequency-based analysis technique), and to show that this can be exploited to allow policy intervention (by amplifying units in a direction that correlates with an increase in the desired kinematic motion) and high level policy modification by humans.
strengths_and_weaknesses: Strengths:

- This is a creative and interesting idea, which I have not come across before. I enjoyed reading this paper.
- I think this work is significant from the perspective of trying to understand the implicit representations or primitives captured by neural control policies, and because it introduces a method for humans to controllably modify a learned policy to achieve a goal.
- A thorough set of experiments are provided on a number of experimental settings.

Weaknesses:

- The paper could do with some smoothing, and improvements to the writing. There are numerous typos and grammatical errors, but these did not get in the way of the idea being communicated. 
- Although the proposed approach does allow for some level of human intervention and modification, this appears to be very imprecise and it is unlikely that the proposed approach allows more than very coarse control.


questions: I have a fair idea of how the mapping between neurons and kinematic primitives of interest is identified, but I am less clear about how the reverse is implemented, and how a user keypress is turned into a unit amplification. Please could you explain this in more detail? Eg. How much do you amplify, is this temporary and lasting only a few seconds, or a permanent change.

I would value hearing whether the perceived coarseness of the intervention control is due to the relatively naive mapping (only really rough correlations) or whether this could be improved through a better human intervention control policy?

Minor comments:

This work reminded me of control loop pairing work (relative gain array) where the best input output pairings are selected for multivariate control systems. I would value hearing your perspective on future work and what is needed to get finer control over motion primitives from the network? Does this need better mappings, or better eg. hierarchical policies?

Can you be more specific about the metadrive tasks used to populate Table 1, and the different takeover methods.
limitations: Limitations are discussed (neurons that do not align with any primitives), but I would value more discussion on the coarseness of the method - the video does make human take-over appear quite crude. I would also value a discussion of competing approaches, eg. policies that are explicitly trained to allow for kinematic intervention, and motivation for the proposed approach over this.
review_title: null

Meta-review: metareview: This paper proposes to dissect a trained policy, which finds the correspondence between the neuron activations and the motion primitives. This enables a human to control the agent to complete complex and diverse tasks even though only one simple task is trained. All reviewers agree that the proposed method is a creative solution to an important problem. And it may also have important applications in robotics. In addition, this is an important step towards understanding/interpreting neural network policies, which may inspire follow-up works in the future. 

There are a few concerns and suggestions in the original reviews. Most of them are sufficiently addressed in the rebuttal and discussions. The new experimental results look impressive, and help to resolve a major concern about the coarseness of the human control that this paper enables. While the paper writing is still rough, the creative solution and the potential important applications compensate for the shortcomings. Thus, I recommend accepting this paper. Please revise the paper by incorporating reviewers' comments.
confidence: Certain
Decision: Accept