Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work investigates the interpretability of multi-agent RL and introduces a method for incorporating interpretable concepts from domain expert. 

The concepts can be considered as hidden states of current environment.

The proposed Concept Policy Model decomposes the policy network into several parts: 

1. A feature extractor, 
2. A residual layer that produces "residual vector" that has no information related to concept, 
3. A concept layer that produces "concept vector",
4. Policy layer produces action distribution.

For the concept layer, supervised learning is conducted to fit the concept vector to ground-truth concepts provided by environment.

For the residual vector, the ZCA whitening is used to manipulate the activation matrix and let the output does not contain information about the concepts.

The concept vector output by the concept layer serves as part of the input to the final policy layer. This means that (1) we can query and manipulate the concept during execution.
strengths_and_weaknesses: 
### Strengths

* The interpretability of MARL agents is a valuable research topics. I am happy to see work pursing this.
* The idea that "human can intervene when the agent understands the environment in a wrong way" is inspiring. This reminds me works on intervention-based imitation learning and model-based learning. The concept layers in this work is just like the dynamic models (both are trained with environment-provided ground truth information).


### Weaknesses

* The clarity is poor. As an example, in Sec 4.2, ZCA whitening is conducted on the activation values of residual values. The motivation here is to let "residual layer not to encode information related to concepts" (line 152). What is the connection between "let it not encode something" and "use ZCA whitening"? Also, what is the $\mu_x$ in Equation 4?
* The motivation and whole story are disconnected from the method and experiment that really implemented. See "Summary Of Recommendation".
* The experiment is oversimplified (See "Issues, Method, Q3") and the "concept" is a concept that poorly defined.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: 
Generally, I think the method proposed in this work is far less novel and impacting than it is claimed in this work.


I admit that the problem trying to be addressed is important, the interpretability of MARL algorithms. 
I would expect to see the authors take the unique feature of the problem, multi-agent, into the consideration and propose a method that can infer the causality between agents' behaviors. 


However, this work proposes to decompose the policy network and use extra supervision signal to force the concept layer to predict ground truth concept. Conditioning the policy network with "high-level concept" is not a novel idea and is widely used before, such as [1]. 
It turn out to be a method that adding supervision loss to fit human-labelled concepts and there is no any special design for MARL setting.

[1] End-to-end driving via conditional imitation learning

The mismatch between what it claims and what it does is the major reason for my recommendation.

1. In conclusion, the authors claim "Concept Policy Models incorporate domain knowledge from an expert" (Line 310). Why don't you just use those concept as input directly to the policy network? (See Issues - Method - Q3) Also, the proposed method is not "a general framework" (Line 311) since it requires hand-coded "concepts" from the environment and the "domain of concepts" is designed by human (I can accept this, it is inevitable that we need some prior knowledge. However, this makes the proposed method unqualified to be a "general framework").
2. In abstract, the authors claim "This allows a human operator to both obtain decision making rationale and intervene" (Line 8). However, I can't find the description on how human expert intervene and improve the wining rate. I am doubting whether a human expert can really figure out the logic of agents' behavior by watching per-step predicted concepts (which, basically are a set of informative values of current environment, like locations of nearby enemies).
3. As I discussed above, I can't find any particular design to make the proposed method more suitable in MARL setting.

In conclusion, I think current submission is below the bar of acceptance.
issues: 
### Method

1. For the residual vector, the ZCA whitening is used to manipulate the activation matrix and let the output of residual vector does not contain information about the concepts. I am not fully understand what is happening during this process. Could you explain more here?


2. Line 231: "The iterative normalization layer is applied over the concatenated concept and residual layers." IIUC, the concept layer should not be "whitened"? 


3. Is it possible that the concepts you selected as described in "Concepts" (Line 212) are very strong and informative observation? That is to say, consider a extreme case where your concept layer is perfectly trained, then it should always output ground-truth concepts as described in Line 212. Those information is strong enough, along side with the information came from residual layer that is extracted from the observation, to learn a powerful agent. To justify this, I will expect a new experiment that use directly the ground-truth concepts as input (replacing the output of concept layer) to the policy layer.

4. How you overwrite the activation of concept layer? (Sec 4.3)


5. Other issues on method is in "Summary Of Recommendation".






### Writing

1. Line 59, 112, 207: Wrong latex code for left quotation mark
2. Table 1 is too far from Result section.
3. I would expect to see a figure to illustrate the network architecture of Concept Policy Models in Sec 4.1
4. Line 310 - Line 314: A long and non-informative sentence. I don't think it provide any interesting conclusion to reader.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: To improve the interpretability of the RL policy model, This paper enforces the policy model to encode the "concept" in the intermediate layers of the model. The concepts are categorical or continuous values relevant to the task. For example, in competitive games, the opponent's strategy (offensive/defensive) and range (0~100) can be the concepts. The policy model consists of two major modules. The first model predicts the concepts based on observations. Domain experts can correct the predicted concepts. The second module transforms the concept's value into actions. The advantage is that human experts can intervene in the concept predictions and correct them. The experimental results show performance improvements.
strengths_and_weaknesses: Pros:
- The idea of using concept learning on sim-to-real and interpretability is interesting.

Cons:
- Section 6 is very hard to parse -- especially real-world results. I suggest making those results in the real world as figures or tables instead of describing them in a dense paragraph.
- Lack of experiments. The experiments are limited -- only in 1 task. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend weak acceptance since the author successfully validates the claim described in the paper and there is no obvious technical flaws.
issues:
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper adds additional supervised signals to support the policy training (PPO) in a multi-agent RL setting. In detail, they introduce classifiers and regressors to predict concrete features (concepts) like relative positions or the strategy of enemies and train these modules in a supervised way. The output actions are conditioned on these predicted features. In this way, they state the interpretability of the policy is improved. Experimental results show that this method helps improve the defending team's performance in the tag game in both simulation and real robotics.
strengths_and_weaknesses: I think the research topic is important in reinforcement learning. The writing is in general easy to follow. The work has hardware experiments on real robots.

The main weakness from my perspective is, whether their method indeed improves the interpretability largely, or their method just shows that a better feature extractor helps the performance, which is well-known in deep learning. From Tab.2 we can see that the most important ‘concepts’ are relative position and orientation. Without these two concepts, the win rate drops dramatically from 83% to 23%. The other concepts only affect 3%. As the original observation already contains the absolute position and orientation, knowing the relative version doesn’t improve any ‘interpretability’ of the policy. Converting the absolute version to the relative one is more like input preprocessing. Besides, it is already well known that relative position and orientation are better than the absolute version in such a setting. 
I think I agree that predicting the ‘target’ can improve the interpretability of the policy. However, according to their experiments, this concept doesn’t affect the performance so much. I think they need a better experiment setting (like more interesting concepts) to solve this issue. Besides, I think they can design some experiments to show the improved interpretability of the policy directly. For example, they can show that when they intervene in the ‘target’ concept, the defender agent will change their tagging target in the environment.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I feel what they have done is more like using supervision signals to train a better feature extractor that helps performance instead of introducing something which clearly improves the interpretability of the policy. They should have some experiments to demonstrate the improved interpretability directly. In addition, in their experiments, the concepts that actually matter are relative position and orientation. It is better to have experiments with more interesting concepts.

**After Rebuttal**
I appreciate the authors' efforts on the new experiments in Appendix D that show the interpretability directly. I have increased my score to weak accept. I'll suggest authors include more experimental results that demonstrate the improved interpretability in the main paper.
issues: They use lots of sentences (e.g., the whole Sec.4.2) to describe the soft model. But the soft model works much worse than the hard one. In L250 they explain that this is because soft model uses fewer concepts. In this case, I think an experiment where soft model uses all concepts like the hard one is needed for a clear comparison with hard model. Otherwise, I’d consider it as an interesting but not so important variant, as the soft model’s performance is much worse, and the soft model is more complex than the hard one in both architecture and training way. In this case, the content of the soft model should be reduced in the main paper and maybe moved to the appendix.

Other small writing issues:

1. In Eq.2, I think $\pi^1_{\theta1}$ doesn’t output a distribution over the action. Therefore, it should be $x=\pi^1_{\theta1}(z_a)$ instead of $x=\pi^1_{\theta1}(u_a|z_a)$. It is actually better to use other notation to denote $\pi^1_{\theta1}$, as it is not a policy, but the first part of the policy net (more or less like a feature extractor).

2. In L261-263, I cannot see the win rates of both the concept model and baseline model from Fig.3. This sentence should be rewritten.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposed to adopt the concept bottleneck network to multi-agent RL. The network architecture induces interpretability and stabilizes training, according to experimental demonstration. 
strengths_and_weaknesses: Strengths: 

1. This paper is genreally well written and easy to understand. 

2. The motivation is strong. 

3. The experimental results are strong and convincing. 

Weaknesses: 

1. It could help to present a diagram of the network architecture described in Eq. 2. 

2. I would recommend adding parenthesis to the superscript in the policy, as $\pi^{(1)}_{\theta 1}$. Otherwise, the superscript is easy to be confused as power exponent. 

3. The network architecture does have limited novelty above the conept bottleneck network, but it is refreshing to see it being adopted to MARL. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend this paper for acceptance. In my opinion, this paper contains sufficient ideas and executions for publication. 
issues: This is more of a discussion point, but in Eq. 2 (with notations simplified), the fact that $\pi^2$ is applied on $r(x)$ could arbitrarily amplify its effect. Is it possible to change the equation to $\pi^2(c(x)) + r(x)$, so that the contribution of the residual network is much easily quantifiable?
review_title: null

Meta-review: metareview: # Strenghts
- Interpretability in (multi-agent) reinforcement learning is an important challenge
- Human intervention during the learning process is a promising concept
- The approach was evaluated in a real robot multi-agent scenario.

# Weaknesses
- There are some concerns regarding the novelty of the claimed contributions
- Despite evaluating on physical platforms, the evaluation of the method itself is fairly limited.
- There appears to be a disconnect between the motivation and the actual approach
- Related to the previous point the discussion/interpretation of the experimental results requires additional clarification and justification.

 # Post-Rebuttal Update
The authors addressed the reviewers' concerns very well.
The additional clarifications and relaxed statements/claims improve the paper substantially. 
I still ask the authors to make sure that the points further discussed during the rebuttal phase will be addressed in the camera-ready version. Furthermore, the paper will have to be shortened back to 8 pages.

confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)