Concept Learning for Interpretable Multi-Agent Reinforcement Learning
Authors: Renos Zabounidis
Joseph Campbell
Simon Stepputtis
Dana Hughes
Katia P. Sycara
Venue: CoRL 2022 Poster (2022)
Keywords: Multi-Agent Reinforcement Learning
Interpretable Machine Learning

Abstract: Multi-agent robotic systems are increasingly operating in real-world environments in close proximity to humans, yet are largely controlled by policy models with inscrutable deep neural network representations. We introduce a method for incorporating interpretable concepts from a domain expert into models trained through multi-agent reinforcement learning, by requiring the model to first predict such concepts then utilize them for decision making. This allows an expert to both reason about the resulting concept policy models in terms of these high-level concepts at run-time, as well as intervene and correct mispredictions to improve performance. We show that this yields improved interpretability and training stability, with benefits to policy performance and sample efficiency in a simulated and real-world cooperative-competitive multi-agent game.

Concept Learning for Interpretable Multi-Agent Reinforcement Learning
Renos Zabounidis; Joseph Campbell, Simon Stepputtis, Dana Hughes, Katia Syeara
Carnegie Mellon University
(renosz, jacampbe, sstepput, danahugh, sycara] @andrew. cmu, edu
Abstract: Multi-agent robotic systems are increasingly operating in real-world environments in close proximity to humans, yet are largely controlled by policy models with inscrutable deep neural network representations. We introduce a method for incorporating interpretable concepts from a domain expert into models trained through multi-agent reinforcement learning. by requiring the model first predict such concepts then utilize them for decision making. This allows an expert to both reason about the resulting concept policy models terms of these high-level concepts at run-time, as well intervene and correct mispredictions to improve performance. We show that this yields improved interpretability and training stability, with benefits to policy performance and sample efficiency in a simulated and real-world cooperative-competee
multi-agent game.
Keywords: Multi-Agent Reinforcement Learning, Interpretable Machine Learning
Introduction
With burgeoning adoption in fields such as autonomous driving, service robotics, and healthcare, multi-agent robotic systems are increasingly operating in real-world environments. The actions of these systems have tangible and significant impact, particularly when operating in close proximity to humans. While we expect such systems to exhibit safe and accurate behavior, errors are inevitable, and in such circumstances it is vitally important that the agents able to explain their behavior to human operators. Operators can then ascertain whether the agent is operating erroneously - thus requiring intervention or correctly but in non-obvious manner.
However, state-of- ofee--o-theettte multi- aaeti- systems are often controlled by deep neural network models trained with reinforcement learning techniques [1]. While these methods have shown great ability to generate effective and generalizable models, they do at the expense of interpretability, and the models often remain inscrutable to human operators [2]. This poses a significant risk, especially in end-to-end models, where not clear what information has been extracted from raw observations in order to make policy decision. Domain experts often reason about agent behavior in terms of high-level concepts such as the presence of an obstacle - e.g.. "the robot encountered an obstacle and subsequently changed direction".. However, standard end-to-end models provide no mechanism for this sort of reasoning, let alone the ability to intervene and correct the model when it is wrong e.g., "the robot should have detected obstacle but it tid Such mechanism is particularly important for robotic systems where we often encounter shifts in data distributions, such as when transferring policies from simulated environments to the real world, leading model errors. These errors are exacerbated in multi-agent systems, where errors in each individual agent are compounded and produce large errors in environment dynamics.
In this paper, we propose a method for learning interpretable policies concept policy models for multi-agent reinforcement learning (MARL). Our approach is predicated on the insight that we can leverage domain knowledge from an expert order to regularize the model and influence what information is encoded from observations. We organize this domain knowledge into set of interpretable concepts and enforce the constraint that the model able to predict these concepts from observations, after which the concepts are used to predict policy actions. Concepts are semantically
Equal contribution
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Environment at
Concept Policy Model
Enviroument at L+1
Obeervation Encoder
Concupla
Stralegy
Activation
Agent Policy
Range
Strategy Concept
Concept
Range Concept
Extraction
Removed from Game
Val.
Interv.
Residual
Soft Concept
Only
Figure 1: Concept policy models predict set of interpretable concepts from observations, which are then used along with an (optional) residual predict policy action. A domain expert may intervene and provide corrective concept values to the policy mis-predicted.
meaningful labels that can be extracted from observations, such as the presence of a concrete or abstract feature in observation, e.g., the existence of an obstacle the intention of human. Crucially, we find that the regularization imposed by the concept information helps stabilize the training process, and result leads improved performance and sample efficiency.
typical end-to-end neural network policy model maps observations to actions [3]. Our approach inserts an intermediate concept layer, shown in Fig. I which is required to predict concepts from observations. While this yields an interpretable model [4]. also imposes the assumption that the set of concepts sufficient for policy inference. To residual layer which passes additional information to the subsequent policy layers while ensuring remains decorrelated with the concepts. We posit that the interpretability of the model is proportional to the capacity of the residual layer; intuitively, the more residual information available, the less the model may rely on the concepts. We show that this result in trade-off between interpretability and accuracy, given the expressivity of the concepts. Our contributions are as follows, we
this constraint,
introduce scalable
Introduce a method for learning concept policy models in MARL utilizing expert domain knowledge, enabling the expert reason about policy behavior in terms of high-level concepts and improving accuracy, sample efficiency, and training stability.
Develop two specific formulations based on this method: soft-concept models and hard- concept models, and empirically show the trade-off between accuracy and interpretability.
Formulate an intervention methodology and show how this can be used offset model errors in general and in transfer learning (sim-to-real) scenarios.
Empirically show that our proposed approach produces interpretable, intervenable MARL policy models which exceed the accuracy of baseline MARL policies simulated and real-world game of "tag", between two teams of 2, 3, and robots each.
2 Related Work
Interpretability in supervised learning: Intepretability has been extensively studied within the field of supervised learning [5], which be largely grouped into two categories: the explicit creation of an intrinsically interpretable model, or the post-hoc transformation of an uninterpretable model interpretable one. The former case typically revolves around considering interpretable classes of models decision trees [6, 7], linear models [8, 9], or rule-based methods [10, 11] for example and developing algorithms for these models. In the latter case, uninterpretable models either transformed into interpretable ones [12, 13, 14], or interpretable models are extracted from an uninterpretable model for the purpose of explaining model's decision rationale [15, 16].
Concept models often fall into the transformation case, and have been studied within the context of transforming set of uninterpretable feature embeddings into of interpretable concepts [4, 17. 18]. recent approach [19] similarly uses concepts, but rather than directly predicting such concepts attempts align the internal model representation coincide with them.
Interpretability in reinforcement learning: As supervised learning, interpretability for reinforce- ment learning largely falls into the two categories of intrinsically interpretable models and post-hoc

---

transformations, However, there is an additional line of work in which methods are devised to explain aspects particular the Markov decision process model employed by RL methods. Some approaches have focused on interpretable representation learning [20, 21] and hierarchical decompositions [22], while others have opted to tackle MDPP-peciffi explanations such as an interpretable reward sig- nal [23] or action explanation [24, 25, 26]. However, to the best of our knowledge, concept-related methods have not yet been explored in an RL setting, let alone MARL.
3 Preliminaries
Multi-Agent Reinforcement Learning: We model the MARL problem as a decentralized partially observable Markov decision process (Dec-POMDP) [27]. A Dec-POMDP is defined as a tuple (S, P, R, 2,O,n, n, 7) in which S is the state space, U shared action space, the state transition function, R the shared reward function, Z the observation space, O the observation function, the number of agents, and y the discount factor. For a given time step, the environment has a state s E S and each agent E (.o...... an.) samples partial observation Z, E Z according to the observation function O(s, a) E Z. The agents simultaneously sample an action u, E U inducing a state transition according to P(s'ls. Is. u) E [0, 1]. Each agent receives reward ra according to the shared reward function RS..: Un) E R with a discount factor ? E [o, 1]. We follow a centralized training and decentralized execution approach (CTDE), thus learning central policy To (u, k.)) [0. parameterized by 0 by maximizing the discounted expected cumulative reward: E, [E. 7 R(sa, U.)]
Multi-Agent Proximal Policy Optimization: Multi-Agent Proximal Policy Optimization (MAPPO) [28] is straightforward extension to standard PPO [29] under the CTDE assumption in which we learn a single actor, TT, and a single critic, Va. parameterized by 0 and e respectively. When sampling from the environment, each agent executes the same learned policy with their individual observations and actions. As with all policy gradient methods, PPO seeks to compute the policy gradient by differentiating the following objective function:
L(0)= = E, [og k.. Ar),
(I)
where Å is the estimated advantage function. PPO extends this objective function by adaptively clipping the update gradient and applying entropy bonus to the policy to encourage exploration. If the value function and policy function share parameters, i.e., O = Φ. then the objective function must also include the value function loss.
4 Concept Policy Models
We propose a method for learning concept policy models, which integrates domain knowledge from an expert in the form of concepts into a neural network policy model. These concepts are intended to serve two purposes: they are useful predictors for the desired policy behavior and such allow an expert to reason about the policy in terms of high-level concepts, and mispredicted concepts can be corrected at run-time by the expert in order induce correct behavior. The expert hereafter referred to as an oracle- - provides an oracle function V..) which can be used to predict a ground truth concept vector of size given observation from an agent a, v = V(z.) where R Concepts may be either continuous or discrete, and represent interpretable features which are assumed to be relevant to the task at hand. As an example, let consider a cooperative-commetiie
multi-agent game in which two teams of agents play a game of "tag" during which one side must prevent the other from reaching specific location. In this game, an expert might identify specific features such as the location of the nearest enemy, or the opposing team's 's strategy as concept.
4.1 Policy Concept and Residual Layers
We integrate this concept information into an end-to-end neural network policy no (u,, k.,) which predicts the probability for agent taking action u, given observation z, and parameters 6. This is accomplished by inserting an intermediate layer coe(-) into the network to predict the concept vector, dividing the network into two parts: T (.) representing the portion of the network before the new layer, and T (.) representing the portion of the network after the new layer, such that T Tu, (z))
T82 (c(x) r(x)) where x=
(2)
m8 (z.)
(1)
(2)
and r(-) is a residual layer of size i designed to pass through non-concept information and the concept layer acts concept predictor, such that i = c(x). In our proposed concept policy model,
3

---

(2) |R - RRA acts as a feature encoder mapping an observation to feature embedding. The To1
newly inserted concept layer serves bottleneck such that c(-) R R' maps the feature embedding to concept vector, while the residual layer r(-) R + R$ maps the feature embedding to residual vector. The final policy layer 702 (-) TRi + R jll maps the aggregated concept and
residual vectors to a policy action. We train the concept layer c(-) by imposing an additional auxiliary loss (0) in the objective function optimized by MAPPO:
I(0)- (oooTTo (2a) Aa] (0) where,
3 (0 Σ 15( and L5(0)= { (v, 0x)
if discrete
(3)
MSE(U,, 0,) if continuous.
This loss is summed over each concept: mean squared error (MSE) is used for continuous concepts, and focal loss (FL) [30] for discrete concepts. The focal loss is a cross-entropy variant designed for class imbalanced situations which are likely to occur in our concept setting, as some concepts may be significantly rarer than others. In our above example with the strategy concepts, some strategies may be much less likely to occur than others, for instance. Note that for multi-class discrete concepts, a single abstract concept may consist of multiple nodes, and we refer to this as concept group. In the strategy case, suppose an agent team may only execute one of strategy A, B, or C at a time, thus these three concepts represent a single concept group and so when we pass the discrete concepts through a softmax activation in order to calculate the focal loss we do so in group-wiss wise manner.
The goal of the residual layer is to pass through information from Tel (-) that is not captured by the concept vector. Without the residual, the concept vector must sufficiently represent the observation so that T accurately infer the agent's action from concepts alone (a strict assumption in practice). We define two concept policy model variants: hard concept policy models which contain no residual (k = 0), and soft concept policy models which do (A: 0).
By examining the concept layer activations, an oracle can query the predicted concepts 0 and understand what concepts the policy model used for prediction. However, we conjecture that there is an inherent trade-off between the size of the residual layer k and the interpretability of these activations. While full interpretability analysis is outside the scope of this work, we posit that the greater the residual dimension, the less that (2) (-) must rely on the concept vector, i.e., there is larger amount of non-concept information on which to base its prediction which follows that is
inversely proportional to interpretability.
4.2 Concept and Residual Whitening
In order to constrain the residual r(.) such that it does not encode information related to the concepts, we decorrelate the neuron activation vectors via whitening. Given a matrix X E Rbxj+* consisting of the activations from the concatenated concept and residual vectors over a mini-batch of samples, we aim to produce a whitened matrix X' with ZCA whitening via iterative normalization [31]
X' DA D pe)
where D and A are the eigenvectors and eigenvalues of X respectively. Iterative normalization uses an iterative optimization technique to incrementally whiten the matrix X, where the hyperparameter T dictates the number of optimization iterations. This gives us the flexibility of only partially decorrelating the residual and whitening layers, if desired, by setting T to a smaller value, e.g., T 2. In practice, we find that performing fewer iterations is often necessary to stabilize the training process, as higher I'' tends to increase the stochastic normalization disturbance and leads to reduced training efficiency [31], which is particularly noticeable in MARL setting. At each training iteration, we first perform whitening then backpropagate our computed gradients, thus allowing us to decorrelate the concept and residual layers without requiring an additional optimization step as in prior work [19].
4.3 Policy Intervention
In addition to querying the predicted concepts 0, an oracle may also decide to intervene when these predictions are incorrect. This can be achieved by explicitly overwriting the concept layer node activations (or softmax activations for discrete concepts) with the appropriate values. We denote the modified concepts as ō which leads to the following intervened concept policy model:

---

Step
Step
Step
Step 36
Figure 2:
A sequence of steps from a single episode during a policy execution in the real-world. The blue circle is the attacking team's 's (red) goal while the defending team (green) attempts to stop them.
(u La, 8) = (2) (o + r(x)). This policy intervention corrects prediction errors in the feature
encoder Tol (.). We find that these sorts of errors are particularly prevalent when transferring robot policies from simulation to the real world due to different observation distributions, and show that policy interventions are effective at reducing such errors in Sec. 6. Similar to interpretability, we hypothesize that intervention effectivvness is inversely proportional to the size of the residual k; we cannot intervene on the residual layer activations so any resulting errors will persist. While our empirical results hint in this direction, we save a full exploration for future work.
5 Experimental Setup
We show that proposed concept policy models achieve high policy success rates and concept accuracy, in addition to improved training stability and sample efficiency, over standard MARL models in cooperative-competiivv multi-agent game of "tag" previously described in Sec. 4. We empirically analyze our approach in both simulated and real-world versions of this game, and explore its strengths and weaknesses especially with respect interventions and sim-to-real transfer in Sec. 6.
5.1 Tag Game
In our game shown in Fig. two equally sized teams of agents compete with each other in which one team attempts to reach a specified goal location while the other team defends it and attempts to keep them away, which we refer as the attacking and defending team respectively. To allow for complex behaviors and strategy, an agent from each team may "tag" an agent from the opposing team as long as it lies within given proximity and is facing the opposing agent, removing the tagged agent from play. The attacking team wins if any agent is able to reach the goal location, while the defending team wins the attacking agents are all tagged or the maximum number of time steps elapses.
Observation and Action Space: The observations are a set of extracted features consisting of the positions, velocities, orientations and tagged status of all agents. Actions consist of accelerating forward or backward by a fixed amount, rotating left or right by a fixed offset, and tagging.
restricted our game such that only the defending tcam's policy is trained via MAPPO. While it is a straightforward extension to train both an attacker and defender policy iteratively, we opted to restrict the attacking team sampling strategies from fixed policy distribution to better investigate the effects of our concept policy model on performance. We sample attacker strategies from a distribution consisting of three "types" with equal probabilities, {random, left, right). where the attackers execute random actions, move towards the goal by sweeping along the left side of the environment, and move towards the goal by sweeping along the right, respectively. Given a sampled team level strategy, each agent then sampled an individual policy with noise from the strategy distribution, so to generate stochastic policies.
Strategy:
Furthermore,
We utilized the following concepts: (Range, Strategy. Target, Orientation. Position). in which Range is boolean concept indicating whether the opposing agent specified by Target is within tagging range, Strategy is a categorical concept mapping to the above team-level strategies. Target is categorical concept indicating opposing agent that should be pursued, and Orientation and Position are continuous concepts encoding the relative orientation and position of each opposing agent, respectively. The hard concept policy models are trained with the full set of concepts, while the soft models only employ a subset consisting of [Range, Strategy, Target).
Concepts:
Real-world Equivalent: The real-world version of our tag game is played in 2v2 scenario on 6' 6' play area, with four Khepera IV [32] robots. Policies are trained in the simulation environment, then executed in the real-world environment; no additional training and no fho-shot conditioning
5

---

Baseline
Hard
Soft
GI
0.8
G2
A1
0.6
Win
A2
A2
A1
A2
A2 G2 Range Concept
A2
A2
G1
G2
G1
G1
G2
2,50
500
750
1000
12.50
1500
Targeting Concept
Time Sceps [x 1000]
(a)
(c)
Figure 3: Left: training curves showing win rate vs iterations over 5 random training seeds for each tested model type 2v2 scenario. Right: a sequence of episode steps showing the concept activations for agents on the defending team (green).
employed. The robot positions and orientations are extracted from a Vicon [33] motion capture system and converted into the model's expected observation format. The real-world version of the game exhibits significant differences the dynamics between the simulated robots and the real- world robots particularly in velocities, accelerations, and even control presenting challenging environment for sim-to-real. Further, tagged agents disappear simulation while the real-world agents driven out of the play area, providing temporary obstacle.
5.2 Concept Policy Models and Baselines
We trained hard and soft concept model for 10M time steps for each scenario - 2v2, 3v3, and 5v5 along with standard policy model without concepts. Each model consists of series of fully connected layers, recurrent layers, and the iterative normalization layer applied over the concatenated concept and residual layers, with full details given in the supplementary material. The concept dimension j for each hard model differ for each scenario due to the number of agents: j = 13, j = 18. and j 28 for 2v2, 3v3, and 5v5 respectively. The concept dimensions for the soft models are j = 9, j = 12, and j = 18 for 2v2, 3v3, and 5v5. For the soft models, we additionally provide residual layer with dimension
52, 78 for 2v2, 3v3, and 5v5, respectively, leading to combined bottleneck size of 32, 64, and 96. The baseline model lacks concept layer (j 0) and has full-width residual 128. Residual layers sizes and other hyperparameters are given the supplementary material and were chosen through extensive hyperparameter optimization.
23,
Results
The win rates and concept accuracy errors for the defending team in both simulation and real- world are shown in Table 1. These values are computed by training two seeds with the best set of hyperparameters found during optimization, then rolling out each policy for 100 evaluation episodes in simulation, and 20 in real-world.
We first observe that both concept policy model variants out-perform the baseline model each scenario, with the hard concept model outperforming the others by large margin. This in itself is unsurprising, given that the concepts were hand deninde so to provide a sufficient amount of information for the policy, and the hard concept policy model heavily regularizes the learned model such that it learns this information. The decreased performance in the soft model is due to the fact that it is only trained with a subset of concepts and consequently the residual struggles to reliably encode this information on its own. As evidenced by the large win rate std shown in Table 1 on Lines I and 4, some seeds yield similarly strong performance to the hard models while others perform much worse with the exception of 5v5 where they failed to learn all. From this we can conclude that the soft concept policy models can offer comparable performance to hard concept models without requiring a fully-descriptive concept set, at the cost of increased training instability (see Figure 3 for a 2v2 scenario). The intervened win rate follows when an algorithmic oracle intervenes at every time step and sets the correct concept value when concept is incorrectly predicted by the model, improving the win rate, particularly in the more complex 3v3 and 5v5 scenarios.
Simulation:
Real-world: In the real-world environment shown in Fig. 2, we can see in Table 1 on Lines 10-12 that the win rates are drastically reduced for the concept policy models, but surprisingly not for the baseline model. Qualitatively, have observed that this is because the baseline model became

---

Setup
Model
WR
Inter. WR
Range
Strutegy
Target
Orientation
Position
Soft
0.51 (0.35)
0.55 (0.09)
0.03
0.04
0.24
Simulation
2v2
2
Base
Hard
0.19 (0.07)
0.83 (0.01)
0.84 (0.01)
0.04
0.07
0.20
0.10
0.11
4
Soft
0.55 (0.31)
0.57 (0.27)
0.03
0.10
0.17
3v3
Hard
0.74 (0.01)
0.80 (0.01)
0.03
0.13
0.23
0.11
0.14
Base
0.16 (0.06)
Soft
0.32 (0.01)
0.40 (0.01)
0.02
0.25
0.52
5v5
Hard
0.78 (0.04)
0.86 (0.01)
0.03
0.14
0.13
0.11
0.21
Base
0.31 (0.04)
Real
2v2
10
Hard
Soft
0.25
0.10
0.00
0.95
0.03
0.04
0.53
0.53
0.92
0.13
3.48
0.81
12
Base
0.35
Table 1: The win rate (WR) and concept errors for our proposed models (Soft and Hard) and a baseline without concepts (Base). The Hard model is trained over all concepts, the Soft model over a subset, and the Base model with none. The win rate is the average standard win rate of the policy when the policy is executed over multiple seeds with the standard deviation shown parentheses, while the intervened win rate (Inter. WR) is the average win rate when an oracle intervenes over all concepts and sets correct values. Range, Strategy, and Target are discrete concepts and as such the error shown is the error in accuracy score, while Orientation and Position are continuous and indicate mean squared error. Orientation is in radians and Position is unit-less value in [-1, 1].
trapped in a local minima and learned policy which was semi remi-performant and independent of the actions of the opposing team, causing it to drive in circles while constantly "tagging". This behavior highlights the difficulties standard MARL algorithms have when attempting learn meaningful feature embeddings. The other interesting result from this experiment is the gain in performance by the hard model when interventions occur, and similarly the lack of improvement when the soft model is intervened. We can draw two insights from this: the distribution shift from the simulated to the real world environment is largely contained within the feature extractor, which is compensated for by the interventions in the hard model; and that the Orientation and Position concepts are by far the most important as when we are unable to intervene on them and correct for dynamics errors as in the soft model, performance fails to improve.
Next, examine ablated hard con- cept policy models which are only trained over a subset of concepts in the 2v2 simulated scenario. The results are shown in Table 2 (left) and further support the evidence that the Orientation and Position concepts are by far the most important with respect to the win rate. In the sim- ulation environment, the win rate drops 23% (Line 2) in the absence of those concepts. while in the real-world environment it drops to 27%. Note that the only differ- ence between the hard model and the soft model with this concept set is the presence of residual layer; when this residual is present and allowed to encode additional infor- mation the win rate is nearly doubled 51% as the soft model in Table 1 (Line 1).
Concept A Allaioon:
Concept
Ablation
Rng.
Str.
Tgt.
Pos.
of
Concept
Interv.
2v2 3v3
Sim Real
83% 74%
84% 95%
23%
27%
84% 20%
80% 69%
81%
60%
80% 72%
78% 80%
Table 2: Concept ablations when only a
subset of concepts trained in simula- tion, and intervention ablations in both the real and simulated 2v2 scenarios when only a subset of concepts are in- tervened over.
We performed ablation over the set of intervened concepts, with the results shown in Table 2 (right). We can first observe that ignoring interventions over the Orientation and Position concepts does not affect the win rate (Line 2). likely because the associated errors for those concepts are already low as shown Table (Line 2). As we intervene over fewer and fewer concepts, the win rate further drops: however, paradoxically the win rate drops to below the base win rate without any interventions at all. In the real-world we observe more pronounced effects due to the differing dynamics, where the strategy concept is more difficult to predict and impacts the behavior more significantly. We conjecture that this is because the agents are less agile in the real-world, and errors in the predicted strategy lead to less-than-optina positioning. We note that there is an unexpected result here: intervening over every concept but Strategy (Line 3) yields a lower win rate than only Orientation and Position (Line 4). This appears to be consequence of two factors: intervening over
Intervention Ablations:
7

---

Move
Move Right
No Intervention
0.5
Position
0.4
Pitag)
0.6
Win Rate
0.6
0.34
0.4
0.2
0.2-
0.-
0.1
0.0
-0.30
-0.05
0.00
0:05
0.10
0.0
Range 0
Rangee =
None
0.0
Nona
X Position
Target
Bagget
Figure 4: (Left) Intervening over Strategy for the 3v3 hard concept policy model in simulation, Each point is the average position of all the defenders as an episode progresses, where lighter values indicate earlier in the episode, averaged over 100 episodes. (Center) The probability of an agent performing the tag action over 1000 episodes when Range is predicted (None) or intervened (Range 0/1) in a simulated 2v2 scenario. (Right) The win rate of the defending team over 1000 episodes when Target is predicted (None) or intervened (Target A/B) such that both defenders share the same target in a simulated 2v2 scenario. The mean and 95% concept intervals are shown.
Range and Targer but not Strategy increases the prediction error of Strategy which in turn lowers the win rate; and the policy resulting from intervening over only Orientation and Position results in more collisions with other agents during execution, making them easier to tag.
Behavioral Effects: Figure 4 shows the effects of concept values concept policy model behavior. The left figure visualizes the average position of the defending team as an episode progresses when the Strategy concept is intervened to always take specific value, showing that the Strategy concept clearly correlated with the defending team's ' movement. Specifically, when the attacking team is predicted move to the left the right, the defending team moves in the corresponding direction and towards the attackers to intercept. The two figures on the right further illustrate behavioral effects resulting from concept values; the middle figure shows that the probability of performing the tag action dramatically increases when the Range concept is intervened and forced to 1, while the right figure shows that the win rate decreases when the agents are forced have the same Target rather than predicting their own independently.
These results reveal that high-level concepts impart meaningful behavioral effects the down- stream policy that align with the concept's interpretation; thus while the concepts do not provide full transparency into the policy's decision making process they still provide the ability for an expert to understand what high-level pieces of information are informing a policy and how these might influence behavior.
Limitations: order analyze the performance of our model for single team of agents, we have restricted the variability in our environment and reduced the complexity of possible behaviors. In the future, we would like to evaluate asymmetric team compositions and learn a policy for the attackers. We have also only considered low-dimensional inputs in our experiments, and although we expect our approach to scale well to rich input representations such as images, since concept models have been traditionally applied in vision domains, this remains open question. Additionally, would like to expand the complexity of our real-world environments by incorporating additional robots.
7 Conclusion
In this work have introduced concept policy models for Multi-Agent Reinforcement Learning which incorporate domain knowledge from an expert in the form of concepts. Concept policy models allow an expert to query the model at run-time and analyze policy behavior in terms of high-level concepts, and crucially, intervene and correct concept predictions when errors occur. We empirically show that concept policy models regularize the underlying policy, yielding improved accuracy and sample efficiency while stabilizing training. and demonstrate how oracle-based interventions can be leveraged to partially compensate for distributional shift in sim-to-real transfer scenarios. We find that such interpretability is particularly important in multi- settings even small changes in agent performance can lead to large changes in team coordination. Allowing an expert to understand what high-level concepts are used by a policy and reason about how these concepts affect individual agent behavior provides insight into team dynamics, and provides a mechanism for intervening and correcting behavior when necessary.
8

---

Acknowledgments
This work was supported by DARPA award HRO01 12000036, AFRL/AFOSR award FA9550- 1-0251, AFOSR award 1A9550-15-1-044 ARL 19---19-- NSF ISS-122422, and the National Science Foundation Graduate Research Fellowship under Grant No. 1745016.
References
[1] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. Nature, 5 518(7540):529-533 533, 2015.
[2] F. Doshi-Velez and B. Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv: 1702.08608 2017.
[3] S. Stepputtis, J. Campbell, M. Phielipp, S. Lee, C. Baral, and H. Ben Amor. Language- conditioned imitation learning for robot manipulation tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 13139-13150. Curran Associates, Inc., 2020. URL http:::tt//// http::/://////p//peeeeee neurips. cc
/0T00/////0///000000000600068830 Paper pdf.
[4] P. W. Koh, T. Nguyen, Y. S. Tang, S. Mussmann, E. Pierson, B. Kim, and P. Liang. Concept bottleneck models. In International Conference on Machine Learning, pages 5338-5348.
PMLR, 2020.
[5] R. Guidotti, A. Monreale, S. Ruggieri. Turini, Giannotti, and D. Pedreschi. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 51(5):1 2018.
[6] J. Huysmans, K. Dejaeger, C. Mues, J. Vanthienen, and B. Baesens. An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models. Decision Support Systems, 31(1):141-155) 2011.
[7] M. T. Ribeiro, S. Singh, and Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference knowledge discovery and data mining, pages 1135-10 1144, 2016.
[8] S. Haufe, F. Meinecke, K. Görgen, S. Dähne, J.-D. Haynes, B. Blankertz, and F. BieBmann. On the interpretation of weight vectors of linear models in multivariate neuroimaging. Neuroimage. 87:96-110; 110, 2014.
[9] Le Nguyen, S. Gsponer, I. Ilie, M. O'Reilly, and G. Ifrim. Interpretable time series classifi- cation using linear models and muuti-resslltionn multi-domain symbolic representations. Data mining and knowledge discovery, 33(4): 183-1222, 2019.
[10] X. Yin and J. Han. Cpar: Classification based on predictive association rules. Proceedings of the 2003 SIAM international conference data mining, pages 331-335, SIAM, 2003.
[11] Wang and C. Rudin. Falling rule lists. In Artificial intelligence and statistics, pages 1013- 1022. PMLR, 2015.
[12] R. Agarwal, L. Melnick, N. Frosst, X. Zhang, B. Lengerich, R. Caruana, and G. E. Hinton. Neural additive models: Interpretable machine learning with neural nets. Advances in Neural Information Processing Systems, 34:4699 4711, 2021.
[13] O. Bastani, C. Kim, and H. Bastani. Interpretability via model extraction. arXiv preprint arXiv: arXiv:1706.099 1706.09763. 2017.
[14] G. Liu, O. Schulte, W. Zhu, and Q. Li. Toward interpretable deep reinforcement learning with linear model u-trees. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 414- Springer, 2018.
[15] R. Guidotti, Monreale, Ruggieri, Pedreschi, Turini, and F. Giannotti. Local rule-based explanations of black box decision systems. arXiv preprint arXiv:1805.0 10820, 2018.

---

[16] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668-2677 PMLR, 2018
[17] K. J. Wu, C. Gan, A. Torralba, P. Kohli, and J. Tenenbaum. Neural-symbolic vqa: Disen- tangling reasoning from vision and language understanding. Advances in neural information processing systems, 31, 2018.
[18] M. Losch, M. Fritz, and B. Schiele. Interpretability beyond classification output: Semantic bottleneck networks. arXiv preprini
10882, 2019.
[19] Z. Chen, Y. Bei, and C. Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782, 2020.
[20] Lesort, N. Díaz-Rodríguez, J.-F. Goudou, and Filliat. State representation learning for control: An overview. Neural Networks, 108:379-392, 2018.
[21] Z. Lin, K.-H. Lam, and A. Fern. Contrastive explanations for reinforcement learning via embedded self predictions. arXiv preprint arXiv:20100051800,2 2020.
[22] B. Beyret, A. Shafti, and A. A. Faisal. Dot-to-dot: Explainable hierarchical reinforcement learning for robotic manipulation. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5014-5019. IEEE, 2019.
[23] Z. Juozapaitis, A. Koul, A. Fern, M. Erwig, and F. Doshi-Velet Explainable reinforcement learn- ing via reward decomposition. In IJCAI/ECAI Workshop on explainable artificial intelligence, 2019.
[24] F. Cruz, R. Dazeley, P. Vamplew, and I. Moreira. Explainable robotic systems: Understanding goal-driven actions in a reinforcement learning scenario. Neural Computing and Applications, pages 1-18, 2021.
[25] H. Yau, C. Russell, and S. Hadfield. What did you think would happen? explaining agent behaviour through intended outcomes. Advances in Neural Information Processing Systems, 33: 18375-18833 18386, 2020.
[26] Y. Guo, J. Campbell, S. Stepputtis, R. Li, D. Hughes, F. Fang, and K. Sycara. Explainable action advising for multi-agent reinforcement learning, arXiv preprint arXiv:2211.07882, 2022.
[27] F. A. Oliehoek and C. Amato. A concise introduction to decentralized POMDPs. Springer, 2016.
[28] C. Yu, A. Velu, E. Vinitsky, Y. Wang, A. Bayen, and Y. Wu. The surprising effectiveness of mappo in cooperative, multi-agent games. Advances in Neural Information Processing Systems, 2022.
[29] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 170700637777777 2017.
[30] T.-Y. Lin, Goyal, R. Girshick, K. He. and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[31] L. Huang, Y. Zhou, F. Zhu, L. Liu, and L. Shao. Iterative normalization: Beyond standardization towards efficient whitening. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages
2019.
[32] M. Soares, I. Navarro, and A. Martinoli. The khepera iv mobile robot:
performance evaluation, sensory data and software toolbox. In Robot 2015: second Iberian robotics conference, pages 767-781. Springer, 2016.
[33] Vicon Motion Systems. Vicon motion capture system. URL https://ww..wiceee com/ hardware/cameras//
10


Reviews:
[{"review_id": "yqqDnm2q1RR", "reviewer": "Reviewer_dgWp", "summary": "To improve the interpretability of the RL policy model, This paper enforces the policy model to encode the \"concept\" in the intermediate layers of the model. The concepts are categorical or continuous values relevant to the task. For example, in competitive games, the opponent's strategy (offensive/defensive) and range (0~100) can be the concepts. The policy model consists of two major modules. The first model predicts the concepts based on observations. Domain experts can correct the predicted concepts. The second module transforms the concept's value into actions. The advantage is that human experts can intervene in the concept predictions and correct them. The experimental results show performance improvements.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- The idea of using concept learning on sim-to-real and interpretability is interesting.\n\nCons:\n- Section 6 is very hard to parse -- especially real-world results. I suggest making those results in the real world as figures or tables instead of describing them in a dense paragraph.\n- Lack of experiments. The experiments are limited -- only in 1 task. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend weak acceptance since the author successfully validates the claim described in the paper and there is no obvious technical flaws.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": " ", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}, {"review_id": "u37QhzeuK3m", "reviewer": "Reviewer_p3Pf", "summary": "This work investigates the interpretability of multi-agent RL and introduces a method for incorporating interpretable concepts from domain expert. \n\nThe concepts can be considered as hidden states of current environment.\n\nThe proposed Concept Policy Model decomposes the policy network into several parts: \n\n1. A feature extractor, \n2. A residual layer that produces \"residual vector\" that has no information related to concept, \n3. A concept layer that produces \"concept vector\",\n4. Policy layer produces action distribution.\n\nFor the concept layer, supervised learning is conducted to fit the concept vector to ground-truth concepts provided by environment.\n\nFor the residual vector, the ZCA whitening is used to manipulate the activation matrix and let the output does not contain information about the concepts.\n\nThe concept vector output by the concept layer serves as part of the input to the final policy layer. This means that (1) we can query and manipulate the concept during execution.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n### Strengths\n\n* The interpretability of MARL agents is a valuable research topics. I am happy to see work pursing this.\n* The idea that \"human can intervene when the agent understands the environment in a wrong way\" is inspiring. This reminds me works on intervention-based imitation learning and model-based learning. The concept layers in this work is just like the dynamic models (both are trained with environment-provided ground truth information).\n\n\n### Weaknesses\n\n* The clarity is poor. As an example, in Sec 4.2, ZCA whitening is conducted on the activation values of residual values. The motivation here is to let \"residual layer not to encode information related to concepts\" (line 152). What is the connection between \"let it not encode something\" and \"use ZCA whitening\"? Also, what is the $\\mu_x$ in Equation 4?\n* The motivation and whole story are disconnected from the method and experiment that really implemented. See \"Summary Of Recommendation\".\n* The experiment is oversimplified (See \"Issues, Method, Q3\") and the \"concept\" is a concept that poorly defined.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "\nGenerally, I think the method proposed in this work is far less novel and impacting than it is claimed in this work.\n\n\nI admit that the problem trying to be addressed is important, the interpretability of MARL algorithms. \nI would expect to see the authors take the unique feature of the problem, multi-agent, into the consideration and propose a method that can infer the causality between agents' behaviors. \n\n\nHowever, this work proposes to decompose the policy network and use extra supervision signal to force the concept layer to predict ground truth concept. Conditioning the policy network with \"high-level concept\" is not a novel idea and is widely used before, such as [1]. \nIt turn out to be a method that adding supervision loss to fit human-labelled concepts and there is no any special design for MARL setting.\n\n[1] End-to-end driving via conditional imitation learning\n\nThe mismatch between what it claims and what it does is the major reason for my recommendation.\n\n1. In conclusion, the authors claim \"Concept Policy Models incorporate domain knowledge from an expert\" (Line 310). Why don't you just use those concept as input directly to the policy network? (See Issues - Method - Q3) Also, the proposed method is not \"a general framework\" (Line 311) since it requires hand-coded \"concepts\" from the environment and the \"domain of concepts\" is designed by human (I can accept this, it is inevitable that we need some prior knowledge. However, this makes the proposed method unqualified to be a \"general framework\").\n2. In abstract, the authors claim \"This allows a human operator to both obtain decision making rationale and intervene\" (Line 8). However, I can't find the description on how human expert intervene and improve the wining rate. I am doubting whether a human expert can really figure out the logic of agents' behavior by watching per-step predicted concepts (which, basically are a set of informative values of current environment, like locations of nearby enemies).\n3. As I discussed above, I can't find any particular design to make the proposed method more suitable in MARL setting.\n\nIn conclusion, I think current submission is below the bar of acceptance.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "\n### Method\n\n1. For the residual vector, the ZCA whitening is used to manipulate the activation matrix and let the output of residual vector does not contain information about the concepts. I am not fully understand what is happening during this process. Could you explain more here?\n\n\n2. Line 231: \"The iterative normalization layer is applied over the concatenated concept and residual layers.\" IIUC, the concept layer should not be \"whitened\"? \n\n\n3. Is it possible that the concepts you selected as described in \"Concepts\" (Line 212) are very strong and informative observation? That is to say, consider a extreme case where your concept layer is perfectly trained, then it should always output ground-truth concepts as described in Line 212. Those information is strong enough, along side with the information came from residual layer that is extracted from the observation, to learn a powerful agent. To justify this, I will expect a new experiment that use directly the ground-truth concepts as input (replacing the output of concept layer) to the policy layer.\n\n4. How you overwrite the activation of concept layer? (Sec 4.3)\n\n\n5. Other issues on method is in \"Summary Of Recommendation\".\n\n\n\n\n\n\n### Writing\n\n1. Line 59, 112, 207: Wrong latex code for left quotation mark\n2. Table 1 is too far from Result section.\n3. I would expect to see a figure to illustrate the network architecture of Concept Policy Models in Sec 4.1\n4. Line 310 - Line 314: A long and non-informative sentence. I don't think it provide any interesting conclusion to reader.\n\n"}}, {"review_id": "cNVCEwxLJq", "reviewer": "Reviewer_avMX", "summary": "This paper proposed to adopt the concept bottleneck network to multi-agent RL. The network architecture induces interpretability and stabilizes training, according to experimental demonstration. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. This paper is genreally well written and easy to understand. \n\n2. The motivation is strong. \n\n3. The experimental results are strong and convincing. \n\nWeaknesses: \n\n1. It could help to present a diagram of the network architecture described in Eq. 2. \n\n2. I would recommend adding parenthesis to the superscript in the policy, as $\\pi^{(1)}_{\\theta 1}$. Otherwise, the superscript is easy to be confused as power exponent. \n\n3. The network architecture does have limited novelty above the conept bottleneck network, but it is refreshing to see it being adopted to MARL. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend this paper for acceptance. In my opinion, this paper contains sufficient ideas and executions for publication. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "This is more of a discussion point, but in Eq. 2 (with notations simplified), the fact that $\\pi^2$ is applied on $r(x)$ could arbitrarily amplify its effect. Is it possible to change the equation to $\\pi^2(c(x)) + r(x)$, so that the contribution of the residual network is much easily quantifiable? "}}, {"review_id": "71VvVcTeJR", "reviewer": "Reviewer_EPqJ", "summary": "This paper adds additional supervised signals to support the policy training (PPO) in a multi-agent RL setting. In detail, they introduce classifiers and regressors to predict concrete features (concepts) like relative positions or the strategy of enemies and train these modules in a supervised way. The output actions are conditioned on these predicted features. In this way, they state the interpretability of the policy is improved. Experimental results show that this method helps improve the defending team's performance in the tag game in both simulation and real robotics.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the research topic is important in reinforcement learning. The writing is in general easy to follow. The work has hardware experiments on real robots.\n\nThe main weakness from my perspective is, whether their method indeed improves the interpretability largely, or their method just shows that a better feature extractor helps the performance, which is well-known in deep learning. From Tab.2 we can see that the most important ‘concepts’ are relative position and orientation. Without these two concepts, the win rate drops dramatically from 83% to 23%. The other concepts only affect 3%. As the original observation already contains the absolute position and orientation, knowing the relative version doesn’t improve any ‘interpretability’ of the policy. Converting the absolute version to the relative one is more like input preprocessing. Besides, it is already well known that relative position and orientation are better than the absolute version in such a setting. \nI think I agree that predicting the ‘target’ can improve the interpretability of the policy. However, according to their experiments, this concept doesn’t affect the performance so much. I think they need a better experiment setting (like more interesting concepts) to solve this issue. Besides, I think they can design some experiments to show the improved interpretability of the policy directly. For example, they can show that when they intervene in the ‘target’ concept, the defender agent will change their tagging target in the environment.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I feel what they have done is more like using supervision signals to train a better feature extractor that helps performance instead of introducing something which clearly improves the interpretability of the policy. They should have some experiments to demonstrate the improved interpretability directly. In addition, in their experiments, the concepts that actually matter are relative position and orientation. It is better to have experiments with more interesting concepts.\n\n**After Rebuttal**\nI appreciate the authors' efforts on the new experiments in Appendix D that show the interpretability directly. I have increased my score to weak accept. I'll suggest authors include more experimental results that demonstrate the improved interpretability in the main paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "They use lots of sentences (e.g., the whole Sec.4.2) to describe the soft model. But the soft model works much worse than the hard one. In L250 they explain that this is because soft model uses fewer concepts. In this case, I think an experiment where soft model uses all concepts like the hard one is needed for a clear comparison with hard model. Otherwise, I’d consider it as an interesting but not so important variant, as the soft model’s performance is much worse, and the soft model is more complex than the hard one in both architecture and training way. In this case, the content of the soft model should be reduced in the main paper and maybe moved to the appendix.\n\nOther small writing issues:\n\n1. In Eq.2, I think $\\pi^1_{\\theta1}$ doesn’t output a distribution over the action. Therefore, it should be $x=\\pi^1_{\\theta1}(z_a)$ instead of $x=\\pi^1_{\\theta1}(u_a|z_a)$. It is actually better to use other notation to denote $\\pi^1_{\\theta1}$, as it is not a policy, but the first part of the policy net (more or less like a feature extractor).\n\n2. In L261-263, I cannot see the win rates of both the concept model and baseline model from Fig.3. This sentence should be rewritten.\n"}}]

Author rebuttal:

Decision: Accept (poster)