Beyond Rewards: a Hierarchical Perspective on Offline Multiagent Behavioral Analysis
Authors: Shayegan Omidshafiei
Andrei Kapishnikov
Yannick Assogba
Lucas Dixon
Been Kim
Venue: NeurIPS 2022 Accept (2022)
Keywords: multiagent systems
interpretability
multiagent reinforcement learning
reinforcement learning
behavioral analysis

Abstract: Each year, expert-level performance is attained in increasingly-complex multiagent domains, where notable examples include Go, Poker, and StarCraft II. This rapid progression is accompanied by a commensurate need to better understand how such agents attain this performance, to enable their safe deployment, identify limitations, and reveal potential means of improving them. In this paper we take a step back from performance-focused multiagent learning, and instead turn our attention towards agent behavior analysis. We introduce a model-agnostic method for discovery of behavior clusters in multiagent domains, using variational inference to learn a hierarchy of behaviors at the joint and local agent levels. Our framework makes no assumption about agents' underlying learning algorithms, does not require access to their latent states or policies, and is trained using only offline observational data. We illustrate the effectiveness of our method for enabling the coupled understanding of behaviors at the joint and local agent level, detection of behavior changepoints throughout training, discovery of core behavioral concepts, demonstrate the approach's scalability to a high-dimensional multiagent MuJoCo control domain, and also illustrate that the approach can disentangle previously-trained policies in OpenAI's hide-and-seek domain.

Beyond Rewards: a Hierarchical Perspective on Offline Multiagent Behavioral Analysis
Shayegan Omidshafiei somidshafiei@googg com
Andrei Kapishnikov kapishmikor@google...o com com
Yannick Assogba yassoggaagooglee.com
Lucas Dixon 1di Idixon@google.com com
Been Kim
beenkimggoogle.com com
Google Research
Abstract
Each year, expert-level performance attained in increasingly-complex multiagent domains, where notable examples include Go, Poker, and StarCraft II. This rapid progression accompanied by a commensurate need better understand how such agents attain this performance, to enable their safe deployment, identify limitations, and reveal potential means of improving them. In this paper we take a step back from performance-focuse attention towards agent behavior analysis. We introduce model-agnostic method for discovery of behavior clusters in multiagent domains, using variational inference learn a hierarchy of behaviors at the joint and local agent levels. Our framework makes no assumption about agents' underlying learning algorithms, does not require access to their latent states or policies, and is trained using only offline observational data. We illustrate the effectiveness of our method for enabling the coupled understanding behaviors at the joint and local agent level, detection behavior changepoints throughout training, discovery of core behavioral concepts, demonstrate the approach's 'ss scalability to a sigh-dimensional multiagent MuJoCo control domain, and also illustrate that the approach disentangle previously- trained policies in OpenAl's hide-and-seek domain.
multiagent learning, and instead turn our
Introduction
Multiagent approaches have driven numerous advances in artificial intelligence research, with seminal examples including TD- ID-gammon [T]. DeepBlue [2], AlphaGo B. AlphaZero [4], Libratus [5].
AlphaStar [6], OpenAI Five 77 and Pluribus [8. During training, many of these approaches seek to
push the performance of agents as measured by reward signal, or derivatives thereof.
Despite this, post-hoc methods that seek understand agent interactions often use less reward-centric techniques. Instead, insights are drawn from behavioral analysis to identify unique or interesting agent strategies. Examples include clustering-based analysis of neuron activations and trajectories in capture-the-flag [91. inspection of trajectories in a hide-and-seek domain to detect interesting behaviors such agents that learn exploit the underlying physics engine [0], monitoring of statistics such as pass ranges and frequencies in humanoid football [ and analysis of A AlphaZero's acquisition of chess knowledge [1]. Crucially, such insights are often drawn via manual analysis
and detection of behavioral clusters, or use of statistics associated with certain behaviors as defined by humans experts (c.g., various skills and relevant metries humanoid football).
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Jeint (z,)
1000
Return
500
Trial A
Additional trials
Trial B
Mean (all trials)
0
10000
20000
30000
40000
50000
Training step
(a)
Trial
Trial
1,0
1.0
0.5
Roward
0.5
Roward
Agent
Agent (22)
Agent 2 (zg)
0.0
0.0
-0.5
-0.5
0
(b)
(c)
Figure 1: Reward alone is not enough to understand underlying behaviors in 3-agent hill-climbing domain. Agents here start at the origin, each receiving rewards by navigating to any of 3 equidistant hills. visualizes the total returns of agents throughout training, over 50 independent trials. Two trials (A and B) with similar final returns are highlighted. (b) visualizes the actual converged behaviors of the agents the end of Trials A and B, which distinct despite their similar returns. Visualizing these same trajectories in the behavior space learned by our approach immediately reveals differences in the joint behavior of agents in the top panel of [c) where the two color markers correspond the trajectories from each trial. Simultaneously analyzing the agent-wise latent spaces in the bottom panels of (c) highlight that agent behaves the same way in both trials, in contrast agents O and 2.
As evident above, understanding emergent multiagent behaviors is enriched by techniques beyond pure reward-based analysis, as behavioral signifiers are not always discernible via rewards. Figure la provides intuition on this notion, illustrating returns (sum of rewards) throughout training for multiagent hill-climbing domain (later described in detail). We highlight two independent training trials (A and B) with similar final returns. Despite similar returns, comparing the trajectories generated via the agents' deterministic policies following each trial's training (Fig. Tb reveals entirely different behaviors. Analogous examples are evident in the above works (e.g.. Fig. I of Baker et al. [10], where substantial behavior changes occur in multiagent hide-and-seek despite smooth reward curve).
This paper formalizes the problem of offline multiagent behavior analysis. Our proposed algorithm, Multiagent Offine Hierarchical Behavior Analyzer (MOHBA), learns hierarchical latent space that simultaneously reveals behavior clusters the joint level (i.e., interactions between agents) and local level (i.e., behaviors of individual agents). Our method is agnostic of the underlying algorithm used to generate agents' behaviors, requires no access or control of the underlying environment, does not assume availability of reward signal, and does not require access to agents' models or internal states. Our experiments investigate the structure of the learned behavior space, which goes beyond prior works latent-clustering by identifying relationships between individual agent and joint behaviors. We illustrate that clusters identified by MOHBA are useful for highlighting similarities and differences in behaviors throughout training. We also quantitatively analyze the completeness of discovered behavior clusters by adopting a modified version of the concept-discovery framework of Yeh et al. 13 to identify interesting behavior concepts in our multiagent setting. We then test the scalability of our approach by using it for behavioral analysis of several high-dimensional multiagent MuJoCo environments [14]. Finally, we evaluate the approach on the open-sourced OpenAl hide-and-seck policy checkpoints [10], confirming that the behavioral clusters detected by MOHBA closely match those of the human-expert annotated labels provided in their policy checkpoints.
2 Related Work
Significant research has been conducted in single-agent skill discovery, which seeks to learn reusable policies useful for downstream tasks [15)28) Related approaches discover motor primitives to express longer-horizon policies (29), including use of offline reinforcement learning (RL) for learning useful
behaviors [B0]. Option discovery methods learn temporally-bbstracted abstracted actions (i.e., options (3I), chained together to form cohesive skills B2 37]. In contrast to our work, these approaches focus on maximizing performance in single-agent settings. There also exists a related line of work for learning diverse policies in RI. settings 46) despite their focus on policy diversity, several of these approaches make much stronger assumptions than ours (e.g., access to the policies generating agent trajectories, unique identifiers of the policy that generated each trajectory). By contrast, our

---

Join
Latunt
Joint prior
Local Libent
iE T
Joint Term (Eq. 7)
Policies
Ta(a"
18, 2.2
Offine Data
Joint
priors
1
D
ga(Zu|7)
Pol & [2_.)
Actions Suites
si+1
S1+2 ...
KL Term (Eq. 6)
Local encoders
Policies
7o (7)
Sigla 8.
Reconstruction Term (liq.)
(b)
Figure 2: Approach overview. Graphical model of the lotent-conditioned trajectory generation process that MOHBA uses to learn multiagent behavior clusters. The joint behavior latent param- eter zu informs local (agent-wise) behavior parameters = which affects their behavioral policies. Given a state-action trajectory dataset, our approach learns these joint and local behavior spaces. (b) Architecture of the MOHBA model with variational lower bound terms to indicated.
work assumes no access to any of the underlying raw policies, or even labels of which policies agent trajectories were obtained from. Recent works have also focused on hierarchical skill learning in multiagent reinforcement learning (MARL). Lee et al. 44 use the mutual information maximization objective introduced in Eysenbach al. [16] learn multiagent policies. Yang et al. [48] use bi-level policy to learn agent skills: a high-level policy first generates latent vectors for each agent, and low-level policy conditions behaviors on said vector to perform the task. In Wang al. [4 distinct 'roles' are learned for agents to enable decomposition of tasks. Mao et al. (50) investigate use of consistent latent cognition variables in agent neighborhoods induce increased cooperation. Many MARL approaches have noted the emergence of interesting behaviors in multiagent systems in
specific domains of interest [5] In contrast to our focus, these approaches use RL maximize agent performance, rather than understand arbitrary behaviors via offline analysis.
RL interpretabilityy methods primarily focus single-agent settings and either modify the RL al- gorithm itself to increase transparency, or conduct post-hoc explainability [56]. These approaches represent agent policies as programming languages [57], extract visual summaries of behaviors using 'interestingness' statistics such as uncertainty in selected actions or the value of state-transitions (58), or combine agent neuron activations with gradient information construct behavioral embed- dings [59]. Behavioral clusters in our work share similarities with concept-based explanation approaches in non-RL domains. Detecting 'concepts' pre-trained models has been explored in vision (60)62), discrete games [12), and language [63, 66]. In vision, clustering based approaches describe discovered concepts using examples 660 or use generative modeling create new data method (i.e., superpixels) to sub-divide input before conducing clustering; while sub-division is less natural in RL, our LSTM and VAE baselines serve as an RL-aadpted counterpart of such works.
to describe concepts [62.. Ghorbani et al. [60 uses a vision-
Works using latent-clustering and analysis are also related ours. These include approaches using multi-level variational autoencoders [65H68] to learn compositional latent spaces, although not in decision-making domains such ours. Hierarchical latent approaches have been used in single-agent RL 669 77). Behavior analysis has also been conducted by embedding agent neuron activations into dimmeniiiiinal space, using
non-hierarchical) variational approach [9] The representation power of agents' internal states has also been gauged by predicting future events [ Overall, the key difference between the above works and ours that our method combined hierarchical learning with behavior analysis and applies it to the multiagent setting.
3 Offline Analysis of Multiagent Behaviors
This section introduces the offline multiagent behavioral analysis problem and our proposed algorithm.
Preliminaries. We first formalize the problem of offline multiagent behavioral analysis. Consider a rewardless multiagent Markov Decision Process (MA-MDP), defined by tuple (I, S, A, T), where I (1,..., N) is the set of N agents, S is the state space, A is the action space, and T denotes the state transition probability function. By not relying on the presence of rewards, behaviors generated without reliance on reward function (c.g., human interactions, agents using curiosity-
3

---

exploration) can be considered. We use the term 'local' for elements associated with individual agents, and 'join'' for those associated with the entire system. At each timestep t {O..... T-,,, the agents execute joint action de E A in state using joint policy (( (s),, causing the state to transition to 8141 with probability ((8t+1/81, a+) = ((+++1 0t; 8t). As standard multiagent frameworks [71], we assume the joint action space factorizes as A = x, such that = (a...... ........ a²") ), where i E I and aj E AA'. Similarly, x² (aj(ss) is the local policy for agent i, and π = II, n' is the joint policy.
ST-E:....,,,.111 (ar- 1ir-11eEz.5 ST) denote trajectory induced by this process, and D = {7 (.,...,, TK] denote dataset of K such trajectories. This dataset may consist of trajectories from multiple training runs, including variations over agent algorithms, hyperparameters, random seeds, or other factors influencing emergent behaviors. Given dataset D, the offline multiagent analysis problem seeks to uncover potential clusters of agent behaviors.
Let == (so.
Approach. Our approach, called the Multiagent Offline Hierarchical Behavior Analyzer (MOHBA), uses offline trajectory data to discover behaviors exhibited by agents at the local and joint level.
We first use Fig. 2a to build intuition before discussing technical details. Let the agent interactions exhibited in a trajectory 7 be encoded by latent variable, 2u E IR Dw capturing their joint behavior. For example, 2u may encode (at high level) whether agents were cooperating a given trajectory; conditioned on joint signal Zw. each agent then exhibits its own local behavior.
competing in
Let local latent vectors 2a (=),- ..... = encode individual agent behaviors, where 2 E RR'.. Conditioned on the local behavior vector z/, each agent then executes actions using a behavior- conditioned policy T''a'ls, [s, *. Given trajectory dataset DD we seek to learn the latent-conditioned policies and distributions over latent vectors, such that we can reconstruct any behaviors exhibited by the agents in D. Thus, latent vectors zة and 2a will encode the agents' behavioral spaces and, ideally, identify behavioral clusters the dataset. Given this framework, the joint policy is decomposed,
/
T((e(s)))
T(a St, Za (z lw )p( Zzw dzn d
N
II
x (aflor, =1)p((z (zw)p(z )dz. dew
(2)
1=1
where in (T) we have assumed that each agent's latent-conditioned policy is conditionally conditonally-indeeenden of the high-level latent behavior zu given its low-level latent = (see Appendix 1.1.1 for discussion). Next, given initial state distribution p(so) and latent behavior spaces, the probability of a trajectory under joint policy (--) is follows:
T-1
(T((T) = p(5))
II 1=0
p ( 1 (t)
(3)
N
(s))
Il p(s s a:) II
n (afls, 21 (p(2 low (p(p))((d)))))) dz...
(4)
We seck to learn the distributions over variables 2a and z.o alongside the latent-conditioned policies x² a'(a") (s,, ), which maximize trajectory probabilities 4 In Appendix derive the following variational lower bound, enabling approximation of these components using parametric models:
Jub = -E Ra -E (a
Σ a,a
log n r 2
(5)
E
(
(6)
+E,-D (KI (zw p[oo
where (o(2)(T) and (o lzw) are, respectively, learned encoder (posterior) and prior distributions over the local behavior latents o likewise, (=w /T) and po (2..) are, respectively, learned encoder and prior distributions over the joint behavior latent zwi 3 is a KL-weighting term as in 8- V-Es (72].

---

Joint (2,))
Agents
Cluster
Cluster
Cluster
Joint Clusters
Cluster
Cluster
Cluster
Agent (2() Agent 1 (2g) Agent 2 122)
Local Clusters
(a)
(b)
Figure 3: Results for 3-agent hill climbing domain (see interactive version here).
a Example
behavioral clusters discovered by MOHBA. (b) Trajectories corresponding each cluster, with reward-hills shown in grey. Clusters A to C show joint behaviors, whereas D to F separately show local agent behaviors (with other agents faded in local trajectory plots for readability).
Figure 2b illustrates MOHBA's model architecture, which is informed by the three bound compo- nents (5) to 7 During training, each trajectory T D simultaneously passed through the joint and local encoders, which respectively produce parameters for distributions over and (c.g.. parameters of Gaussian distributions). Samples of low-level latent vectors 2a are passed to reconstructed agent policies T9 (a/ St. 2 which are trained via the reconstruction component (5). The local KL-divergence component (6) induces the local encoder distribution (which conditioned directly on 7) to be similar to the local prior distribution (which is conditioned only on samples A thus enabling meaningful correlations between encoded local and joint latent space, as later shown. Finally, the joint KL-divergence component 7 is akin to that in standard variational autoencoder (73] [4]. Overall, MOHBA enables learning of a hierarchical behavioral space (at the joint and local agent levels, Zu and 2a, respectively) that exposes interesting behavioral clusters.
4 Experiments
We showease various use-cases for MOHBA a range of domains including continuous coordination games, multiagent MuloCo [14). and OpenAl hide-and-seek [10]. Appendix A.2 provides data generation, networks, computation, and hyperparameter details. Appendix provides pscudocode.
Data generation. Multiagent trajectory data is generated for each domain via the Acme RL li- brary [75]. using the TD3 algorithm (76) in a decentralized MARL fashion, with datasets managed using RLDS (77). Trajectories are collected at constant intervals throughout training, which also enables analysis of behavioral emergence. conduct wide sweep over random seeds for data generation, yielding diverse trajectory dataset (see Appendix A.5 for dataset details and statistics).
МОHBА setup. To analyze the above data using MOHBA, we use a GMM (Gaussian Mixture Model) for the joint prior, bidirectional LSTM (long short-term memory network) with GMM head for the joint encoder, an MLP (multi-layer perceptron) with Gaussian head for the local priors, bidirectional LSTM with Gaussian head for the local encoder, and an MLP for reconstructed policies. GMMs are used for the joint prior and encoder as they produce discernible joint behavior clusters 778, whereas the conditioning of the local prior on zu yields such clusters at the local level with standard Gaussian head. We use parameter-sharing across local priors, local encoders, and reconstructed policies, common in multiagent setups [79], with unique one-hot vector identifier appended to agent-specific network inputs to enable heterogeneity in model outputs.
Independent analysis of joint (2..) and local (2a) behaviors. We here analyze the hierarchical latent structure learned by MOHBA. Specifically, we highlight differences agent behaviors as identified at the joint and local levels, respectively, by zu and 2n, We first conduct simple sanity check in 3-agent hill-climbing domain (our earlier example Fig. [b). States and actions correspond, respectively, 2D positions (z, and forces (Ar, Ay) imparted by each agent for
5

---

Jeint
Joint
Joint
Thilting
20
Trairing Run juitt Rul
Agent (2))
Agent (e))
Agant I23)
Agent 1 (2])
Agent LZ?
Agent 1 (2))
Agent (2)
Agant 125
B
B
B
Traniing Joint Guu
C
C
C
Agent
Agents
(a)
(d)
Figure 4: Results for two-agent coordination game (see interactive version here). (a to show key joint clusters, the coupled local clusters for each agent, and associated trajectories in the domain (over all 50 MARL training runs in the dataset). (d) visualizes the progression of agent behaviors throughout the original MARL training phase, for two example training runs (top and bottom panels).
movement. Agents spawn at the origin and are each rewarded for climbing any of three hills (shown in grey the domain figures) within the domain in episodes of 50 timesteps. We generate a dataset using 50 independent MARL training trials, each conducted for 1e5 environment steps; trajectories saved every 200 steps, yielding 1.25e6 frames of data (25000 trajectories).
Figure 3a highlights several example clusters joint and local behaviors identified by MOHBA, using samples of 2ي and 2a from the joint and local encoders. We use Euclidean distances in the original latent spaces to identify nearby vectors, then visualize their 2D projection using principal component analysis. For each cluster, visualize all associated trajectories in Fig. 3b thus enabling analysis of behaviors captured latent space. Clusters A, B, and C correspond to examples of joint behaviors (zw). Cluster A contains trajectories early in training, where agents have not learned to converge to a particular hill in the domain. In Cluster B, all agents have learned to navigate towards the top-left hill. In Cluster C, agents 0 and 2 prefer the bottom-left hill, whereas agent prefers the right hill. Overall, clusters at the joint level capture meaningful collective behaviors of the agents.
Next, analyze individual agent clusters. Note that for all agents, three prominent clusters apparent in their respective za spaces, as each training trial in the data generation process can lead to various agent-wise hill preferences. We compare in the trajectory plots that these corresponds to this agent preferring the top-left and bottom-left hills, respectively. Similarly, Cluster F corresponds trajectories where agent prefers the bottom-left hill. These results illustrate that the local latents reasonably disentangle each agent's 's observed behaviors.
such clusters, D and E, for agent 1, observing
Coupled analysis of joint (z.) and local (z,) behaviors. The above experiments independently analyzed the joint and local latent spaces. We can also concurrently analyze them to better understand local agent contributions to joint behaviors. Consider two-agent domain with close inter-agent coordina- tion (visualized in the top of Fig. 4a). with state and actions- spaces similar the hill domain and episodes consisting of 50 timesteps. Three regions are defined in this domain: A and B (circular regions), and C' (region exterior to the circles). When an agent enters given region, it 'activates' the corresponding strategy in Table with agents receiving rewards at each timestep according to the joint strategy they have activated. For example, both agents enter region A, they each receive reward of 1, whereas one agent enters A while the other is in exterior region C, neither receives reward. This domain involves a significant degree of coordination as agents must discover the rewarding regions, while receiving sparse reward signal until valid combination of strategies is discovered. There is also
Table 1: Coordination rewards.
Agent
A
B
A
(1,1)
(1,1)
(0. 0)
Agent
B
11,1)
(0.
(0,0)
C
(0,0)
(0.0)
(0,0)

---

potential for miscoordination: navigating to region B is rewarding assuming the other agent navigates to A, but yields O reward if the other agent instead navigates to B (potentially destabilizing training). We run 50 independent MARL, sweeps in this domain, yielding 1e6 data frames (20000 trajectories). As shown in Fig. A.g
MOHBA discovers three dominant joint behavior clusters 2y here. In each of Figs. 4a to 4c we highlight one of these clusters and its corresponding local behavior latents &o for each agent. MOHBA reveals that across the dataset, the agents have learned to cover all 3 optimal joint behaviors in Table I (A, A). (A, B3, and (B, A). Moreover, despite each agent discovering two local behaviors (A and B), zu highlights only the observed joint behaviors (i.e., does not simply highlight 4 clusters consisting of the Cartesian product of individual agents' behavior spaces).
Behavior emergence throughout MARL training. We next use MOHBA's 's latent space inspect behavior emergence during MARL training. Figure 4d visualizes the training progression of two MARL runs used for data generation, tracking behavior changes throughout. In Training Run (top panel of Fig. 4d), the agents converge to and maintain a fixed joint behavior throughout training. By contrast, Training Run (bottom panel) has numerous behavior changepoints, where agents flip back and forth between preferring one of two clusters in zي. Concurrently inspecting the Aa space for Training Run 2, we observe that agent converges to and maintains a consistent behavior, whereas agent 1 changes its preference sporadically, also explaining the detected changes in joint behaviors.
Baseline comparisons. We next compare against baselines previously used for multiagent behav- ioral analysis in the literature I. An LSTM baseline conducts next-action prediction at each step in trajectory т, using action-prediction loss (APL), defined as the L2 loss over predicted vs. ground truth actions. This baseline targets using the LSTM hidden states, rather than a learned distribution over latent variables, for understanding and clustering agent behaviors (akin to the analysis in [ IT). A fAa baseline provides a non-hierarchical ablation of MOHBA that simply feeds the joint latent zw (rather than 2 to reconstructed agent policies (i.e., x (a" [sss )):;; we use our usual loss (with the local KL term (6) removed) to train this VAE. We conduct a hyperparameter sweep for the baselines (see Appendix A.2)) reporting the best results averaged over 3 random seeds.
Comparisons are conducted at the joint behavior level: for the LSTM, we use the fi- nal hidden state as an encod- ing of each trajectory; for our method and the VAE, we use zu directly. Table S com- pares the methods in the hill-
Table 2: Baseline comparisons. Action-prediction prediction loss (APL) and intra-cluster trajectory distance (ICTD); lower is better for both.
Hill-climbing domain
Coordination game
APL
ICTD
APL
ICTD
LSTM VAE MOHBA
4.8 - 0.1
0.47
2.9 : 0.1
0.34 + 0.15
6.9
0.18 1: 0.13
5.0 : 0.3
0.16 0.10
1.1
0.17
2.4 0.3
0.16 0.10
climbing and coordination game domains. Here we note the APL, which measures how well each method reconstructs ground truth policies. Additionally, we use K-means (sweeping over the # of clusters) to identify behavior clusters for each method, then report the intra-cluster trajectory distance (ICTD), defined as the average distance of all trajectories in a cluster from the mean trajectory in said cluster (akin to intra-cluster point scatter, a common cluster analysis statistic (80)) Combined,
these measures provide a proxy for evaluating the latent representations in terms of enabling accurate reconstructions (APL) while clustering similar trajectories together (ICTD). MOHBA significantly outperforms the LSTM and VAE baselines in terms of API., while also clustering similar trajectories at the z, level in terms of ICTD. We provide additional results including APL throughout training, full ICTD sweeps, and visualizations of the latent spaces for the basclines in Appendix A.3.2)
Behavior concept discovery. Next, we test the representation power of the latent spaces learned by MOHBA, and illustrate a means of discovering 'behavior concepts' in the latent space. We adopt the completeness-avaant -waeteeennssssaeeeeeeeeeee concept explanations framework of Ych et al. [13), with slight changes to make it amenable to this setting (see Appendix A.2.3). At a high-level, given set of inputs, a set of concept vectors C' in the same space as inputs, and prediction targets (e.g., domain characteristics of interest), Yeh et al. [33 define a framework to compute class-conditioned Shapley values, called ConceptSHAP. To do so, every input is projected onto each of the concept vectors, yielding a vector of "concept scores', which is then passed to a simple prediction head y (e.g., small MLP or linear model) to predict the targets and compute the ConceptSHAP. ConceptSHAP provides numerical scores interpreted as the importance of each concept in C' for predicting a given target class, which is useful for identifying key concepts (and nearby inputs) associated with certain domain characteristics.
7

---

Table 3: Discovered concepts using Aw hill-climbing domain. For each characteristic (agent dispersion and return), 5 classes are constructed using the ground truth information in trajectories The concept explanation framework of Yeh et [13] is then used first predict the correct classes (using only rather than the trajectory 7, input), then identify core concepts related to each class.
Concept
Top discovered concept using 2u (per class, with average concept measure beneath)
Class 0
Class
Class 2
Class
Class 4
Agent
Dispersion
(Classification
accuracy:
54.27%)
0.05 + 0.13
0.15 + 0.06
1.79 + 0.10
2.18 + 0.19
2.36 E 0.10
Total Return
(Classification
accuracy:
60.98%)
361.0 + 256.0
671.4 115.1
611.1 + 244.2
764.9 72.4
820.4 155.0
Example (see animated version here
Joint (zw)
Example B (see animated version here
Agent (25)
Agent (e))
Example (see animated version here
(a)
(b)
Figure 5: Multiagent MuJoCo HalfCheetah behavioral space (see interactive version here Agent and 1, respectively, control the back and front limbs, coordinating move the cheetah. (a) Detected behavioral space, with three example trajectories indicated. (b) HalfCheetah behavior corresponding to the same three examples (left panels showing trajectory frames, and right panels showing the same frames with the cheetah torso aligned to disambiguate the back and front leg agent behaviors).
In Table 3 we use this technique to identify concepts in 2u space associated with varying characteris- ties in the hill-climbing domain. The concept set C' considered is generated using K-means in the zu space (see Appendix [A.2.5 for details). The first row of Ssoos increasing levels of agent dispersion in the domain (with dispersion defined the sum of L2 di-distanc of agents from their centroid at the final timestep). For each trajectory т, we compute the associated dispersion, creating classes of equal-sized bins (labels 0 through 4 mapping to bins of increasing dispersion). We create an 80-20 train-validation split, then train a 2-layer (8 hidden units each) MLP g via softmax-cross entropy loss predict the classes using only zu as input (rather than the actual trajectory T). We attain a validation accuracy of 54.27%, signifying the predictive capabilities of Zu. Next, compute the class-conditioned ConceptSHAP, thus identifying the top-scoring concept vector for each class. In T we visualize the 20 trajectories with the closest 2u to the top-scoring concept for each class (with the mean agent dispersion across these trajectories listed under each image). Intuitively, the identified concepts involve agents becoming increasingly dispersed, with agents first nearly stationary at the center (class 0), then all converging the same hill (class 1), spreading across two hills (class 2), and finally covering all hills (classes 3 and 4).
shows classes corresponding to
The second row of Table repeats this experiment, now using classes associated with agents' sum- of-returns. We attain validation set accuracy of 60.98%, and observe behaviors associated with generally increasing reward by identifying top clusters in each class (with some overlaps, e.g., classes and 2 with high standard deviation in returns). Overall, these experiments help quantify the representational capacity of z. and identify clusters associated with distinct behavioral characteristics.

---

Rurning Chising
joint (2,)
Fort Building
Ramp
Hiders
Seekers
Oi Ramp Deflense
Agent 0 (eg)
Agent 1 (x))
Agent (x])
Agent (
Figure 6: Results for the OpenAl hide-and-seck environment [10. involving teams of agents (2 hiders and seekers). We mix the trajectories collected from 4 Open behavioral checkpoints (indicated the left panel) together, then train MOHBA this shuffled dataset. Using MOHBA, we observe the presence of behavior clusters that correspond well to human-annotated policy labels, both the joint and local agent levels, despite our algorithm not having access policy labels during training. Domain screenshots reproduced with permission from Baker et al. [T]]
Scalability to high-dimensional domains. We next test MOHBA's 's scalability to nigh-ddmenniionl domains using the 2-Agent HalfCheetah multiagent MuJoCo domain [14], where agents 0 and respectively, control the back and front limbs of a chectah to coordinate movement, Each agent's 's state consists of a 6D vector summarizing velocity and position information for the joints controls, with its 3D action space corresponding to motor torques applied to these joints. The agents coordinate maximize the forward-speed of the cheetah over episodes of 200 timesteps each. We generate data using 30 independent trials of MARL training, collecting 1e5 total trajectories to train MOHBA.
Figure 5a visualizes the behavior space learned by MOHBA, wherein observe several clusters. We highlight three example trajectories stemming from distinct clusters in the joint behavior space Zu. showing the corresponding HalfCheetah behaviors in Fig. 5b The left panel of each example provides a view of the cheetah's overall movement, where we observe key behavior differences: in Example the cheetah runs forward using subtle vibration its limbs; in Example B, it bounces forward with its torso arched up due to its front limb (agent 1) being more extended than its back limb (agent 0); in Example C, moves closer the ground with its torso arched down. In the latent space (Fig.[aa) we observe that for Examples B and C, the =0 (back limb) latents are close to one another, while the =/ (front limb) latents are far apart. To investigate these local agent behavioral differences. the right panel of Fig. providdvsdes an overlaid view of the same frames, with the torso now aligned (making easier to discern the behaviors of agent vs. agent 1). Here we observe that the back limb (agent 0) behaves similarly across both examples, while the front limb (agent I) stays much closer to the head for Example C, contrast Example B, which coincides with the findings in the 2, space. Appendix A.3.I provides additional results for 4-agent MuJoCo AntWalker environment.
Application to externally-traine policies: hide-and-seek game. We next apply MOHBA to policies trained by external teams. We consider the OpenAl hide-and-seck environment [0 where hiders and 2 seekers compete in rich environment with various interactive objects (boxes and ramps). OpenAl has open-sourced policies [8]] annotated by humans as exhibiting distinctive behaviors at key stages of training. We consider four policy checkpoints that correspond to the following human annotations: A) 'running and chasing", B) 'fort building'. C) 'ramp use", and D) 'ramp defense". The state-space used for each of the agents is 100-dimensional, consisting of the agent's own state (position, rotation, and velocity), states of the other agents, and the states of 3 boxes and I ramp in the environment (position, velocity, and box size); each agent's 's action space consists of

---

3- 3-dimensional force vector, and a "glue" and 'lock' action for interacting with objects. We collect 100 trajectories per policy checkpoint, each being 200 timesteps long. These trajectories exhibit wide distribution of behaviors, as agent and object initializations are random in each episode. We then mix the trajectories collected from all policy checkpoints, then train MOHBA on this shuffled dataset.
Figure 6 visualizes the behavior spaces discovered by MOHBA in the hide-and-seek domain. Agents 0 and in this figure correspond to 'hiders', whereas agents 2 and 3 are 'seekers'. We label each of the trajectories in this figure with the human-expert annotations provided by OpenAl. In Fig. O we observe the presence of behavior clusters that correspond well to the human- expannteetrttte labels, both the joint and local agent levels. Interestingly, for the seekers (agents 2 and 3), policy A ('running and chasing") ) highly distinctive and well-separated from the other behaviors. Moreover, despite the order of emergent behaviors in the original hide-and-seck MARL training being A - B - C - D, policies ('fort building') and D ("ramp defense') appear to be behaviorally slightly closer to one another than C ("ramp use") and D, both in the joint space and also the local spaces of the seekers (agent 2 and 3). This could perhaps be due to both the 'fort building' and 'ramp defense' policies being associated with situations where the seekers cannot easily find the hiders, due to the hiders using obstacles to block entrances (B) and moving ramps to prevent their effective use (D), Overall, these experiments help to validate MOHBA's learned latent spaces using policy labels manually annotated by human experts, and highlight its applicability to behaviorally- domains.
5
Discussion
Our proposed method, MOHBA, leverages trajectory data to better understand multiagent behaviors during and after training. MOHBA assumes no knowledge of agents' underlying training algorithms, does not require access to their hidden states or internal models, and applies even to reward-free settings. Our experiments showcased variety of applications of MOHBA, including the analysis of joint and local agent behaviors, monitoring of behavior emergence throughout training, discovery of behavioral concepts associated with certain domain criteria, and disentanglement of third-party- labeled behaviors from open-source policies such those for OpenAl hide-and-seck [10].
While we believe our approach is an important step in terms of increasing the understanding of multiagent systems, there are several limitations and potential societal impacts of note. One limitation related to the collection of a dataset of agent behaviors. Understanding emergent behaviors throughout MARL training using MOHBA requires the storage of large-scale trajectory datasets, which could potentially take lot of storage and compute power to generate. It would be interesting to consider uollow-uss to our model that learn behavioral clusters in streaming fashion, thus building knowledge of behaviors over time and permitting older data to be discarded throughout agent training. Moreover, in our datasets, we collected trajectories at uniform intervals throughout original MARL training. However, it might be interesting to consider a non-uniform collection scheme, c.g., collecting trajectories only when they detected to diverge from behavioral clusters of pre-trained MOHBA model. Finally, it might be interesting to enable the agents to first learn joint behaviors / 'skills' using the proposed method, and leverage them transfer to related downstream tasks (potentially more sample-efficiently than training from scratch).
The study of behavioral interactions in multiagent systems can potentially be used for both positive and negative societal applications. For example, such an approach could be used to prevent certain undesirable behaviors by agents that interact with humans (e.g, self-driving cars), but potentially also used by adversaries to predict and exploit other behaviors (e.g., exploiting certain human preferences for harm), or even inadvertently cause harm due to misinterpretation of certain behavioral modes. As such, further research and evaluation will be required prior to deployment of this and related behavioral analysis approaches to human- facing domains. Nonetheless, as agent capabilities continue to grow, our view is that behavioral analysis of multiagent systems will become increasingly important and should complement traditional reward- based performance monitoring.
Acknowledgments and Disclosure of Funding
We thank Asma Ghandeharioun, Meredith Morris, and Kathy Meier-Hellstern for their helpful feedback and support during the paper writing phase.
10

---

References
[1] Gerald Tesauro et al. Temporal difference learning and DD-gammon. Communications of the ACM, 38(3):58-66) 68, 1995.
[2] Murray Campbell, A Joseph Hoane Jr, and Feng-hsiung Hsu. Deep blue. Artificial intelligence, 134(1-2):57-83, 2002.
[3] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mas- tering the game of with deep neural networks and tree search. nature, 529(7587):484 2016.
[4] David Silver, Thomas Hubert, Julian Schrittwisser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1 11011199:11111111 140-1144, 2018.
[5] Noam Brown and Tuomas Sandholm. Superhuman Al for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374):418 424, 2018.
[6] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung. David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft using multi-agent reinforcement learning. Nature, 575(7782):350-354,, 2019.
[7] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota with large scale deep reinforcement learning. arXiv preprint arXiv: 1tt2.::99/06666 2019.
[8] Noam Brown and Tuomas Sandholm. Superhuman AI for multiplayer poker. Science, 365 (6456):885-880, 2019.
[9] Max Jaderberg, Wojciech M Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, al. Human-level performance in 3D multiplayer games with population- basedd reinforcement Iearn- ing. Science, 364(6443):859 (00 865, 2019.
[10] Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In International Conference on Learning Representations, 2020. URL http:t////ppeereeeeee http://ppenrreeeeeeeeffrrrrrmmm id=SkxpxJBKwS
[11] Siqi Liu, Guy Lever, Zhe Wang, Josh Merel, SM Eslami, Daniel Hennes, Wojeiech M Czarnecki, Yuval Tassa, Shayegan Omidshafiei, Abbas Abdolmaleki, et al. From motor control to team play in simulated humanoid football. arXiv preprint arXiv:2105.121 2021.
[12] Thomas McGrath, Andrei Kapishnikov, Nenad Tomasev, Adam Pearce, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik, Acquisition of chess knowledge in alphazero. CoRR, 0tss/2111.9922 2021. URL https://arxii wwwpsrvv.org//eoo abs/ 2011..91111119
[13] Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-awe aware concept-based based explanations in deep neural networks. Advances in Neural Information Processing Systems, 33:20554 20565, 2020.
[14] Christian Schröder de Witt, Bei Peng, Plerre-Alexandre Kamienny, Philip HS Torr, Wendelin Böhmer, and Shimon Whiteson. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. 2020.
[15] Özgür Şimşek and Andrew Barto. Skill characterization based betweenness. Advances in neural information processing systems, 21, 2008
[16] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. In International Conference on Learning Representations, 2018.

---

[17] Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learning Representations, 2018.
[18] Thomas Kipf, Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro Grefenstette, Pushmeet Kohli, and Peter Battaglia. Compile: Compositional imitation learning and execution. In International Conference on Machine Learning, pages 3418 3428. PMLR, 2019.
Edward
[19] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman, Dynamics- aware unsupervised discovery of skills. In International Conference on Learning Representa- fions, 2019.
[20] Leonard Hasenclever, Fabio Pardo, Raia Hadsell, Nicolas Heess, and Josh Merel. Comic: Complementary task learning & mimicry for reusable skills. In International Conference Machine Learning, pages 4105-4115. PMLR, 2020.
[21] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pages 1113-1132. PMLR, 2020.
[22] Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinchart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In International Conference on Learning Representations, 2020.
[23] Tanmay Shankar and Abhinav Gupta. Learning robot skills with temporal variational inference.
In International Conference on Machine Learning, pages 8624 8633. PMLR, 2020.
[24] Víctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torrs. Explore, discover and learn: Unsupervised discovery of ctvterio skills. In International Conference Machine Learning, pages 1317-1327, PMLR, 2020.
[25] Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning, pages 188-204. PMLR, 2021.
[26] Kate Baumli, David Warde-Farley, Steven Hansen, and Volodymyr Mnih. Relative variational intrinsic control. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6732-6744 2021.
[27] Valentin Villecroze, Harry J Braviner, Panteha Naderian, Chris J Maddison, and Gabriel Loaiza- Ganem. Bayesian nonparametrics for offline skill discovery. arXiv preprint arXiv:2202.04655. 2022.
[28] Yifeng Zhu, Peter Stone, and Yuke Zhu. Bottom-up skill discovery from unsegmented demon- strations for long-horizon robot manipulation. IEEE Robotics and Automation Letters, 2022.
(29) Tanmay Shankar, Shubham Tulsiani, Lerrel Pinto, and Abhinav Gupta. Discovering motor programs by recomposing demonstrations. In International Conference on Learning Represen- tations, 2019.
[30] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. In International Conference on Learning Representations, 2020.
[31] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2): 181-211, 1999.
[32] Aravind Srinivas, Ramnandan Krishnamurthy, Peeyush Kumar, and Balaraman Ravindran. Option discovery in hierarchical reinforcement learning using spatio-temporal clustering. arXiv preprint arXiv: 16f5::66000053533 2016.
12

---

[33] Peter Henderson, Wei-Di Chang. Puerre-Luc Bacon, David Meger, Joelle Pineau, and Doina Precup. Optiongan: Learning joint reward-policy options using generative adversarial inverse reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[34] Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arxiiv::00) 10299, 2018.
[35] André Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Daniel Toyama, Shibl Mourad, David Silver, Doina Precup, al. The option keyboard: Combining skills in reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
[36] Arjun Manoharan, Rahul Ramesh, and Balaraman Ravindran. Option encoder: A framework for discovering policy basis in reinforcement learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 509-50999 Springer, 2020.
[37] Jaekyeom Kim, Seohong Park, and Gunhee Kim. Unsupervised skill discovery with bottleneck option learning. In International Conference on Machine Learning, pages 5572-5582. PMLR, 2021.
[38] Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. International Conference on Machine Learning, pages 7204-7213. PMLR, 2021.
[39] Jack Parker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Ef- fective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems, 33: 1850018000,, 2020.
[40] Zihan Zhou, Wei Fu, Bingliang Zhang, and Yi Wu. Continuously discovering novel strategies via reward-switching policy optimization. International Conference on Learning Representations, 2021.
[41] Justin K Pugh, Lisa Soros, and Kenneth O Stanley, Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, page 40, 2016.
[42] Zhenggang Tang, Chao Yu, Boyuan Chen, Huazhe Xu, Xiaolong Wang. Fei Fang, Simon Du, Yu Wang, and Yi Wu. Discovering diverse multi-agent strategic behavior via reward randomization. arXiv preprint arXiv:2103.04556 2021.
[43] Yuhang Song. Jianyi Wang, Thomas Lukasiewicz, Zhenghua Xu, and Mai Xu, Diversity-driven extensible hierarchical reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 4992-4999, 2019.
[44] Zhang- Wei Hong, Tzu- Yun Shann, Shih-Yang Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun-Y Yi Lee. Diversity-drine driven exploration strategy for deep reinforcement learning. Advances in neural information processing systems, 31, 2018
[45] Tanmay Gangwani, Qiang Liu, and Jian Peng. Learning self-imitating diverse policies. In International Conference on Learning Representutions, 2018
[46] Andrew Cohen, Xingye Qiao, Lei Yu, Elliot Way, and Xiangrong Tong. Diverse exploration via conjugate policies for policy gradient methods. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages
34
[47] Youngwoon Lee, Jingyun Yang, and Joseph J Lim. Learning to coordinate manipulation skills via skill behavior diversification. In International Conference Learning Representations, 2019.
[48] Jiachen Yang, Igor Borovikov, and Hongyuan Zha. Hierarchical cooperative multi-agent the 19th International Conference on Autonomous Agents and MultiAgent Systems, pages 1566-1574, 2020.
reinforcement learning with skill discovery. In Proceedings the
13

---

[49] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. In International Conference on Learning Representations, 2020.
[50] Hangyu Mao, Wulong Liu, Jianye Hao, Jun Luo, Dong Li, Zhengchao Zhang, Jun Wang. and Zhen Xiao. Neighborhood cognition consistent multi-agent reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7219-7226, 2020.
[51] Michael Bradley Johanson, Edward Hughes, Finbarr Timbers, and Joel Z. Leibo. Emergent bartering behaviour in multi-agent reinforcement learning, 2022. URL hattts:///aaatt org/ 0bs/2205.000
[52] Kris Cao, Angeliki Lazaridou, Mare Lanctot, Joel Z Leibo, Karl Tuyls, and Stephen Clark. Emergent communication through negotiation. In International Conference on Learning Representations, 2018.
[53] Michael Noukhovitch, Travis LaCroix, Angeliki Lazaridou, and Aaron Courville. Emergent communication under competition. In Proceedings of the 20th International Conference Autonomous Agents and MultiAgent Systems, pages 974-982, 2021.
[54] Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Heess, and Thore Grae- pel. Emergent coordination through competition. In International Conference on Learning Representations, 2018.
[55] Trapit Bansal, Jakub Pachocki, Szymon Sidor, Iya Sutskever, and Igor Mordatch. Emergent complexity via multi-agent competition. In International Conference on Learning Representa- tions, 2018.
[56] Alexandre Heuillet, Fabien Couthouis, and Natalia Díaz-Rodríguez Explainability in deep reinforcement learning. Knowledge-Baud Bassed Systems, 214:106685, 2021.
[57] Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaudhuri. Programmatically interpretable reinforcement learning. In International Conference on Machine Learning, pages 5045-5054. PMLR, 2018.
[58] Pedro Sequeira and Melinda Gervasio. Interestingness elements for explainable reinforcement learning: Understanding agents' capabilities and limitations. Artificial Intelligence, 288: 103367, 2020.
[59] Tom Zahavy, Nir Ben-Zrihem, and Shie Mannor. Graying the black box: Understanding dqns. In International Conference on Machine Learning, pages 1899-1908. PMLR, 2016.
[60] Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept- based explanations. Advances in Newral Information Processing Systems, 32, 2019.
[61] David Bau, Lun Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, 2020 ISSN 0027-8424. doi: 10. (tt.////a..1907351111 https://wwy pnas ottp://ootea//eaall/////////////
URL
[62] Asma Ghandeharioun, Been Kim, Chun-Liang Li, Brendan Jou, Brian Eoff, and Rosalind W. Picard. DISSECT: disentangled simultancous explanations via concept traversals. CoRR, abs/2105.15164, 2021. URL htttppp:/aaa..i..or//aa///////// 15164
[63] Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how BERT works. CoRR, abs/2002. 2020. URL /tttss///arxivv wrg. org/abs/ abs/
2002. 12327
[64] John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pages 4129 4138, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 18155//v1/N1914411 1419. URL. https: //c 1 aclanthology
org/N19 2tg/N19-1443 1419
14

---

[65] Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoen- coder: Learning disentangled representations from grouped observations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[66]
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder, Advances in Neural Information Processing Systems, 33:19667-1967, 2020.
[67] Dinghan Shen, Asli Celikyilmaz, Yizhe Zhang, Liqun Chen, Xin Wang, Jianfeng Gao, and Lawrence Carin. Towards generating long and coherent text with multi-level latent variable models. In ACL 2019-57th Annual Meeting of the Association for Computational Linguistics. Proceedings of the Conference, pages 2079-2089. Association for Computational Linguistics (ACL), 2020.
[68] Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. In International Conference on Learning Representattonss 2020.
[69] Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. In International Conference on Machine Learning. pages 1851-1860. PMLR, 2018
[70] Dushyant Rao, Fereshteh Sadeghi, Leonard Hasenclever, Markus Wulfmeier, Martina Zambelli, Giulia Vezzani, Dhruva Tirumala, Yusuf Aytar, Josh Merel, Nicolas Heess, et al. Learning transferable motor skills with hierarchical latent mixture policies. In Deep RL Workshop NeurlPS 2021, 2021.
[71] Frans A Oliehoek and Christopher Amato. A concise introduction to decentralized POMDPs. Springer, 2016.
[72] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Machine Learning. 2017.
[73] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference Learning Representations, 2014.
[74] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference machine learning, pages 1278-1286. PMLR, 2014.
[75] Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmalcki, Albin Cassirer, Fan Yang. Kate Baumli, Sarah Henderson, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979 2020.
[76] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference machine learning, pages 1587-1596. PMLR, 2018
[77] Sabela Ramos, Sertan Girgin, Léonard Hussenot, Damien Vincent, Hanna Yakubovich, Daniel Toyama, Anita Gergely, Piotr Stanezyk, Raphael Marinier, Jeremiah Harmsen, Olivier Pietquin, and Nikola Momchev. RLDS: ecosystem to generate, share and use datasets in reinforcement learning. XrXiv preprint arXiv:2111.0276) 2021.
[78] Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: An unsupervised and generative approach to clustering. Proceedings of the Twenty-Siste International Joint Conference on Artificial Intelligence, IJCAI-17, pages 1965-1972, 2017.
[79] Filippos Christianos, Georgios Papoudakis, Muhammad A Rahman, and Stefano V Albrecht. Scaling multi-agent reinforcement learning with selective parameter sharing. In International Conference Machine Learning, pages 1989-1998. PMLR, 2021.
15

---

[80] Trevor Hastie. Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining. inference, and prediction, volume Springer, 2009.
[81] OpenAl. Multiagent emergence environments (github codebase). https://github.c com/
2019.
[82] Xiaodong Liu Jianfeng Gao Asli Celikyilmaz Lawrence Carin Hao Fu, Chunyuan Li. Cyclical annealing schedule: A simple approach to mitigating KL vanishing. In NAACL, 2019.
[83] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
[84] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
[85] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXtv:2004.072119 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We include these in the Discus- sions section.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We include these in the discussions section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] These are provided in the main paper and the proofs included in the appendíx.
(b) Did you include complete proofs of all theoretical results? [Yes] Proofs are provided in the appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as a URL)? [Yes] We provide
details for experiment reproducibility in Appendix A.2 We also include model high- level code in Appendix A.7
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We provide all training details in Appendix
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We provide all computational details in Appendix A.2
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A] Our work does not existing assets.
(b) Did you mention the license of the assets? [N/A] Our work does not use existing assets.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A] Our work does not use existing assets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] Our work does not use existing assets.
16

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] Our work does not crowdsourced human subject data.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A] Our work does not crowdsourced or human subject data.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A] Our work does not crowdsoureed human subject data.
17


Reviews:
[{"review_id": "kQl8Idjkvv0", "reviewer": "Reviewer_Nrc6", "summary": "The authors develop a hierarchical clustering technique for sequential data and apply it to RL agents by optimizing a variational objective. The method is tested on simple multi-agent environments including a MuJoCo example. \n", "questions": "What is the sample complexity of this approach? How many traces of behavior are needed to get good behavioral clusters?\n\nIs access and control of the environment required or does one just need traces of behavior? \n\nHow will this algorithm perform in environments that have stochastic transitions?\n\nHow does this model differ from a generic model for clustering sequences? Is there something in the model structure that makes it more likely to detect behavioral signatures as opposed to physical signatures? \n", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is very well written and easy to follow. The methods are explained clearly and each experiment and example is well motivated. All results are discussed and the figures are clear and well explained. \n\nThe problem is an important and impactful one and progress here would be exciting to many in the MARL community. \n\nHowever, the current work suffers from the following weaknesses:\n\nToo much time spent on the hill-climbing problem. While this problem is very intuitive, it has almost no distinctly multiagent content (interaction between the agents) and thus makes it hard to see how this approach will handle more interesting kinds of multi-agent behavior. For example, the authors allude to \"cooperate vs. compete\" but there is no analysis of sophisticated behavior. \n\nThe evaluation of the method needs significantly more work. To raise my score, I would like to see is the analysis of an existing agent (not trained by the authors) that demonstrates some kind of sophisticated behavior and to have the model presented here discover some kind of novel insight into what is going on. It would be particularly interesting for example to analyze a complete system with its ablations to give finer-grained insight into why an ablation doesn't work as well. \n\nUPDATE: Scores raised in response to new experiments and the answers to my questions. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_npGKEaYLfc", "reviewer": "Reviewer_zatT", "summary": "This paper introduces a method for discovery of behavior clusters (both individual-agent-level and joint-agent-level) in multiagent domains with the given offline trajectory, without assumption about the underlying learning algorithms and world models. Specifically, the method uses joint-level latent behavior factor z and individual-level latent behavior factor zi with the assumption that each agent’s latent-conditioned policy is conditionally-independent z given zi, and drive a variational lower bound to learn z and zi by maximizing the probabilities of the given offline trajectory probabilities. The authors illustrate the effectiveness of their method for enabling the coupled understanding of behaviors at the joint and local agent level, detection of behavior changepoints throughout training, discovery of core behavioral concepts (e.g., those that facilitate higher returns), and demonstrate the approach's scalability to a high-dimensional multiagent MuJoCo control domain.\n", "questions": "To be honest, I am not an expert of multiagent behavior analysis. I try to give some suggestions as follows.\n\n* Could the authors give some reasons for the assumption that \"each agent’s latent-conditioned policy is conditionally-independent z given zi\"?\n* I think it will be better to give more clear description about how to draw the figures of z and zi.\n* I found that NCC-MARL [1] also proposes latent behavior analysis, and it shows that the consistent latent cognition of individual agents will result in good joint cooperation of all agents. I think this is also a representive paper that should be cited.\n\n\n[1] Neighborhood cognition consistent multi-agent reinforcement learning. AAAI 2020.", "limitations": "-", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Good Research Point. The agent behavior analysis is less studied compared to the agent performance enhancement as far as I konw. I think the proposed method (which is less reward-centric) is very necessary for the multi-agent community.\n* Clear Method and Good Paper Writting. The latent-conditioned trajectory generation process that the authors use to learn multiagent behavior clusters is very clear and easy to understand. \n* Sufficient Experiments. Including both coupled analysis and indepedent analysis of both joint and local behavioral clusters analysis, emergent behavior analysis and the analysis of scalability to high-dimensional domains. And I also found that the figures is clear to understand.\n\nWeaknesses\n* I do not know whether the assumption that \"each agent’s latent-conditioned policy is conditionally-independent z given zi\" is true for real world scenarios? I understand the authors try to simplify the problem formulation, but I have some concerns for this assumption. For example (the same example used in the paper), if z encodes (at a high level) whether agents were cooperating or competing, I think the agent’s policy should certainly be dependent on z, no matter what meanings zi may represent. \n* Since the paper focus on behavior analysis (ie, the analysis of z and zi), I think the authors should give clear description about how to draw the figures of z and zi, but some figures do not have such descriptions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_Y7vd-Fd9pC", "reviewer": "Reviewer_do8M", "summary": "This paper studies an interesting question: how to model different behaviors from an offline multi-agent RL dataset. The authors propose a hierarchical modeling approach and present empirical experiments over a 2-D particle world navigation scenario and a MuJoCo HalfCheetah control task. \n\n\n++++++++ post-rebuttal +++++++++\nI think the updated draft is clearly a much stronger one as a solid work. I have updated my score accordingly.", "questions": "I'm curious about answers to the following questions:\n\n1. How does the proposed method perform if the dataset includes a large portion of sub-optimal and random trajectories?\n2. How does the proposed method perform on more complex domains with more agents?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\nI personally like the topic of this paper and I strongly agree that the analysis of RL applications should be beyond rewards, i.e., considering the behaviors of the learned strategies. The use of a hierarchical generative model is intuitive (although it isn't surprising). I do feel this draft has the potential to become a much stronger paper if the authors can step further in this research direction. \nRegarding the writing, the method and motivation part is clearly written but I don't think many of the arguments from this paper are well justified. \n\n### Weakness:\nThe current content looks a bit preliminary. There are two major issues: (1) the technical content is not well justified. (2) the experiments are a bit too toy for an analysis paper. Please see my concerns below.\n\n1. **Regarding the technical content:** The use of the generative model in modeling agent behaviors isn't new at all. Lots of techniques, such as [GAN-style](https://arxiv.org/abs/1802.06070), [VAE](https://arxiv.org/abs/2010.01523), and [diffusion model](https://diffusion-planning.github.io/), have been widely adopted. The authors choose a VAE-style generative modeling approach, which is technically standard. The interesting part appears to be the use of two latent variables, i.e., one for joint behavior and the other for agents. But I don't really get why this hierarchical design is necessary. At least for the motivating example (the hill climb game), a flattened VAE model is clearly sufficient. Even in the experiment part (Table 2), I cannot see any critical advantages ------ yes, it is true that the bi-level model has a higher prediction accuracy but why do we really care about prediction accuracy? From the motivation of the paper, the paper focuses on behavior analysis and aims to discover different behavior modes, which are well captured by the ICTD metric. I couldn't see a big difference between the two methods under the ICTD metric. Moreover, at least from the current content of the paper, I think both the half cheetah and the MPE motivating example can be perfectly analyzed by the flattened model for the purpose of discovering strategy modes. I do agree that the bi-level model discovers some agent-wise behavior modes. But so what? If we are really interested in agent-wise behavior analysis, we can simply run another flattened VAE model for each agent over its own trajectories and that's all set. To sum up, the technical content of the paper looks so arbitrary. Some further analysis and research would be required for a solid work. \n2. **Regarding the experiment section:** First of all, the experiment section is way too toy for an analysis paper. It is okay to take the hill-climb environment as a motivating example. But I would expect much more complex domains. The MoJoCo control domain is a good example. But why only the simplest scenario with just two agents is considered? Why not consider more scenarios with much more agents? The authors talk about the hide-and-seek project in the introduction section repeatedly. So why not run the model over the hide-and-seek project? The learned policies are released. You don't even need to train the policies. Moreover, the author claimed in the introduction section that the proposed method is reward-agnostic and is generic to offline RL settings. However, the dataset construction doesn't really follow the offline RL convention. The dataset is simply constructed by running MARL training repeatedly. Note that the MARL algorithms are optimizing the environment reward, which implies all the behavior data are leaning towards high rewards. What if some sub-optimal or even random trajectories are included in the dataset? Can the model still work? We would like to point out that including sub-optimal and random trajectories is a common practice in [offline RL literature](https://arxiv.org/abs/2004.07219). \n\nIn general, I would suggest the authors re-think the emphasis of the paper. What is really the main contribution? Is it the hierarchical modeling technique? Or the behavior analysis problem itself? If the paper focuses on the former one (bi-level modeling), the motivating example should make people understand why a bi-level model is necessary instead of a flattened model. The experiments should also focus more on the bi-level v.s. flatten part. If the authors would like to focus more on a new problem, then we would expect much more experiments and analysis, hopfully including novel insights or findings beyond cheatah skills (there are so many papers showing different emergent motion skills over MuJoCo domains).\n\n\n### Minor Issues\n1. **A missing line of research**: I would suggest the author also take into account the works on RL diversity, which focuses on learning a collection of policies covering different behavior modes. These works typically use a discriminative approach to model behavior modes, i.e., defining a distance metric or a heuristic density, rather than a generative model. I would like to point out that a distance metric can be perfectly used for clustering although these methods simply optimize the policies to ensure diversity (remark: optimizing diverse policies is a much harder problem than simply clustering over a fixed dataset.) Representative metrics include [trajectoriy diversity](https://proceedings.mlr.press/v139/lupu21a.html), [DvD](https://github.com/jparkerholder/DvD_ES), [cross-entropy](https://openreview.net/forum?id=hcQHRHKfN_), and [quality diversity](https://www.frontiersin.org/articles/10.3389/frobt.2016.00040/full).. I would suggest the authors include this line of research and, if possible, discuss why a generative modeling approach is preferred here? E.g., why cannot we just design a quality diversity function and run a t-SNE analysis or just run a kernel-version of GMM?\n2. **Experiments on _``user-desired characteristics''_**. I don't really get the motivation of this part (line 239). Why does \"user-desired\" necessarily mean \"high rewards\"? Also, as I mentioned above, the dataset is constructed by running MARL algorithms, so many of the trajectories are indeed having high rewards. I wouldn't be surprised that high-reward concepts can be learned. But what if more random data are introduced? What if most data are random? In addition, I couldn't understand why a train-validation split makes any sense here. Why do we really care about testing accuracy? The authors state that they are following [13]. However, I would point out that [13] is in the setting of supervised learning, which is different from RL. Why could all the evaluation settings of SL be simply generalized to RL? It is natural for supervised learning methods to focus on loss function/prediction accuracy. But why do we care about the loss in RL? Note that even the title of the paper claims \"beyond rewards\". Should we measure something beyond prediction accuracy? I would suggest the author think a bit further to present more convincing experiments. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CLceFnhD9ZE", "reviewer": "Reviewer_c5wJ", "summary": "The paper focuses on behavior analysis of multiagent reinforcement learning. The approach is to discover behaviors at the local level and joint level separately from offline data based on an information-theoretical objective. The authors did an empirical analysis on two simple domains as well as the multiagent mujoco domain. The results show some interesting multiagent behaviors visualized in the learned latent space.", "questions": "For the 3-agent hill climbing environment, it seems to me that the goal of the domain is more like to train the three agents independently (reach one of the three goal positions) and there's no multiagent cooperation/interactions happening.\n\nI don't understand the LSTM baseline on page 7. Is it more like an ablation study of the network structure?\n\nIt would be nice to see some ablation study on the KL weighting term $\\beta$ in the algorithm's main training objective. Besides, how will the size of the latent vector $z$ affect the overall results?\n\nAnother suggestion (probably for future work) is that maybe the authors can try to let the agent learn some transferable multiagent \"skills\" using the proposed method and see if they can be reused to solve some other similar tasks.\n", "limitations": "See above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The high-level idea of analyzing multiagent behaviors is appealing and I think is an important direction for multiagent RL research. The authors propose to learn hierarchical latent embeddings of the agents' behaviors by focusing on the local and joint level separately. The proposed approach is simple and easy to follow. The empirical analysis is thorough by first analyzing the two levels of clusters independently and then their relationships.\n\nI think one of the weaknesses of the submission is lack of empirical analysis on more complex domains that requires structured cooperation/competition strategy, like simulated football game. It would be interesting to see how different categories of skills (e.g. dribbling the ball) look like in latent space. I'm also a little doubtful whether the information-theoretical training objective will work properly for such domains. Besides, the first domain of empirical analysis does not seem to be a typical multiagent environment with interactions between agents.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept