Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors develop a hierarchical clustering technique for sequential data and apply it to RL agents by optimizing a variational objective. The method is tested on simple multi-agent environments including a MuJoCo example. 

strengths_and_weaknesses: The paper is very well written and easy to follow. The methods are explained clearly and each experiment and example is well motivated. All results are discussed and the figures are clear and well explained. 

The problem is an important and impactful one and progress here would be exciting to many in the MARL community. 

However, the current work suffers from the following weaknesses:

Too much time spent on the hill-climbing problem. While this problem is very intuitive, it has almost no distinctly multiagent content (interaction between the agents) and thus makes it hard to see how this approach will handle more interesting kinds of multi-agent behavior. For example, the authors allude to "cooperate vs. compete" but there is no analysis of sophisticated behavior. 

The evaluation of the method needs significantly more work. To raise my score, I would like to see is the analysis of an existing agent (not trained by the authors) that demonstrates some kind of sophisticated behavior and to have the model presented here discover some kind of novel insight into what is going on. It would be particularly interesting for example to analyze a complete system with its ablations to give finer-grained insight into why an ablation doesn't work as well. 

UPDATE: Scores raised in response to new experiments and the answers to my questions. 

questions: What is the sample complexity of this approach? How many traces of behavior are needed to get good behavioral clusters?

Is access and control of the environment required or does one just need traces of behavior? 

How will this algorithm perform in environments that have stochastic transitions?

How does this model differ from a generic model for clustering sequences? Is there something in the model structure that makes it more likely to detect behavioral signatures as opposed to physical signatures? 

limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper focuses on behavior analysis of multiagent reinforcement learning. The approach is to discover behaviors at the local level and joint level separately from offline data based on an information-theoretical objective. The authors did an empirical analysis on two simple domains as well as the multiagent mujoco domain. The results show some interesting multiagent behaviors visualized in the learned latent space.
strengths_and_weaknesses: The high-level idea of analyzing multiagent behaviors is appealing and I think is an important direction for multiagent RL research. The authors propose to learn hierarchical latent embeddings of the agents' behaviors by focusing on the local and joint level separately. The proposed approach is simple and easy to follow. The empirical analysis is thorough by first analyzing the two levels of clusters independently and then their relationships.

I think one of the weaknesses of the submission is lack of empirical analysis on more complex domains that requires structured cooperation/competition strategy, like simulated football game. It would be interesting to see how different categories of skills (e.g. dribbling the ball) look like in latent space. I'm also a little doubtful whether the information-theoretical training objective will work properly for such domains. Besides, the first domain of empirical analysis does not seem to be a typical multiagent environment with interactions between agents.
questions: For the 3-agent hill climbing environment, it seems to me that the goal of the domain is more like to train the three agents independently (reach one of the three goal positions) and there's no multiagent cooperation/interactions happening.

I don't understand the LSTM baseline on page 7. Is it more like an ablation study of the network structure?

It would be nice to see some ablation study on the KL weighting term $\beta$ in the algorithm's main training objective. Besides, how will the size of the latent vector $z$ affect the overall results?

Another suggestion (probably for future work) is that maybe the authors can try to let the agent learn some transferable multiagent "skills" using the proposed method and see if they can be reused to solve some other similar tasks.

limitations: See above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces a method for discovery of behavior clusters (both individual-agent-level and joint-agent-level) in multiagent domains with the given offline trajectory, without assumption about the underlying learning algorithms and world models. Specifically, the method uses joint-level latent behavior factor z and individual-level latent behavior factor zi with the assumption that each agent’s latent-conditioned policy is conditionally-independent z given zi, and drive a variational lower bound to learn z and zi by maximizing the probabilities of the given offline trajectory probabilities. The authors illustrate the effectiveness of their method for enabling the coupled understanding of behaviors at the joint and local agent level, detection of behavior changepoints throughout training, discovery of core behavioral concepts (e.g., those that facilitate higher returns), and demonstrate the approach's scalability to a high-dimensional multiagent MuJoCo control domain.

strengths_and_weaknesses: Strengths
* Good Research Point. The agent behavior analysis is less studied compared to the agent performance enhancement as far as I konw. I think the proposed method (which is less reward-centric) is very necessary for the multi-agent community.
* Clear Method and Good Paper Writting. The latent-conditioned trajectory generation process that the authors use to learn multiagent behavior clusters is very clear and easy to understand. 
* Sufficient Experiments. Including both coupled analysis and indepedent analysis of both joint and local behavioral clusters analysis, emergent behavior analysis and the analysis of scalability to high-dimensional domains. And I also found that the figures is clear to understand.

Weaknesses
* I do not know whether the assumption that "each agent’s latent-conditioned policy is conditionally-independent z given zi" is true for real world scenarios? I understand the authors try to simplify the problem formulation, but I have some concerns for this assumption. For example (the same example used in the paper), if z encodes (at a high level) whether agents were cooperating or competing, I think the agent’s policy should certainly be dependent on z, no matter what meanings zi may represent. 
* Since the paper focus on behavior analysis (ie, the analysis of z and zi), I think the authors should give clear description about how to draw the figures of z and zi, but some figures do not have such descriptions.
questions: To be honest, I am not an expert of multiagent behavior analysis. I try to give some suggestions as follows.

* Could the authors give some reasons for the assumption that "each agent’s latent-conditioned policy is conditionally-independent z given zi"?
* I think it will be better to give more clear description about how to draw the figures of z and zi.
* I found that NCC-MARL [1] also proposes latent behavior analysis, and it shows that the consistent latent cognition of individual agents will result in good joint cooperation of all agents. I think this is also a representive paper that should be cited.


[1] Neighborhood cognition consistent multi-agent reinforcement learning. AAAI 2020.
limitations: -
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies an interesting question: how to model different behaviors from an offline multi-agent RL dataset. The authors propose a hierarchical modeling approach and present empirical experiments over a 2-D particle world navigation scenario and a MuJoCo HalfCheetah control task. 


++++++++ post-rebuttal +++++++++
I think the updated draft is clearly a much stronger one as a solid work. I have updated my score accordingly.
strengths_and_weaknesses: ### Strengths:
I personally like the topic of this paper and I strongly agree that the analysis of RL applications should be beyond rewards, i.e., considering the behaviors of the learned strategies. The use of a hierarchical generative model is intuitive (although it isn't surprising). I do feel this draft has the potential to become a much stronger paper if the authors can step further in this research direction. 
Regarding the writing, the method and motivation part is clearly written but I don't think many of the arguments from this paper are well justified. 

### Weakness:
The current content looks a bit preliminary. There are two major issues: (1) the technical content is not well justified. (2) the experiments are a bit too toy for an analysis paper. Please see my concerns below.

1. **Regarding the technical content:** The use of the generative model in modeling agent behaviors isn't new at all. Lots of techniques, such as [GAN-style](https://arxiv.org/abs/1802.06070), [VAE](https://arxiv.org/abs/2010.01523), and [diffusion model](https://diffusion-planning.github.io/), have been widely adopted. The authors choose a VAE-style generative modeling approach, which is technically standard. The interesting part appears to be the use of two latent variables, i.e., one for joint behavior and the other for agents. But I don't really get why this hierarchical design is necessary. At least for the motivating example (the hill climb game), a flattened VAE model is clearly sufficient. Even in the experiment part (Table 2), I cannot see any critical advantages ------ yes, it is true that the bi-level model has a higher prediction accuracy but why do we really care about prediction accuracy? From the motivation of the paper, the paper focuses on behavior analysis and aims to discover different behavior modes, which are well captured by the ICTD metric. I couldn't see a big difference between the two methods under the ICTD metric. Moreover, at least from the current content of the paper, I think both the half cheetah and the MPE motivating example can be perfectly analyzed by the flattened model for the purpose of discovering strategy modes. I do agree that the bi-level model discovers some agent-wise behavior modes. But so what? If we are really interested in agent-wise behavior analysis, we can simply run another flattened VAE model for each agent over its own trajectories and that's all set. To sum up, the technical content of the paper looks so arbitrary. Some further analysis and research would be required for a solid work. 
2. **Regarding the experiment section:** First of all, the experiment section is way too toy for an analysis paper. It is okay to take the hill-climb environment as a motivating example. But I would expect much more complex domains. The MoJoCo control domain is a good example. But why only the simplest scenario with just two agents is considered? Why not consider more scenarios with much more agents? The authors talk about the hide-and-seek project in the introduction section repeatedly. So why not run the model over the hide-and-seek project? The learned policies are released. You don't even need to train the policies. Moreover, the author claimed in the introduction section that the proposed method is reward-agnostic and is generic to offline RL settings. However, the dataset construction doesn't really follow the offline RL convention. The dataset is simply constructed by running MARL training repeatedly. Note that the MARL algorithms are optimizing the environment reward, which implies all the behavior data are leaning towards high rewards. What if some sub-optimal or even random trajectories are included in the dataset? Can the model still work? We would like to point out that including sub-optimal and random trajectories is a common practice in [offline RL literature](https://arxiv.org/abs/2004.07219). 

In general, I would suggest the authors re-think the emphasis of the paper. What is really the main contribution? Is it the hierarchical modeling technique? Or the behavior analysis problem itself? If the paper focuses on the former one (bi-level modeling), the motivating example should make people understand why a bi-level model is necessary instead of a flattened model. The experiments should also focus more on the bi-level v.s. flatten part. If the authors would like to focus more on a new problem, then we would expect much more experiments and analysis, hopfully including novel insights or findings beyond cheatah skills (there are so many papers showing different emergent motion skills over MuJoCo domains).


### Minor Issues
1. **A missing line of research**: I would suggest the author also take into account the works on RL diversity, which focuses on learning a collection of policies covering different behavior modes. These works typically use a discriminative approach to model behavior modes, i.e., defining a distance metric or a heuristic density, rather than a generative model. I would like to point out that a distance metric can be perfectly used for clustering although these methods simply optimize the policies to ensure diversity (remark: optimizing diverse policies is a much harder problem than simply clustering over a fixed dataset.) Representative metrics include [trajectoriy diversity](https://proceedings.mlr.press/v139/lupu21a.html), [DvD](https://github.com/jparkerholder/DvD_ES), [cross-entropy](https://openreview.net/forum?id=hcQHRHKfN_), and [quality diversity](https://www.frontiersin.org/articles/10.3389/frobt.2016.00040/full).. I would suggest the authors include this line of research and, if possible, discuss why a generative modeling approach is preferred here? E.g., why cannot we just design a quality diversity function and run a t-SNE analysis or just run a kernel-version of GMM?
2. **Experiments on _``user-desired characteristics''_**. I don't really get the motivation of this part (line 239). Why does "user-desired" necessarily mean "high rewards"? Also, as I mentioned above, the dataset is constructed by running MARL algorithms, so many of the trajectories are indeed having high rewards. I wouldn't be surprised that high-reward concepts can be learned. But what if more random data are introduced? What if most data are random? In addition, I couldn't understand why a train-validation split makes any sense here. Why do we really care about testing accuracy? The authors state that they are following [13]. However, I would point out that [13] is in the setting of supervised learning, which is different from RL. Why could all the evaluation settings of SL be simply generalized to RL? It is natural for supervised learning methods to focus on loss function/prediction accuracy. But why do we care about the loss in RL? Note that even the title of the paper claims "beyond rewards". Should we measure something beyond prediction accuracy? I would suggest the author think a bit further to present more convincing experiments. 
questions: I'm curious about answers to the following questions:

1. How does the proposed method perform if the dataset includes a large portion of sub-optimal and random trajectories?
2. How does the proposed method perform on more complex domains with more agents?
limitations: N/A
review_title: null

Meta-review: metareview: The reviewers agreed this paper studies an interesting problem and provide an interesting contribution to the multi-agent community. We urge the authors to include the added experiments and information (e.g., suggested related work) into the main text. 
confidence: Certain
Decision: Accept