Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose "local information agent modeling" (LIAM), i.e. using a supervised auxiliary predictive task to help learn a representation of the history which is useful to predict other agent's behaviors, and which is then used as a basis for decision making.  The task encoded by LIAM involves predicting the modeled agents' trajectories based on the controlled agent's trajectory.  LIAM is presented both as a general idea, and as a specific implementation which uses encoder-decoder models (controlled agent trajectory -> latent representation -> modeled agent trajectory).  The auxiliary learning task can be performed with supervision in an offline learning framework, where the privileged information of all agents is available.  During execution, the fully trained encoder can be used without requiring the privileged information.

The experiment section involves 3 multi-agent environments, and 5 baselines closely related to LIAM.  On top of the standard performance-based RL results, the authors include a) a model evaluation section where they analyze the learned latent representations and their ability to encode important information, and b) an ablation study which shows the empirical importance of multiple individual components of their specific LIAM implementation.
main_review: EDIT:  The author's response addresses my primary concerns regarding fairness, so I've updated my score based on their response.  (more updates may come based on the committee discussion).

---

The work seems like a solid advancement in the field of multi-agent control, and the authors have positioned it well in the current literature.  The (offline self-)supervised task encoded by LIAM is fairly simple and perhaps in retrospect an obvious one (which I consider a strength!).  While it is a variant of other similar auxiliary learning tasks, LIAM does address significant drawbacks made by prior work (e.g. full online observability, or (partial) knowledge of the domain model).  In contrast, LIAM primarily makes use of a single but fairly common assupmtion (two more are stated, but it is not clear to me that they are necessary.  More on this later):  the ability to have offline training in a controlled environment where privileged information is available.  The task encoded by LIAM makes ample sense in the specific context of multi-agent RL problems which *require* some form of interaction between agents.  As a bonus, LIAM makes sense in all sorts of generic stochastic multi-agent games, i.e., cooperative, competitive, or mixed games.  The model evaluation and ablation study sections are very welcome additions which complement well the standard performance-based RL results.

Nonetheless, both LIAM and the evaluation performed by the authors come with some limitations and lingering questions which in my mind need to be addressed.  My current rating is based on the current submission as-is, under my current understanding and the temporary assumption that there will not be an author response;  I will gladly adjust my rating based on the author's response.

In order of importance:

1) Some baseline methods use RNNs which receive observation-action sequences (CBAM), while others also receive the rewards as input (NAM and VariBad).  For some methods, it is unclear whether rewards are used or not (LIAM,FIAM,CARL).  I hope this is simply a mistake in the descriptions of the methods, but will pose my question as if the descriptions were accurate.  This setup is very problematic for multiple reasons:

1.a) Such an inconsistency between the information available to each method/baseline wholly undermines the empirical evaluation.  All methods/baselines should access the same information (in principle, either all should observe the rewards, or all should not observe the rewards).  In the general case, I believe that observable rewards are a problem on their own right, as described next.

1.b) "Observable" rewards (which are fed into the RNN and influence the choice of future actions) are IMO extremely problematic in partially observable control problems.  The rewards contain information about the underlying state which are not available from the observations alone.  This not only significantly alters our understanding of the problem, but in some cases may allow the agents to significantly bypass the need to develop information-gathering and communication strategies.  E.g., in the DSL environment, being able to observe the reward encodes the agent distances from their goal, and provides a strong signal to be able to locate the landmarks without the need to communicate.

2) The assumption that other agent policies are taken from a fixed set of stationary policies which does not change over time seems to be a relatively strong one;  however, it is not clear to me that this assupmtion is intrinsically necessary.  Is there any reason why this assumption is made, aside from making the RL and LIAM tasks easier?  A natural idea would be to run evaluations where all agents are concurrently trained using LIAM;  any reason why this evaluation was not performed?

3) Treating the policies and behavior of all other agents as a monolithic joint entity is perhaps the biggest limitation of the current implementation as a whole, and one which I must assume makes LIAM scale poorly to the number of agents.  Given that the evaluation environments involve only 1 or 3 other modeled agents, it certainly is not sufficient to confirm generous statements like "LIAM [...] can model an arbitrary number of agents".  Is it possible to factorize the predictions of modeled agent trajectories?

4) Another limitation is the assumption that the modeled agent policies are reactive, i.e. that the next actions are solely chosen based on the current local observations (rather than the entire history of each agent's respective local observations).  Again, it is not clear to me that this assumption is intrinsically necessary, except for making the action-prediction problem of LIAM simpler.  Any reasons why more general history-based policies were not considered for the modeled agents?

5) The NAM baseline is described as being a "lower performance of LIAM", and I'm not convinced that this is strictly true.  While LIAM does seem like a useful task, there is no reason to believe that it is always strictly better than no auxiliary task at all (as in NAM).  Similarly, I am not convinced that FIAM is indicative "upper performance of LIAM".  Again, LIAM and FIAM are similar but ultimately simply different tasks, one more centered on the ability to predict other agents, and the other on the ability to synthesize the agent's own history.  While both are relevant for decision making, claiming that the information encoded by FIAM is strictly better than that encoded by LIAM needs backing.  I would assume that broadly speaking, LIAM, FIAM, and NAM are stronger in different types of domains.  Is there any further evidence for these statements, or am I maybe reading those statements as more strongly than the authors intended?

6) Why is the decoder architecture factored into the observation and policy reconstruction models, as opposed to a single model which outputs both observation and action?  (aside, perhaps, to facilitate the ablation study?).  Is the observation used in the policy reconstruction model always the one generated by the observation reconstruction model, or do you also use the ground truth observation, akin to teacher forcing?

The submission reads very well, and there are very few minor issues concerning readability:

* l.220:  "than" -> "then".

* l.270:  Confidence interval of what kind of uncertainty measure?  Standard deviation or standard error?  I assume standard deviation, but this should be stated explicitly.

* Fig.4. (and to a lesser extent, Fig.3):  Labels and ticks are too small and should be enlargened.
limitations_and_societal_impact: The submission addresses fairly generic multi-agent RL settings, rather than specific applications.  I do not believe this work warrants any specific statements about societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces the Local Information Agent Modelling (LIAM), which learns a latent representation of the other agents. By using an encoder-decoder structure, the latent inference could be performed only based on the ego agent's local information, without knowing the observations and actions of other agents during the test time. The learned latent state is then used to augment the observation of the ego agent for reinforcement learning.  The experiment results show the advantage of LIAM compared with other baselines tested.
main_review: The contributions of the paper, to my knowledge, are novel. The method proposed is intuitive and well-explained by the authors. The experiment results and discussions deliver convincing evidence on the advantage of the proposed method. The paper is well-written. The problem that the paper aims to address is important since, in many practical multi-agent problems, the actions and observations of other agents are unknown to the ego agent. 

There are some minor concerns about the paper: 
First, simultaneously training the policy and the encoder where the two modules interact with each other may cause training stability issues even the policy is trained under an off-policy RL algorithm. In the paper, the author mentions that they are using different learning rates for the two modules. It would be good if there are some more studies and explanations on this issue. 
Second, in the ablation study, the author compares the different encoders. I think maybe some move ablation studies on the decoder are also important. For example, instead of predicting both the observation and the action, how would it influence the performance of the encoder only predicts one of them? 
Third, the assumption of a fixed finite set of other agents' policies limits the application of the method and seems unnecessary. Some more extensions would be good. 
Fourth, it would be great if the authors could also discuss the scalability of the method.
limitations_and_societal_impact: The limitations discussed by the authors in the appendix are important, and the potential solutions mentioned are reasonable.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors introduce a method for learning in a multi-agent setting that centers around predicting what other agents will do in the given context. Crucially, global information is not assumed (at execution), and so the techniques introduced have a focus on agent perspective in the information used to drive behaviour.

The methods are evaluated on three domains requiring multi-agent consideration (coordination or combativeness), and the results demonstrate the merit in taking the local perspective for these partial-information domains.
main_review: ## Originality

The problem setting is certainly interesting. Assumptions of full observability and perspective of all agents, by all agents, is certainly far too strong of an assumption. This work takes a reasonable step to relax that.

The one area that seems to be lacking from the related work is the growing body on embodied agents. There are a mix of papers at various conferences, but these two workshop series seem directly aligned:
- https://embodied-ai.org/
- https://eyewear-computing.org/EPIC_CVPR21/past-editions


## Quality
The paper is well written, and the quality of the work is good. My biggest complaint on the framing of the solution (which also is an issue with the significance) is the idea that _the perspective of the other agents is available, in full, during training_. Projecting (ala Theory of Mind) during the training process (and evaluation, for that matter) seems to be a far more realistic avenue than the strong assumption that you have access to full agent trajectories when training.

## Clarity
The paper is generally very clear. There are a few things that can be improved, however:

1. Lines 45-47 are confusing, and only made sense after having read the full paper at least once.
2. The plots are often too small to be legible (e.g., Fig 4).
3. You should emphasize from the beginning that this is a multi-agent setting, and from the perspective of one agent. Partial observability exists in the single-agent setting, and your work is more ambitious than just that.
4. The "modelled agent" used throughout is actually capturing several (all) other agents. I would suggest changing the language here, as it feels as though you are only considering 2-player games.


## Significance
There are a fair number of assumptions that come into play with this work, and that affects the significance substantially. I applaud the authors for trying to move to the ego-centric setting, as it is far more realistic than the omnipotent view many multi-agent approaches assume, but the assumption of full observability during training remains a very strong one. It is clear how much this assumption helps, given the poor performance of LIAM-Local (a more realistic setup for what a controlling agent would have access to).

Another one of the assumptions is that the other agents' behaviour depends entirely on the working memory of the controlled agent. There should be elements of what the other agents do/see/experience that influence their behaviour, and in no way is something the controllable agent sees. This is touched upon by the authors as the limitation of needing to be roughly local (so some level of mutual observability is maintained), but manifests as a questionable architecture choice (having the predicted $o_t^{-1}$ based entirely on $z_t$). I get that the other agents' knowledge may be captured somehow in $z_t$, but the only vector for this is through $o_t^{1}$, which feels like a major limitation to me.

The final major limitation is with the complexity of behaviour on the other agents. There is little justification as to how realistic having pre-determined policies might be, and even with just a handful of individual policies or actions availability, the cross-product in action-space of the collective "other agent(s)" can quickly grow out of hand. It's unclear why you need the assumption of a fixed set of policies, rather than simply attempting to learn the function mapping the (projected) observation space of the other agents to their actions.

## Post-Rebuttal Comments
It seems as thought my final concern on policy representation is in fact not an issue. I would recommend that the authors make it clear in the intro and section 3.1 text that the fixed policies are not something the local agent has access to.
limitations_and_societal_impact: There are some substantial limitations/assumptions on the approach, and the authors address some of them. At this stage in the research, I don't believe there are any major societal impact concerns to address.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes an encoder-decoder type neural network for agent modelling in partially observable domains. The method seeks to learn a useful encoding of a controlled agent's action-observation histories to a latent vector. By useful, it is meant that the learned representation used as an input to a downstream RL method allows finding "best response" policies to a set of fixed opponent/other agent policies. At training time, decoders are trained to predict the observations and actions of the other agents. At test time, the decoder is discarded, and the encoder part is used to output latent vectors. The proposed approach is demonstrated in three MARL domains, showing improvement over baselines such as simply classifying other agents' policies, or discarding the agent information completely. Ablation studies confirm the effectiveness of the design choices for the proposed method.
main_review: # Positives

- As measured by the returns, the proposed method has good empirical performance in the domains considered. It improves on several baselines, that seem to be well chosen. It is also interesting to see that simply identifying which of the fixed policies the other agents are using (CBAM) has very limited usefulness, even in simple domains.

- The ablation studies are very nice, showing that the components of the proposed method are indeed useful. The encodings learned are reasonable in the speaker-listener domain. The ablations also clearly make the point that the entire trajectory of actions and observations matter in a partially observable setting.

- It is nice to see that the paper tackles the challenging partially observable setting, rarely considered in other works in agent modelling I am aware of. The paper is very well written and organized, and provides a good overview of recent agent modelling methods.

# Limitations

1. The set of possible opponent policies is assumed fixed. This limits the usefulness of the proposed approach somewhat, at least in competitive settings it is reasonable to expect some adaptation from the opponents.

2. As acknowledged by the authors (Appendix D), the method as presented is unlikely to scale beyond low-dimensional observation/action spaces, as it requires predicting the complete actions and observations of the other agents. It is not obvious to me how representation learning methods could help here.

3. In the domain studied in the paper, the proposed method works well. What could be an interesting future addition is to find some failure cases, which could point out the limits or perhaps features of the domain that make it challenging to apply this type of agent modelling.

4. The proposed LIAM method can be seen as a type of extra regularization, which could also partly explain the authors' observation of its stability. The assumption of this paper is that it is not enough to learn any representation of the controlled agents' action-observation history, but it should be one that enables prediction of the other agents' actions/observations. This suggests a potential additional weakness for the proposed method; in the presences of nuisance/unnecessary information, the method might overregularize to be able to predict useless details of the other agents' actions/observations. This complements point 2. above, and can appear even in low-dimensional settings.


Other comments:

- I was initially a bit confused why the POSG is chosen as a formalism, since the paper assumes that the policies of the other agents are fixed. In fact, the problem being solved is a POMDP, specifically searching for the best response policy for the controlled agents while other agents act according to some fixed policies randomly drawn from a set of policies.

- In the ablation study, the VAE method does not seem to help much. The reason stated is the restrictiveness of the KL regularization. But given that the messages received can uniquely be mapped to a specific fixed policy, it seems that there is generally not much to gain from a stochastic VAE? Did you observe the VAE to be helpful in the other domains?

- Line 117 defines an independent observation model for each agent. But I did not see anything that prevents applying this method when the observation model specifies a joint distribution over all agents' observations, so this seems like an opportunity for a small additional generalization.

- Line 87: remove extra "sample"

# Post rebuttal
I thank the authors for their response, in particular for clarifying their ideas regarding representation learning. Overall, this is a nice paper with some interesting observations and contributions.
limitations_and_societal_impact: Limitations of the work are appropriately addressed. Paper does not discuss societal impact.
review_title: null

Meta-review: This paper considers the problem of partial observability in multi-agent learning, and takes the approach of learning the latent representations of other agents from local observations. This is shown to be beneficial during learning.
The reviewers were generally positive about this paper, which presents an interesting and effective solution to an important problem.
The reviewer scores were improved as a result of the rebuttals, and the authors should include the promised revisions in the paper.
Decision: Accept (Poster)