Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Multi-Agent Joint-Predictive Representations (MAJOR), which applys the self-supervised learning (SSL) technique to MARL, trying to improve the learning efficiency. Specifically, MAJOR builds a transformer-based transition model, which takes all agents observations and actions as inputs and predicts the observations of the next $K$ steps in the latent space. Experimental results on four MARL benchmarks show the SSL auxiliary task could slightly improve the performance of the underlying algorithms.


strength_and_weaknesses: **Strengths**:
   * Studying how to learning efficient state abstraction in MARL is an interesting and open problem.
   * The paper conducts extensive experiments on wide-range MARL environments.

**Weaknesses**: 
  * The performance improvement of MAJOR is minor compared with the baselines. The training curves of different algorithms are overlapped and thus the improvements are not significant.
  * The benefit of applying SSL to vision-based single-agent RL algorithms is that the policy is learned in the transition-irrelevance or policy irrelevance low-dimensional latent space rather than the original high-dimensional visual space. The paper simply applies SSL to existing MARL methods and shows the experimental results. But the motivation is not very clear. Why applying SSL to MARL is beneficial especially for state-based inputs? 
  * Predicting future observations simply based on all agent's current observations may also be inaccurate. In POMDP, all agent's observations may not contain all the information in the state due to the partial observation issue. 
  * Minor:
    * "And we define the following mean squared error between the normalized predictions and target projections". 'mean squared error' should be 'cosine similarities'.
clarity,_quality,_novelty_and_reproducibility:  * Most parts of the paper are clear but the writing could be further improved.
 * Although, the code is not attached in the supplementary material, the author provides detailed parameter settings in the Appendix. I think the results could be reproduced according to these detailed settings.
summary_of_the_review: This paper simply applies SSL to existing MARL methods and shows the experimental results. But the motivation is not very clear. Besides, the experimental results are not significant and the writing of the paper could also be improved. So, in its current form, the reviewer holds the point that the paper is below the acceptance threshold.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper describes Multi-Agent Joint-Predictive representations (MAJOR), which is a self-supervised learning mechanism that allows the MARL system to learn policies in a data-efficient manner.
In MAJOR, observations obtained by individual agents are treated as a masked sequence for representation learning, i.e., masked prediction. 
The learned representation should be jointly temporally predictive and consistent across different views overall agents.
The performance  is evaluated through both visionbased
and state-based cooperative MARL benchmarks.
The empirical results show that MAJOR outperforms pre-existing methods. 
strength_and_weaknesses: Strength:
-The empirical result solidly shows the performance of the proposed method.
-The proposal is based on the recent progress and success in SSL, and is insightful.

Weakness:
-The qualitative evaluation and discussion are limited.
clarity,_quality,_novelty_and_reproducibility: The paper describes its background and proposal clearly. The quality is high. The experiments are comprehensive, and the results look sound. In addition, experiment settings are described clearly to a certain extent.
summary_of_the_review: The proposed method is new and improves the performance of MARL.
Though further discussion is expected, the paper has sufficient strength.
The proposal is insightful and interesting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to integrate an SSL auxiliary objective with standard model-free MARL methods (specifically MAT in this paper) to reap the same benefits as CV and single-agent RL has seen with these types of approaches. Because each agent in a typical MARL setup receives a partial observation of the world, in order to apply the SSL auxiliary objective, they treat the individual observations as a sequence of masked views of the global state. Concretely, they use a joint transition model that inputs the individual agent observations and outputs the future observation representation for each agent, which implicitly requires a reconstruction of the global state. The joint-predictive auxiliary objective they introduce on top of MAT is BYOL-like prediction loss. The authors note that a similar objective can be applied to other model-free MARL methods. The new approach, MAJOR, is evaluated on a number of vision- and state-based multi-agent settings. Performance gains in sample efficiency are reported over MAT and other model-free MARL methods across the various settings. 
strength_and_weaknesses: Strengths: 

* The problem is well-motivated; there is a lack of MARL approaches that take advantage of SSL, and there are a number of challenges associated with doing so. 

* The proposed objective is a very natural incorporation of a BYOL-like SSL objective into an existing MARL approach. Moreover, the approach appears to be relatively applicable to other model-free MARL methods.

* The writing is generally clear and gets the main ideas and implementation details across. Figure 1 (the illustration of the full MAJOR method) is also a very useful figure for understanding the gist of the method. 

* The method shows improvements in sample efficiency (and in some, final convergence in alloted training time) across the board on numerous and diverse multi-agent environments. 

Weaknesses: 

* Given that other baselines are used apart from MAT in the evaluation section, and that the authors proclaim the approach is “a plug-and-play module”, it would have been really nice to see the loss incorporated into all of the baselines to really drive home this point. Without this, it is less convincing that the module is indeed as simple as “plug-and-play”.

* The methodology section is quite dense, and very few clarifying breathers are given while the stream of details are being revealed. This required me to jump back and forth quite a bit between the more conceptual explanations and the more technical explanations.

* A lot of the intuition/understanding may require familiarity with previous works such as MAT and especially BYOL.

* The results are nice, but the performance improvements (even in terms of sample complexity) are quite modest given the involvement of the approach. Especially for this reason it would have, again, been nice to have seen this module applied to multiple approaches. 

* Some grammatical issues / typos (there are others too—I suggest a careful rereading at the sentence-level):

  * First sentence of 3.1 is very hard to parse (run-on sentence and probably grammatically incorrect): “We start with the intuition that encourages the observations and actions representations integrating other agents’ information and embodying agent-level relationships and interactions should reduce the non-stationary during the learning process, thus improving the data efficiency of MARL”

  * Second paragraph 3.1 “since this way can execute” (not proper grammar)

  * First sentence of 3.2 doesn’t make sense: “Multi-Agent Joint-Predictive Representations(MAJOR) is an auxiliary objective to promote the learned representations from the latent space of the sequential observation and action embeddings” (promote what in the learned representations?)

  * Grammar and spelling at top of experiments: “We consider wide-rage MARL benchmarks for evaluating MAJOR and compared MARL algorithms” (”wide-rage” and “and compared”; maybe something like “a wide ranging set of” and “in comparison to relevant SOTA” is meant?). 

  * Confusing phrase at the end of “Multi-Agent Quadcopter Control” paragraph in 5.2: “our proposed representation learning framework shows its strength if the underlying approaches don’t work at all” (unclear what is meant here).
clarity,_quality,_novelty_and_reproducibility: * The paper is generally well-written and of good quality, except for the weaknesses listed above. 

* The method and empirical analysis is novel. 

* There is no reproducibility statement or promise to publish code, and I could imagine it being difficult to reproduce method/results even with provided detailed explanation of components.
summary_of_the_review: * Overall the paper is well-written and provides a meaningful contribution to the field of MARL. There are some confusing and grammatically incorrect sentences here-and-there that should be corrected, and the claim of the module being plug-and-play is unsubstantiated (and if it were to be substantiated it would make up for the modest improvements in most of the tasks), but it is likely this paper will be useful to the research community in its present state.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a self-supervised learning approach for MARL which builds on multi-agent transformers to improve results in several domains. This approach consists of learning a centralized world model as an auxiliary task for learning better representations.
strength_and_weaknesses: # Strengths
* The main idea is straightforward and improves results.
* The proposed approach is well motivated.

# Weaknesses
* The writing could be more clear. 
* Improvements over MAT (which MAJOR builds on) are marginal in many environments.
* Baselines outside of MAT are methods which learn decentralized policies which seems an unfair comparison (unless my interpretation of the proposed approach as being fully centralized is incorrect - see questions).

# Questions
* From the description of the multi-agent transformer, it seems that observations from *all* agents are passed into a single network which computes actions. Given this description, the approach would be fully centralized, correct?
* The authors state that the proposed framework could be applied to CTDE methods, though it is not clear to me how that is the case since the representation learning objective requires sharing information across agents.. Can the authors provide an example of how this may be accomplished in practice?
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper uses somewhat unnecessarily complex exposition to put across a relatively simple point, hindering clarity at times.

Quality: The work is solid and builds on prior work. Presentation could be improved in places (e.g. it's not clear why the Meetup task is included)

Novelty: The work is a simple combination of ideas from different sub-fields (MAT + SSL).

Reproducibility: Source code, hyperparameters, and pseudo code are all provided, so reproducibility should be straightforward.
summary_of_the_review: I like the fact that the proposed approach is simple, builds on existing work without adding an overly complicated architecture, and is effective. My main hesitation with the paper is that it overcomplicates the exposition of a relatively simple idea and the presentation in the experimental section could use some work. However, these concerns are relatively minor.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new state encoder / auxiliary task for multi-agent learning in partially observable environments. Since this is a borderline paper the AC and reviewers discussed the contributions in an online meeting and arrived at the following conclusions:
1) The paper is interesting since it presents a potentially innovative solution 
2) However, the method is also extremely complicated and it's unclear based on the experimental results whether the complexity is indeed resulting in better performance. For example, the additional results for QMIX vs QMIX + MAJOR do not look significant based on the standard deviation / mean. 
3) The final, major concern is the selection of the value "k" for the final run and the lack of significance analysis in all of the results.

The last few points were raised in the online meeting and then posed as questions to the authors. Unfortunately the response was not satisfactory - in particular the authors failed to clarify why a given value of "k" was chosen. Picking this best one a posteriori can be an easy source of bias. 

justification_for_why_not_higher_score: As mentioned above, the experimental results where not convincing in the current version of the paper and the method seems unnecessarily complicated given the limited gains. I also posted a last minute request for clarification to the authors based on the AC meeting and did not obtain a satisfactory answer. 

justification_for_why_not_lower_score: NA
Decision: Reject