Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes a learning framework, MAGENTA, that can tackle multiple multi-agent environments with one transformer agent. The authors focused on real-time strategy games in this work, including Honor of Kings (HoK), Starcraft II micromanagement (SMAC), and Neural MMO (NMMO). They treat agent entities as tokens and learn separated tokenizers, a unified transformer body, and separated output layers. It shows that it is possible to train generalist agents for multiple multi-agent environments in an online manner (using PPO), and shows that such online generalist agents can rapid adapt to never-seen-before games or scenarios with fine-tuning. The results empirically show that the proposed model learn common knowledge about multi-agent games across various categories.
strength_and_weaknesses: Strengths
* It is a good proof of existence of generalist agents for multiple multi-agent environments. I think showing the community how to learn such agent is a valuable empirical contribution.
* The transfer-to-unseen-scenario results and transfer-to-new-game results are convincing, showing that training the pre-trained model with more games are usually beneficial.

Weaknesses
* I found Sec. 5.5 (scaling and comparison to single-game agents) confusing. With only scores normalized by single-agent performance at each size tick. I think it doesn't have enough information for me to interpret the results and the conclusion "We interpret this result as overparameterization, where a richer model is fitting than necessary" is kinda hand-wavy. I would suggest adding raw scores, which could help the reviewers to inspect the claim and help reader to understand better.
* The learning curriculum (Figure 5) is adhoc, and there is no ablation on the curriculum. The current curriculum basically uses Honor of Kings as an anchor. I think even just explaining what could happen otherwise would be helpful. 
clarity,_quality,_novelty_and_reproducibility: As pointed out above in weakness, clarity has some room for improvement. Some further questions/comments I have include:
* There are many different (sub-)version of models. I think having a table to list them and also list the transfer experiments would be helpful.
* Why in Figure 4 both decoder and pooling outputs $z_i$?

Nitpick:
* Gammer error in "How the performance of the entity transformer in single game?" (beginning of Sec. 5)
summary_of_the_review: I'm not able to interpret the results given the current manuscript, so couldn't recommend acceptance now. I hope the authors would be able to fix this.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces MAGENTA, which is a variant of PPO with a transformer architecture. Based on experiments in large cooperative video games, MAGENTA appears to be able to transfer knowledge across games. 

strength_and_weaknesses: I think this is a nice paper. Although similar research has been done in language, offline RL, and multi-game RL, this is the first to show that the same trend of scale helping generalization also works in cooperative multiagent RL.
The experiments are on very large video games, and it is not obvious that so much would transfer to different video games, especially because the representation is not the part that gets transferred, but the transformer that decides which actions to take. 
One thing I would have liked to seen would be better small-scale experiments where comparisons with many baselines could be tested. 
The results are not striking, but I think they clearly show the benefit of MAGENTA
I think this paper will have some impact in the related literature on cooperative RL and transformers for decision making
Figure 7 should be clearer, I think this is head to head against the FC+LSTM

clarity,_quality,_novelty_and_reproducibility: I thought it was quite clear. The authors do not have code and do experiments on large games, so I don't think it is very reproducible.  
summary_of_the_review: I think this paper is the first to show what many might have suspected: that large models transfer and generalize in cooperative multi-agent environments. This is interesting and I think future work can build off of this fact.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed a novel transformer architecture to solve multi-agent games. The model contains two major parts (1) game specific tokenizers and output layers which encode different game observations into the same token space and output the right actions (2) the main transformer model with ​​permutation invariant pooling and encoder-decoder modifications. With a multi-stages training, the proposed method is able to fast adapt to novel games or new game settings. 

strength_and_weaknesses: The strengthnesses of the paper:
1) Proposed a single transformer architecture to solve multiple different battle games.
2) Ablation studies to test the performance of the model(s) at different training stages.

The weaknesses of the paper:
1) The clarity of the paper can be improved. For example in section 4.2 authors discuss the encoder-pooling and the modified encoder-decoder architectures, but leave many missing details: the latent variable z_i seems to be a scalar with encoder pooling due to the symmetry requirements, which looks rather stranger (or is that the case?).  
2) Need more details on the architecture, i.e. how many encoders/decoders/attention heads are used, etc. 
3) Also, the performance comparison plots use different x-values: some use samples and some use time. This is a bit confusing. 

clarity,_quality,_novelty_and_reproducibility: The writing is mostly clear with some details missing. The results and ablation studies are plenty. The originality of the work seems minor since the authors test only minor modifications to the transformer architecture. 
summary_of_the_review: While the paper is mostly clearly written, the novelty of the paper is limited.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper describes a tranformer-based architecture for policies for multi-agent (video) games, that tokenises image-based inputs plus entity-component-system-based inputs (representing data of entities in video games), allowing the architecture to process inputs from various different games in a single and consistent manner. Policies are trained in three different video games (Honor of Kings, Neural MMO, and StarCraft II Micromanagement), some of which have multiple different scenarios. Several experiments show that it is effective to train in some games, transfer to others, and then finetune in the other games.
strength_and_weaknesses: ## Strengths:

The primary strength of the paper is that the empirical results look convincing, and sufficiently extensive with some ablations. To the best of my knowledge, the proposed architecture is also novel, and it has some interesting insights.

## Weaknesses:

The primary weakness is that several parts of the paper are unclear, and some statements even seem wrong. See below for detailed comments related to this.

---

**Note after discussion with authors**: I feel that the paper has been substantially improved during the discussion phase and have updated my score accordingly.
clarity,_quality,_novelty_and_reproducibility: My primary concerns with this paper are related to clarity, and sometimes quality (in the sense of correctness).

## Detailed comments

- "we analogize games as languages" (in abstract) --> I think this is potentially confusing. When I first read this, I thought it would mean that the authors would use a single language in which to describe (the rules of) the different games. But the only relation to languages seems to be that tokenization is also commonly-used there... which hardly seems an important enough similarity to describe it as "analogize" like this.
- "due to trail-and-error, RL is inclined to overfit to training environments" --> how is this related to trial-and-error? If, hypothetically, we could exhaustively enumerate the complete space of all possible trajectories and learn from that (so no more need for trial-and-error), would we not still overfit to the environment for which we trained anymore?
- "we investigate whether a single model, with a single set of parameters, can be trained by playing multiple multi-agent games in an online manner." --> this sentence highly suggests, especially due to the phrasing and mention of a single set of parameters, that a single model will be trained that can play *all* games at the same time, with the same weights (i.e. not separated by training runs in between the different games). If I understood all the experiments correctly, there is no experiment where truly the exact same set of weights is used for multiple different games. I have a similar issue with describing the agents as "generalist agents" in the Conclusion. This is an important difference with agents such as Gato.
- Throughout most, or maybe all, of the paper, there are several mentions of permutation invariance, but it is not actually clear in what ways there is permutation invariance: for which dimensions of the input space do we want permutation invariance, and why? After having read the full paper I can understand that the permutation invariance is between different entities: we should not care about the order in which different entities are presented as inputs to the model. But this is not clearly described in the paper, at least not nearly early enough. This intuition is very simple, and would be very helpful to describe early in the paper, but it isn't. In contrast, the phrase "permutation-invariant" already does show up several times early in the paper (e.g., towards the end of the Introduction), but without the intuition about the dimensions of the input space for which we want permutation invariance, and why, this is difficult to follow.
- It is not clear to me how "words" are "typically composed of images and vectors". It would probably help to give some examples of what things would be images, and what things would be vectors.
- One subfigure of Figure 8 is missing the red line, and one is missing the green line. Where did they go?

## Other minor comments

- "(2020), Furthermore" --> looks like a comma instead of a period ending the sentence
- "AlphaStar requires [...] half a month" --> should provide references for all these agents and numbers
- many instances of "casual" which I think should probably be "causal"?
- several cases of "radio" which I think should be "ratio"?
- Subsection 3.3 just lists a bunch of publications that used video games, but does not discuss that work in any way. In this form, I'd say the entire subsection seems unnecessary and could be removed.
- "can be represented as (1)" --> what is (1)?
- Legends in Figures 8 and 9 are impossible to read unless I zoom in to 150%, which probably means they would be unreadable in print.
- The caption of Figure 12 mentions P3, but I dont see P3 anywhere.
summary_of_the_review: The paper is interesting and has some interesting and potentially significant contributions, but in its current form has too many issues in terms of clarity.

**Note after discussion with authors**: I feel that the paper has been substantially improved during the discussion phase and have updated my score accordingly.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper explores the application of pre-trained transformers to decision making in multi-agent systems. Conceptually, this could be a significant step forwards from recent work applying these models in single agent games. However, the reviewers raised many issues with the clarity of the paper initially submitted. The authors have made good progress on addressing the minor issues during the discussion period, but there remains large uncertainty over the precise model architecture and experiments reported on. Without these major issues addressed the insights that can currently be gained by an external reader are limited.
justification_for_why_not_higher_score: Too many unresolved issues with clarity on what experiments were run
Decision: Reject