Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper tackles the problem of efficient offline RL in the multi-agent setting (MARL) that typically suffers from the curse of dimensionality as the number of agents grows. They argue that transformers are ideally suited for estimating the RL components (value functions/dynamics models) and are able to implement efficient relational reasoning between agents.
They combine model-free and model-based RL algorithms with transformers as function approximators for the value function and dynamics model respectively, and present theoretical results for the generalisation and suboptimality gaps for the resulting algorithms.
strengths_and_weaknesses: The papers main contribution is the theoretical analysis of MARL algorithms when using transformers for function approximation, showing that they can get significantly tighter bounds on the generalisation error e.g. for the model-free algorithm the error bound becomes independent of the number of agents.

I am not familiar with details of prior work on offline MARL and the presentation of the paper made it often difficult to judge the significance and the originality of some of the contributions. E.g. "we design offline model-free and model-based RL algorithms with the transformer approximators" It seemed to me that the authors modified existing RL algorithms by simply replacing the function approximator used with transformers?

I would have appreciated more discussion on the assumptions required for the results (to be able to identify which ones are the strongest), instead of simply referring to other works that make similar assumptions. E.g. the IID assumption (instead of sequential) on the offline data.

The paper is missing a discussion and/or experimental demonstration of how much of the favourable scaling properties would carry over to more realistic settings which would make the significance of the results a lot more clear.
questions: See comments above regarding assumptions and applicability in more realistic settings. I would be also curious to know if the authors can say something about the online MARL setting when using transformers?

Given the topic of the paper (how a neural network architecture can give rise to efficient algorithms), it could greatly benefit from an experimental demonstration of the theoretical results.

The conclusions discussion should be part of the main text, not supplementary.


limitations: As mentioned before, the authors should elaborate on the assumptions necessary for the results and what we can expect in practice when they don't hold.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents theoretical analysis on the use of transformers for offline multi-agent reinforcement learning. The main contributions are: i) a proof that approximating the "relational reasoning" of set transformers using feed-forward neural nets requires exponential width, ii) model-free and model-based algorithms for offline MARL using transformers and pessimistic policies, which minimize the effect of distribution shift, and iii) suboptimality gaps for the proposed algorithms showing that they scale well with the number of agents. 
strengths_and_weaknesses: Unfortunately, I found this paper very hard to understand, even after spending several hours on it and multiple reads. Not only it is theory intensive, but it often introduces ideas and terminology too suddenly and with very sparse explanations. Since this is not my area of expertise, I will opt for assuming that the math and derivations are correct, in which case I think this paper is probably a good contribution to the conference. The topic is clearly relevant, and the use of transformers is gaining prominence in in reinforcement learning, so analysis such as the one presented in this paper are of great interest. But again, I must qualify this opinion with the caveat that I'm taking the results offered at face value. 
questions: No questions. 
limitations: I didn't see any discussion on limitations in this paper, which in fact doesn't have a final discussion/conclusion section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper concerns the theoretical understanding and relational reasoning of permutation invariant agents framework in MARL. It proposes offline MARL with the transformer and analyze the error bound.
strengths_and_weaknesses: - It utilizes self-attention mechanism that is widely used in cv and nlp to model relational reasoning between agents
- It proposes both model-free and model-based offline MARL
- It theoretically prove the gap does not scale with the number of agents and the proof is complete 
questions: - The paper compares relational reasoning between self-attention and MLP through the width of NN, what about the parameters and computation?
- Could the method execution decentralized?
limitations: - The environment is simple and can't empirically demonstrate the performance of the method.
- The novelty is not enough. The paper extends single-agent offline RL and utilize set transformers as neural network structure.
review_title: null

Meta-review: metareview: The paper presents theoretical results justifying the use of transformers in cooperative multi-agent RL. The authors demonstrate that, with this choice of architecture, sub-optimality gaps grow independently of the number of agents. The theoretical contribution seems strong, with a more limited experimental evaluation. The paper is dense mathematically and was hard to assess. The theorems were nevertheless deemed strong enough to justify acceptance, however please do address comments from reviewer gWc2 in the final version.
confidence: Less certain
Decision: Accept