Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper designs an offline MARL framework based on decision transformer and policy distillation. And proposes a novel distillation loss to distillate the structural relations among different agents from a centralized teacher policy. The experiment results illustrated the improvement brought by policy distillation.
strengths_and_weaknesses: Strength:
1. The idea of combine knowledge/policy distillation and decision transformer for multi-agent scenario is interesting. 
2. The proposed design of relational policy distillation loss is reasonable in multi-agent cases.
3. The authors conduct thorough experiments on multiple environments. The proposed framework achieves state-of-the-art results and the ablation shows the performance raise brought by relational policy distillation loss is important.

Weaknesses:
1. Although the proposed framework achieves significant performance raise on the four environments designed by the authors. The improvements on SMAC benchmarks are not significant compared to MADT. In the ablation studies, the conventional distillation method performs worse than IDT or MADT on multiple tasks. Is this because the teacher policy itself does not take a big lead on these tasks?
2. The authors did not mention the settings for initial target return, which would significantly influence the performance of DT in single-agent settings. The setting of target return in multi-agent cases should be more complicated as it involves credit assignment. 
3. The authors did not analysis the influence of number of parameters, which might be important as DT is more complicated than networks used in previous offline MARL methods.
questions: See weeknesses before.
limitations: The authors mentioned the scalability problem. Also, the tested environments most have relatively short horizons. I wonder whether this framework can still remains a relatively fast training speed (additional 10% training time) if the horizons are long since attention has O(L^2) complexity.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors propose a distillation-based offline MARL method for improving the quality of pre-trained multi-agent policy online. The authors construct the teach and student policy with access to the global and local information, repsectively. Experimental results on the intuitive and public environments show improvement over many strong baseline methods.
strengths_and_weaknesses: Strengths: The authors propose the utilization of global and local information with teach and student policy distillation.
1. The authors show the reasonable knowledge distillation on the teach policy pre-trained with global information. Then the student policy can be learned by distilled the parameter individually with relational constraints to improve the cooperation. The idea can be validated from the centralized training and decentralized execution (CTDE) framework.
2. The clear writing and sufficient experiments show the effectiveness of this approach compared with several strong baselines including sequential modelling on MARL, MADT. 

Weakenesses: The learning complexity may increase with the scale of multi-agent enlarges.
1. I think this work has the obvious limitation in the computation complexity including learning the teacher policy from offline dataset, and constraining the relational distillation.
2. In line 296, the authors conclude the supereme in the poor offline data compared with MADT. I want to know the theoretical analysis of this results in detail.
questions: 1. The idea of this paper is similar to a recent work named Jump-Start Reinforcement Learning, which learn to use the guide policy to inform the whole online policy. Can you show the difference between the proposed approach of this method?

2. Can you show more detailed distillation process used in this work? And how to distilled a reasonable student policy?
limitations: The scalability may limit the application of this work, which the complexity will increase with the scale of multiple agent enlarges as described above.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors aim to tackle the multi-agent offline RL domain by leveraging the power of transformers for credit assignment using sequence modeling, hence removing the reliance on TD-learning-based bootstrapping methods. The proposed method first trains a “teacher” decision transformer on the full dataset to learn cross-agent interactions. Then the teacher distills relevant knowledge into the “student agents” via “structural relation loss.” Thus the proposed method utilizes the attention mechanism for learning interactions among the agents better than the current offline RL methods.
strengths_and_weaknesses: # Strengths

1. Leverages transformers for learning beneficial cross-agent interactions and distills them into student policies via well-motivated structural-relation loss constraint
2. Achieves strong empirical performance in various multi-agent environments
3. Authors conducted extensive ablation studies to show the importance of every component in the proposed method, as well as discussed.

# Weaknesses

1. Figure 6 compares only the proposed method w/ and w/o pretraining. Authors should include a comparison with MADT-online too.
2. My main concern is that the method does not seem scalable. In environments like SMAC (where number of agents > 2), simpler transformer-based method MADT performs comparable to the proposed method. 
3. Minor: Lots of typos and grammatical errors in the text
questions: 1. In the Algorithm, shouldn't the student-agent gradient update equation (φi = φi − α ∇φi (Li action + αLi rel + βLi KL)) come inside the inner for-loop?
2. I am not sure how the **Fill-In environment** works. If agent 1 has passed a particular block, will that block be then inaccessible for agent 2, and vice-versa?
3. **If the answer to the above is yes, then skip this point.** But if it is not, it is unclear why the agents here even require cross-agent credit assignment during training?
4. Re: Table 11 - Why does the return distribution for the Highway dataset have the opposite order, i.e., poor > normal > good? It seems like a typo to me.
5. Referring to Line 563 - why are obstacles and apples needed in the Fill-In environment? Is this a typo?
6. In the **equal space environment**, what are the initial positions of the three agents?
7. In Table 2, how were the trajectories categorized into “good,” “normal,” and “poor”? Could the authors shed more light on how different qualities of datasets were collected?
8. MADT has been cited with citation number 18 in many places (including the tables), but citation 18 refers to MA-BCQ. The correct citation number for MADT is 27. The authors should thoroughly cross-check the citations in the main text and the supplementary.
limitations: - Authors haven't discussed potential negative societal impacts of their work. 
- Authors have addressed the scalability of their method as a limitation.
review_title: null

Meta-review: metareview: This paper deals with offline multi-agent RL tasks. Based on decision transformer, the authors propose to first train a centralized decision transformer over the offline data to capture the agent interaction patterns and then distill the knowledge from the centralized decision transformer to the individual agents. The idea is clear and the experiment looks comprehensive. As mentioned by the reviewers, the performance of the proposed method looks just comparable to MADT, which is a very simple parameter-sharing architecture. 

confidence: Less certain
Decision: Accept