Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies offline training and online tuning for multi-agent systems, which focuses on the decentralized cooperative setting where each could only access to its offline dataset. Authors discover that the transition dynamics in the offline dataset and online execution can differ very much, and propose to bridge this gap by introducing online transition correction by modifying the sampling probabilities. To accomplish this goal, authors propose practical implementations for the distance function measuring the similarities of states/state-action pairs. Authors conduct extensive experiments on multi-agent MuJoCo tasks, and show that OTC outperforms baselines (BCQ and AWAC).
main_review: - Pros:

(1) Authors study an interesting and practical setting, offline training and online tuning for multi-agent tasks. 

(2) The paper is well-written, with a clear description of the proposed method.

(3) Authors conduct extensive experiments to evaluate OTC.

- Cons:

(1) Related work: The idea of correcting sampling probabilities is very close to that in (Jiang and Lu, 2021), and authors should discuss more about MABCQ in the paper.

(2) Method:
The method involves a pairwise comparison of the sampled transition and all transitions from online execution, which can incur some computation overheads when D_i^k becomes larger. In addition, the paper also assumes there is always <s^*, a_i^*, s^{'*}, r^*> in the dataset, which may not always be the case. In the d-function part (Equation 7), it leverages the difference in value functions for measuring the distance. However, Q(s, a) can be over-optimistic for unseen actions. 
 
(3) Experiments: 

- There is a missing strong baseline MA-ICQ (Yang et al., 2021) which also studies the offline MARL setting.

- Given Figure 4, there seems to be very large variance, and I'm not very confident about the conclusion.

 - In most experiment tasks, there only involves 2 agent, which make it relatively small scale. Can the method scale with more agents? 

summary_of_the_review: The paper studies an interesting setting, but should explain more about the potential problem of the method (please check the main review) and compare it with a more recent MA-ICQ method.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the problem of online tuning for offline decentralized MARL. The online transition correction technique is proposed to correct the bias of the transition probability function between offline dataset and online dataset. Experimental results show that OTC is able to improve the performance of BCQ and AWAC given the same number of deployments and samples.
main_review: Novelty: The bias of transition dynamics in offline decentralized MARL problem is interesting. 

Soundness:
1.This paper studies the deployment constrained setting. However, BREMEN and MUSBO are never compared in the experiments. It is obvious that modifying the transitions given the new data improves the learning performance. 
2.It seems that the OTC technique can also be applied in the single agent setting. Does OTC performs better in MARL than SARL?
3.In each iteration, OTC needs to find the most similar samples for each sample in the mini-batch, which may incur large computation cost.

Significance: 
The performance improvement of OTC compared with BCQ is convincing. But baselines such as BREMEN are missing. 


summary_of_the_review: This paper studies an interesting problem in offline MARL. A transition modification technique, OTC is proposed to correct the transition bias during the learning. However, the experimental evaluation is not convincing as important baselines are missing, and the proposed algorithm seems to be too straightforward due to lack of theoretical analysis. Thus, I recommend the rejection.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel MARL method to bridge offline training and online fine-turning. More specifically, a transition function is maintained and updated according to the data obtained from online deployment. Similar state-action pairs are used to update the RL method.
main_review: The idea that extends deployment efficiency RL to multi agent setting is interesting. Although there are few papers consider offline MARL, they ignore how to consistently improve policy during online deployment. The paper is well-written in most of parts. However, I still have some concerns.

1.	For the high-level idea that only using similar transitions to train the policy, is there any analysis? Intuitively, online exploration aims at finding novel samples that not exist in the offline dataset since these novel samples are helpful to know the environment and build a accurate model. However, the novel samples are not prone to be chosen in the algorithm.
2.	Some details of the transition function are not introduced. How is the transition function represented? Why can the transition probability be modified by the Eq.3? Any deduction or proof? Do agents share one transition probability function? 
3.	For implementation, how many transitions are sampled in Line 16 of the Alg. 1? Since all the transitions should be computed for similarity in Line 13, what is the consumption of time?
4.	For the multi-agent setting, do all agents observe the state rather than their own observations? It seems that the method can be directly applied to single-agent setting. Is there any specific design for multi-agent settings?
5.	For baselines, some offline RL methods are ignored like [1]. I understand that these methods are relatively new and may not be published when authors submit the paper. So I just want to notice authors that offline MARL can also be directly used in the setting of the paper. I would not decrease my score due to this reason.

[1] Offline Decentralized Multi-Agent Reinforcement Learning. Jiechuan Jiang and Zongqing Lu. Arxiv 2021

summary_of_the_review: Although the paper is interesting, I still have some concerns on multiple aspects. Three important issues are: 1. The details of the algorithms. 2. The unique design for multi-agent settings. 3. The high-level idea of using similar samples for update.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the discrepancy between the online and offline transition data in multi-agent RL training. The author proposed OTC to correct the biased transition dynamics and propose a training framework that can leverage offline data for more efficient online training. Results on Mujoco tasks have been provided.  
main_review: Overall, I don't have issues with the technical proposal of this paper. 

* The author studied an important problem in offline MARL. When the opponent agents' actions are not accurate, it can cause mismatch between the transition dynamics in the offline data and the transition dynamics in real online settings. As a remedy, the authors proposed to solve this issue by bridging offline training with online tuning. 


* OTC essentially tries to find the optimal probability function that minimizes the KL divergence between online transition distribution and modified transition distribution, i.e., $P_{B_i}$ and $P_{\epsilon_i}$. 


* The problem formulation of how to select the best-matched transition online data among the offline data is clear. Considering the distance measurement in terms of both embedding space and latent representation is reasonable. 

* Using rank-based prioritization in PER to compute $f$ is reasonable. 




summary_of_the_review: Though the proposed work is sound on the technical side, I feel not surprising about the proposed method. The novelty is rather limited in the sense that the proposed method is a way to shift the offline data for updates in the context of BCQ. The experiment also seems rather limited; showing its effectiveness over BCQ is also not surprising either. Given the novel setting of offline training with online fine-tuning, the author maybe want to consider transfer learning cases like few-shot settings, and experiments on other multi-agent environment such as StarCraftII will also be favoured.
review_title: null

Meta-review: ## A Brief Summary of the Paper

 In offline decentralized MARL, the discrepancy between the offline data and the interactions of agents in the environment can cause discrepancy and as a result, the policies will perform suboptimally. This issue in ORL is known as extrapolation error. This paper is trying to address the issue of extrapolation error with offline decentralized agents. It is possible to alleviate this problem by combining offline RL with online fine-tuning. This paper introduces Online Transition Correction (OTC) approach to address this problem which aims to correct the biased transition dynamics with a form of importance sampling based on embedding and value-based distance metrics.
 
 ## Summary of the Reviews
 
 Below I will outline some important concerns raised by the reviewers.
 
 ### Reviewer T8NR
 **Pros:**
 - Interesting and practical setting.
 - Extensive experiments to evaluate OTC.
 **Cons:**
 - Lack of enough discussions about closely related works, for example, the MABCQ algorithm. 
 - Computational cost of OTC.
 - Experiments: comparisons against baselines (MA-ICQ), Large variance in Figure 4. Small-scale (only two agent) setting, can it scale to more agents? 

### Reviewer 2xuS
 
 **Pros:**
 - The bias of transition dynamics in offline decentralized MARL problem is interesting.
 **Cons**
 - Key baselines such as [[BREMEN]] and [[MUSBO]] that looks into the deployment constrained ORL is not compared against in this paper.
 - The proposed OTC algorithm can easily be applied to Single agent settings. MARL vs SARL comparisons would be interesting.
 - Large computational cost incurred from OTC. Because of the search procedure of finding most similar examples to the examples in the dataset.

### Reviewer a2PP
**Pros:**
- Well-written.

**Cons:**
- Analysis on the behavior of the policy, in particular on novelty seeking during the online finetuning phase.
- The missing details of the transition function.
- Compute constraints and budget.
- Unclear experimental protocol: states vs observations...
- Missing baselines.

### Reviewer ZgL6
**Cons**
- Lack of novelty.
- Limited experimental results: transfer learning scenarios, lack of experiments on multiagent environments such as Starcraft II.

## Key Takeaways and Thoughts

Overall, the authors did a good job addressing the concerns raised by the reviewers. For example, the authors ran additional experiments and compared single-agent BCQ with and without OTC on some D4RL tasks. The authors gave detailed responses to the questions related to the computational cost. However, the initial submission version of this paper feels rushed as it is submitted to the conference. I would recommend the authors, go through the reviews carefully and address the points raised by the reviewers carefully in a future version of this paper. As it stands now, it is difficult to evaluate the results reported by the authors during the rebuttal, due to the lack of clarity about their experimental details.

I think the writing can be improved further. There are several typos in the paper, and most reviewers were confused about the novelty of the paper. I would recommend the authors to provide more detailed discussions about the differences from other similar approaches in the literature. Also, justify the selected experimental protocol better.
Decision: Reject