Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies an offline MARL setting, with a focus on the decentralized case, and discovers that the difference between transitions in the dataset and other agents can be very large. Authors propose two techniques, value deviation and transition normalization, to modify the transition probabilities. Experimental results validate its performance improvement over BCQ on multi-agent MuJoCo tasks.

main_review: Pros:
1. The problem studied in this paper is very interesting, with a clear motivating example emphasizing that in the decentralized offline MARL setting, the difference between transition probabilities in the dataset and other agents can be large and harm the performance. Overall, the paper is well-written.
2. The theoretical analysis shows that the proposed operator is able to converge.

Cons:
1. There is a recent paper about offline MARL, MA-ICQ (Yang et al., 2021) which should be compared or at least discussed in the paper.
2. The assumption in Theorem that gamma<r_min/(2r_max-r_min) seems too strong in the case that r_max>>r_min. 
3. The experimental evaluations are not sufficient as it only compares MABCQ with baselines on a mixed dataset. How does MABCQ perform on other types of datasets?


summary_of_the_review: This paper studies an interesting offline decentralized MARL setting. However, relevant works are not cited and discussed, and experimental evaluation can be improved.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a scheme to modify the transition probabilities in the learning process in a fully decentralised MARL setting. The authors show that the method is still converging to the optimal solution. They also provide some experiments on synthetic environments.
main_review: It is not clear to me when the setting described by the authors occurs in practical cases. I think that the paper lacks a clear applicative example in which the framework proposed by the authors occurs.

The assumption that the transition probabilities are deterministic for the environment seems to be a bit restrictive. Again, if the author could provide a practical example, it would help to motivate the setting described by the authors.

In the first formula of page 5, the first two terms cancel out. I suggest to provide a more direct formula for this probability, e.g., P_B_i(s'|s,a_i) = 1/|S|. Moreover, the presentation of these modifications of the probabilities should be more formally defined with a proper symbol for each one of them.

Proposition 1 is misleading. If I understood correctly the theorem holds if one applies Q-learning over the previously modified transitions. It should be mentioned in the theorem statement.

The title of Section 3.3 (Implementation) is misleading. First, I would have mentioned which are the characteristics and the assumptions required for a generic methodology to be compliant with your proposed approach. Second, this seems a specific case of the application of your method rather than its implementation.

I think that the experiments, even if they present some difference in the expected values of the agents' objective, do not provide any statistical significance of the fact that the proposed method is somehow providing an improvement over the existing ones (other than DDPG).

It would also be interesting to see the performance of fully centralized learning, to understand how much we are losing from the fact that we are learning in a decentralised manner.
summary_of_the_review: The topic is interesting, but the setting lacks a strong applicative example and the formalism of the paper should be improved before it is worthed for publication. Moreover, I think that the statements sometimes are hard to understand. Therefore, I suggest the author should try their best to improve the readability of the paper. Moreover, the experiments should provide some statistical significance that the proposed method is better than the state of the art.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method named MABCQ to utilize offline data in the MARL environment via (1) value deviation and (2) transition normalization. These two techniques are used to correct the bias in the individual observation of the transition probability and improve the discovery of optimal policy. The authors showed the convergence of the Q-learning under the non-stationary transition probabilities after modification.


main_review: I appreciate the ideas proposed by the authors. However, a lot of descriptions in the submission are very vague and the statements are mathematically imprecise. I also have some concerns regarding the generality of the proposed framework. Please find my detailed comments below.

(1) A lot of sentences are very vague and imprecise (without introducing the framework in a rigorous) way. To name a few examples:

(1-A) It's stated in the abstract that "However, the transition probabilities calculated from the dataset can be much different from the transition probabilities induced by the learned policies of other agents..." This really depends on the information observability of the agents. If an individual agent could observe the states and actions of all agents in the system, then this won't be an issue.

(1-B) In the first paragraph, the authors keep switching between "agents" and "agent" and sometimes mentioned "offline RL", it is difficult to understand if you are always talking about the multi-agent setting or moving to talk about the single-agent setting at some point.

(1-C) The authors start with "The main challenge of offline RL is the extrapolation error" in the second paragraph, are you referring to the single-agent case, or multi-agent case, or both? Later in the paragraph, the authors mentioned "... ignore to correct the transition bias" without proper explanation of what you mean by transition bias.

(1-D) The authors mentioned multiple times on "decentralized multi-agent environment" which is confusing to me. Based on my understanding of the paper, the proposed learning procedure is decentralized (choice of the agent or the coordinator) but this has nothing to do with the environment.

(1-E) The second paragraph on page 2 does not make sense to me before talking about what is observable and what is not abservable to each individual agent.

(1-F) I don't understand the sentence "If the transition probabilities of high-value next states are extremely low..."

(2) Set up in Section 3.1:

(2-A) The problem description is not mathematically rigorous. The authors should specify the dimension or the space.

(2-B) Why the reward only depends on the state (not the action)?

(2-C) It reads to me that the centralized training and decentralized execution scheme could still be applied to this setting. In this case, the bias issue mentioned in the motivation should be solved.



summary_of_the_review: A lot of descriptions in the submission are very vague and the statements are mathematically imprecise. I do not think the submission is ready for acceptance with its current presentation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies a MARL offline setting, where each agent is trained independently. value deviation and transition normalization techniques are proposed to modify the transition probabilities. Experimental results on four benchmarks show MABCQ's performance improvement over BCQ.
main_review: Originality: The problem the paper studies is important and interesting. 

Quality:
1.Value deviation aims to increase the transition probabilities of the high-value next states, but it will increase the value estimate of the current state and action. It can worsen the performance in bad datasets, which is discussed in the CQL paper.
2.The evaluation of MABCQ is not sufficient, the dataset seems to be "replay" in RL domain. How does  MABCQ perform in other datasets: Random, Medium, Expert?
3.The interpretation of the effectiveness of transition normalization in Figure 3 is not solid. The difference between the maximum value and minimum value can not quantify the coordination of agents, it would be better to show a real example.

Soundness:
1. A recent MARL offline RL paper "Believe What You See: Implicit Constraint Approach
for Offline Multi-Agent Reinforcement Learning" is not discussed and compared.
2. The weakness of the proposed heuristic method is not discussed, is there any other method that can improve the coordination of agents?

summary_of_the_review: This paper studies a novel problem in MARL area. However, the proposed value deviation method is not well motivated, the experimental evaluation is only done in replay datasets, and other marl offline paper are not cited and discussed. Thus I recommend the rejection.
review_title: null

Meta-review: This paper studies the offline multi-agent RL problem. The finding is that the dataset collected by one agent could be very different for other agents. The authors provide two solutions to this problem. Although being interesting, the reviewers found that the there are many imprecise math statements, and some of the methods are not well motivated. Hence, the overall recommendation is a reject.
Decision: Reject