Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the offline multi-agent RL setting, first demonstrating that optimization is more likely to find bad local optima than in the single agent case. To deal with this problem, the authors propose to add zeroth order optimization to multi-agent training and provide a theorem guaranteeing that this approach leads to safe improvements. The authors conduct extensive experiments and ablations in comparison to relevant baselines on the multi-agent particle environments to demonstrate the efficacy of their approach as a function of the type of data used for offline training. The authors also provide experiments on a slightly larger scale tackling the multi-agent half-cheetah environment. 
main_review: I thought that the paper was pretty well written and did a pretty good job of motivating the fact multi-agent optimization is more likely to fall into bad local minima than single agent optimization. However, I am only so convinced by empirical examples and think the paper can be improved by motivating this fact more based on the theory of MARL i.e. the non-stationary underlying optimization process. I did appreciate, however, the theoretical contribution of Theorem 1 demonstrating that the approach leads to sound updates. Zeroth order optimization has been considered in the RL literature, so its application is not hugely novel in this context. Moreover, I can't shake the feeling that zeroth order optimization is somewhat indirectly related to the multi-agent/offline RL problems specifically. 

The experiments were relatively small scale, but these things are harder in the multi-agent literature and the complexity considered is fairly respectable in comparison to related prior work. I found the experiments provided to be relatively thorough, addressing a lot of key questions I had about the approach. One question I had though was if the bolds in Table 3 are merely higher numbers or statistically significant.


summary_of_the_review: I generally like this paper and am in favor of acceptance. I think the paper presents a pretty good narrative and addresses the noted optimization issues with a somewhat novel and theoretically grounded approach. The experiments are pretty comprehensive although admittedly of very limited scale. What stops me from giving a higher score is mostly the somewhat disconnected nature of the proposed contribution to the theory of MARL, which is at this point mostly based on intuitive/empirical findings.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work looks at the problem of training multi-agent reinforcement learning with continuous action space in an offline setting. The authors identified a saddle point issue of the value function landscape in the existing offline MARL/RL methods, which causes the actor policy to be stuck in a bad local optimum. The proposed method samples and evaluates different actions based on a Gaussian function, and adds a regularizer to the actor loss to encourage the actor policy to take the action with a high Q-value. 

main_review: The authors proposed a simple but seemingly effective method for improving MARL with offline datasets. The paper is overall well written and easy to follow. I think the paper is looking at an important problem and contributing to the development of offline MARL, however, I have a few questions:

1. OMAR uses a regularizer to encourage the actor policy to take action with a high Q-value. However, if the local Q-value gradient direction is opposite compared to the selected sampled action, would the actor policy get stuck in some suboptimal actions (e.g. in the middle of the two high Q-value points).
2. Regarding using the Gaussian function sampling, I wonder if its performance would be affected by the dataset. Since CQL tries to reduce the Q-values of unknown actions, depending on the dataset, if the value function landscape has multiple picks (e.g. mix Gaussian), then the sampling method may result in suboptimal action. Can this issue be handled by this method?
3. Does this value function saddle issue happens only in the offline setting? Can you provide some intuition regarding why the issue is more severe in an offline setting?
4. It would be nice to also include the score of the online learning for the expert dataset.
5. How does the size of the training dataset affects the benefit of this method. Does the low optimal issue go away when you have more data points?

summary_of_the_review: The authors identified a concrete problem in offline MARL training and proposed a method that is addressing the issue. The evaluation shows that overall the proposed method outperforms the existing baselines.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a simple yet effective multi-agent offline reinforcement learning algorithm, called Offline Multi-Agent RL with Actor Rectification, which combines CQL (first-order optimization) and evolutionary algorithms (zero-order optimization) to enhance the applicability of the CQL algorithm on multi-agent tasks.
main_review: ## strengths

The local optimum problem is successfully solved using zero-order optimization for offline training in the multi-agent scenario for the first time.

The paper is clear and easy to understand, and the approach is simple and effective.

## weaknesses


As mentioned in the paper, offline learning on multi-agent tasks is very common but poorly studied. Although the approach proposed in this paper achieves significant performance, the following issues are not explained or investigated very clearly.

1. Regarding the motivation, although MAPPO is successful, it is undeniable that derictly applying the single-agent algorithm to multi-agent tasks may probabily perform not well (e.g. indepdent Q learning). So it is not surprising that MA-CQL performance is poor. The authors claim that this is due to the local optimum, but there is not sufficient experimental arguments to illustrate that. The simple illustration given in Figure 1.d may not appropriate to represent the relationship between the local optimum and the global optimum, because a higher Q value may be caused by overestimation.

2. Regarding the method, zero-order optimization can be applied not just under multi-agent tasks. So I think a key ablation experiment to illustrate the contribution of zero-order optimization to offline reinforcement learning is performing zero-order optimization based on single-agent CQL.

3. There are some details about the methodology that are worth discussing.
3.1 Why does zero-order optimization not consider the minimum of multiple Q's?
3.2 The authors argue that solely regularizing the critic is insufficient for multiple agents to learn good policies for coordination in the offline setting, but in fact, the authors do not analyze coorparation explitily. So is this really the key issue of CQL?

4. There are also some details about the experiment that are worth discussing.
4.1 Why the performance variance in Figure 3 is so large.
4.2 Why is the learning rate set so large (0.01) in the training process? An explanation is needed, since the learning rate is usually set to 3e-4 in CQL or SAC.
4.3 In Table 5, why is the performance worse under the CTDE paradigm? This is somehow contradictory to the conclusion of MADDPG, so a more sufficient explanation is needed.

Overall, this paper addresses important question, but is not well researched. So, I consider the paper to be at the level of borderline for now. I would be happy to raise the score if the authors could more clearly verify the motivation through experiments, and discuss the advantages that zero-order optimization can bring under single-agent offline reinforcement learning tasks. I look forward to the follow-up of this work.

summary_of_the_review: Overall, this paper addresses important question, but is not well researched. So, I consider the paper to be at the level of borderline for now, and I recommend marginally below the acceptance threshold (5).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work considers extending conservatism-based algorithms to offline RL with multi agents. The performance of standard algorithms often degrades significantly in this setting, especially when the number of agents increase. To resolve the issue, the authors propose a simple scheme which essentially combines first-order and zeroth-order policy optimization methods, with the goal to possibly extract the advantages of each method. Some empirical results demonstrate that the proposed algorithm can achieve better performance than standard baselines.

main_review: The paper is overall clearly written. The intuition is discussed and the main idea is easy to follow with enough explanations. It is a bit surprising to see that such a simple combination of first-order and zeroth-order methods is able to deliver a good performance. The algorithm can be appealing given it seems quite easy to implement.

I do however have some comments:
(1) Theoretical: the authors state a theorem regarding the performance bound and argue that "OMAR gives a safe policy improvement guarantee". However, I do not see how "safe" the policy is. Can the authors provide more explanations about the bound? How large the lower bound could be? Is there easy example that one can get some concrete results? It's fine that the theorem is not strong since I consider this as primarily a practical paper, but some explanation is needed to avoid over-claiming the results..
(2) Practical: I think the empirical evaluation is not adequate. In particular, in the beginning, it is motivated that standard methods fail when there are more and more agents. The experiments, while confirming the efficacy of the proposed method, largely focus on cases where there are few agents.I appreciate the detailed ablation study the authors provide for different parameters, but I think a curve showing how the method compares as the number of agents increase to a large number is useful. The proposed method also doesn't seem to have scalability issue to run such an experiment.
summary_of_the_review: Overall, I think this paper contains some interesting and useful ideas. Some parts can be further improved as mentioned before and I hope that the author response will address them.
review_title: null

Meta-review: This paper makes a key observation that the gradient-based method gets more likely to suffer from poor local optima in multi-agent reinforcement learning (MARL) with more agents particularly in the offline setting.  The paper proposes the use of zeroth order optimization method to avoid local optima.  Specifically, it samples multiple actions and regularize the policy to get closer to the optimal action among those.  The use of such zeroth order method to avoid poor local optima is not particularly new, although finding its effective in MARL and the empirical support are valuable.  The main discussion point was the insufficiency of experimental support, and the additional experiments during the discussion have addressed the original concerns of the reviewers to some extent.  Overall, given the limited novelty and inefficiency of support (either theoretical or empirical), the paper is slightly below the borderline.
Decision: Reject