Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper establishes the asymptotic convergence of on- and  off- policy DPG in the multi-agent setting under some assumptions. Overall the paper is well written and easy to follow. Given the practical usefulness of DPG, I think the result in this paper is somehow meaningful to the community.

However, the paper has the following issues:
(1) Since this paper is a theoretical paper, the assumptions are very important to evaluate the signficance of the main theorems. Thus, the assumptions should be moved to the main body of the paper.

(2) The projection is a practical issue that has been criticized by many researchers in the RL community. Although it is necessarily to make this assumption when handling two time-scale algorithm with time-varying MDP, it can be avoid by adopting other updating structures such as nested-loop, mini-batch two time-scale, etc. Or can the author provide an upper for such a projection radius so that the algorithm can at least be implemented in practice?

(3) Although this paper is a theoretical paper, only considering multi-agent bandit in the experiement is not sufficient to verify the theoretical results in this paper. I suggest the author to consider some RL settings to make the experiment results stronger.

(4) The current muti-agent setting in this paper is not that practical, as too many variables are requied to be globally observable.

(5) The update at time step t required the knowledge in step (t+1), which could introduce some pratice issues as this paper focus on online update. Can the author propose some potential idea to fix this issue?

=== After rebuttal ===

Since the technical contribution of this paper is not significant enough, I will keep my score as weakly reject.
review_title: Meaningful results but not strong enough

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a solution to learning deterministic policy in the multi-agent RL setting, where local rewards are private to local agents.  Both on-policy learning and off-policy learning are considered in the paper. 
To do so, the author first extends the policy gradient theorem in MARL for stochastic policy to deterministic policy.
Specifically, in on-policy learning, only global Q-function and local action are needed to compute local policy gradient; in off-policy setting, only averaged reward function and local action are needed.
The author then propose a decentralized algorithm for learning the global Q-function/averaged reward function. Two-timescale technique is adopted to show the convergence of the proposed algorithms.

Strengths: 
(1) the extension of policy gradient to deterministic policy in MARL (locally observable reward) is interesting and important.
(2) the decentralized algorithm itself, although similar to prior work in Zhang et al (2018), provide needed guarantee to the algorithm for convergence.
(3) the paper is well written and easy to follow.

Cons:
The proposed decentralized algorithm closely mimics the one in the prior work, in addition to its analysis framework.  I hope the authors can provide more details on how the analysis of the algorithm differentiates itself from previous work in different aspects. I am happy to increase my rating if the author can address this issue in their response.
review_title: This paper proposes two algorithms for learning deterministic policy in multi-agent RL problem. The authors extend the multi-agent policy gradient for stochastic policies to deterministic policies. Decentralized algorithms are proposed to learn the Q/averaged reward in on/off-policy setting.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper offers a comprehensive theoretical treatment of deterministic policy gradients in a multi-agent setting, working out several key results:
* existence and explicit formulas for the multi-agent deterministic policy gradient in off and on-policy settings;
* convergence of stochastic policy gradients to deterministic ones as policy variance converges to zero; 
* convergence of multi-agent deterministic actor-critic algorithms.

This paper is very well-written and easy to follow, although technically there is not much to expose here, as most of the paper consists of statements of results. I did not check all the math thoroughly, but there does not seem to be any cause for suspicion as far as I can tell. Related work is sufficiently well presented. 

I think that the topic of deterministic policy gradient in a multi-agent is sufficiently interesting and having the foundational results worked out in a single reference is a valuable contribution. The paper is incremental in a way that the deterministic results seem to be extensions of known stochastic results along the lines of well-understood techniques. If any unexpected challenges had to be addressed in order to derive these results (e.g. developing new techniques), this has to be made clearer in the paper. 

Overall I think this is a solid paper.


review_title: comprehensive but somewhat incremental

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper extends the results for actor-critic with stochastic policies of [Zhang, ICML 2018] to deterministic policies and offers the proof of convergence under some specific assumptions. The authors consider both the on-policy setting and the off-policy setting and offers some convincing derivation. It provides a valuable idea and a promising direction in MARL, but the current version has several problems that need to be fixed. Specifically, some parts of equations, algorithms, and expressions are ambiguous and unintelligible. Besides, problems with the format in the formula and citations also exist, which degrade the paper’s quality and clarity.

Pros:
1.	The motivation that considers both off-policy and on-policy is interesting and attractable. This work might provide a promising way to address the problem of inefficiency of exploration in MARL. 
2.	The convergence of MARL is challenging, and this paper gives a valuable attempt.

Cons:
1.	The experiment is insufficient and doubtful, although this paper is convincing in theory, the experiments are not convincing at all. The convergence curves are quite nice, but the authors should compare the results with some other state-of-art algorithms to increase its credibility. 
2.	The differences with [Zhang, ICML 2018] are ambiguous. Some definitions and formulas and even algorithms are almost the same as [Zhang, ICML 2018]. 
3.	There are lots of format problems. For example, “Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. Fully decentralized multiagent reinforcement learning with networked agents. 80:5872–5881, 10–15 Jul 2018.” In it, I cannot find where this paper is published. Some symbols are used without explanation, e.g., t in Algorithms 1 and 2.
4.	The organization of this paper needs to be polished. In the background section, the authors firstly introduce the optimization problem of maximizing the average reward with the objective $J(\pi_\theta)$ on page 2. However, they use $J(\theta)$ when defining the Q_\theta function instead of $J(\pi_\theta)$ just 3 lines below. This simplification is not noted until Section 3. So the authors should re-organize the paper to make the content more readable.
5.	This paper raises some assumptions when discussing, but neither does it offer a list of these assumptions, nor does it analyze the rationality and influence of them.

Questions:
1.	What is the difference between Figure 1 and Figure 2? 
2.	What is the detail of the experiment? 
3.	How well do the results compared with [Zhang, ICML 2018]?
4.	In Algorithm 1, why can you draw an action firstly?

review_title: This paper provides a valuable idea and a promising direction in MARL, but the current version has several problems that need to be fixed. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: Clarity: The paper is well written. The following require some clarifications:
1) $\tilde{a}$ variables are defined by never seem to be used in Algorithm 1, why?

2) In the  off-policy setting (Theorem 1) letting $\mathcal{N}=1$ does not recover the previous result by Silver et al. (January 2014a). This is perhaps due to the fact that eq(1) of this paper has $\bar{R}$ in its objective and eq(14) of Silver et al. (January 2014a) has $Q^{\text{target-policy}}$ instead. It would be great if the authors clarify why the two objectives are different. If my understanding is correct, in the off-policy setting, we are still interested in the maximising the reward with respect to target policy, and the restriction is that the samples are from the behaviour policy.  


Quality:
1) Strength: The theoretical results are stated clearly and detailed proofs have been provided. But for the off-policy case, other results seem to be correct.
2) Weakness: The experiments are on a toy domain. The experiments demonstrate that the proposed algorithms indeed converge as expected. However, the experiments do not add any further insights such as a) what happens when we change the communication matrix $C$? b) What happens when the reward structure varies a lot across the agents?  c) what happens when different agents start from different parts of the state space. In short, the experiments have to be carefully thought out to cover the entire range so as to emphasise the multi-agent flavour. As it stands, the current version of the paper spends too little space (only about 4 lines) has been allocated for the experiments. 


Novelty:
The deterministic policy gradient result is new. The off-policy case seems to have some issue (see above). While the result in itself is new, it is not clear whether it follows trivially from prior results. It would be great if the authors can comment on the this.


Significance:
The significance of the current version is limited due to, 
1) the lack of experiments, 
2) the possibility that the new results could be obtained as trivial extension of prior work (Zhang ICML,2018 + Silver et al. (January 2014a)). (I am not saying that this is true, authors can clarify)


review_title: Detailed theory, however experiments lack coverage

Meta-review: The paper offers a direction for mult-agent RL that builds on results for actor-critic methods [Zhang, ICML 2018], extending that work to address deterministic policies.  The authors establish convergence under a number of assumptions.   Both on-policy setting and off-policy settings are treated.  The reviewers point out several concerns and the consensus seems to be that, while the direction looks promising, the paper deserves further work.

Decision: Reject