Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposed AdaMa, which can automatically use adaptive learning rates for each agent in cooperative Multi-Agent Reinforcement Learning (MARL). AdaMa calculated the learning rate of each actor and critic according to their contributions of locally increasing value functions.  Simple experiments using toy examples show that the proposed AdaMa method can improve fixed learning rate method and other heuristics.

Pros:
1. The topic and idea of using adaptive learning rates to avoid hand-tuning are quite interesting and important. I think the related topics are worth investigating.
2. The proposed AdaMa method looks reasonable to me, at least from an intuitive perspective.
3. Experimental results also look promising.

Cons:
1. The experiments look simple, and the improvements seem to be incremental rather than significant (please correct me if I misunderstood or missed something). I think more experiments on larger scale tasks are needed to make the effectiveness of the proposed method convincing.
2. Using first- and second-order Taylor expansion to obtain the best possible learning rates seems to be a reasonable idea. However, I think some more rigorous theoretical understanding is worth pursuing to show the benefits of the proposed method in a more convincing way, e.g., the speed of convergence for fixed and adaptive learning rate methods.

Overall, I found the problem and the idea important and interesting. The proposed method is intuitively reasonable and verified by small scale experiments. However, the proposed method is not convincing in terms of the lack of larger-scale experiments or theoretical results.
review_title: reasonable idea but the study is not thorough

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:


The paper proposes a new algorithm for multi-agent reinforcement learning (MARL) that adaptively picks learning rates for actor and critic. Specifically, the learning rates are updated to directions maximally affecting the Q-function, and the algorithm dynamically balances the learning rates between actor and critic. In numerical studies, the authors illustrate the efficiency of their method via four toy experimental scenarios and intuitively explain the underlying mechanism.


------------------------------------------------------------------------------------------


Pros:

+ The choice of learning rates in MARL is an interesting and important issue.
+ The paper is well written. The methodology part is clearly organized and easy to follow. The learning rate balance between actor and critic is well motivated.
+ The numerical experiments are well designed. Each model represents a different cooperation mode. The results are well presented.


------------------------------------------------------------------------------------------

Cons:

- The numerical results are not satisfying. The improvement in AdaMa is incremental. In Figure 3 (a), (b) and (d), AdaMa has similar performance with Fixed lr (fixed learning rate).
- Currently each model is trained for 5 runs. More experiments are needed for more reliable results.
- The authors should evaluate the performance of AdaMa in more practical models in addition to these four toy examples.

review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes an algorithm called AdaMa for multi-agent reinforcement learning (MARL). Based on the contribution of the critic and actors, the algorithm adopts adaptive learning rates. Numerical experiments are provided in four cooperation scenarios to show the performance of AdaMa.


In my view, the paper currently falls below the bar for an ICLR publication. The detailed comments are as follows:

- The key concern about this paper is the lack of rigorous analysis. The contents in Section 3 is largely based on heuristic approximations. There is no rigorous analysis, statement or proof. And there is theoretical guarantees of the performance of the algorithm with respect to the sample complexity or time complexity.

- In addition, there is no comparison with state-of-the-art algorithms, which limits the contribution. It would be much better if authors can include this either in terms of theory or numerical experiments.

- The paper is not well written. For example, there is no formal description of the proposed algorithm.
review_title: Review for "Adaptive Learning Rates for Multi-Agent Reinforcement Learning"

final_score: rating: 4: Ok but not good enough - rejection
review_content: ---
Summary

This paper studies automatic learning rate tuning in Multi-Agent Reinforcement Learning (MARL). It proposes AdaMa, an algorithm which balances the learning rates of actors and the critic, and can also make use of the second-order information. Experiments show that AdaMa can learn agents faster. 


---
Writing Quality

The paper is not well-written as I don't fully understand it. Notations are used without definition ($\delta$ in Section 3.3) and inconsistent (some $Q$ has arrow above it and some doesn't) , and $\frac{\partial Q}{\partial \theta}$ has a confusing shape. Simpler symbols can also be used for readability (e.g., $\widehat{\|\overrightarrow{l_a}\|}$ can be replaced by something like $\eta_a$). The derivation of $\Delta Q$ can be deferred to appendix. 

I'd suggest the authors to make an algorithm box for the full algorithm. Equation (1), (2) and (3) might be confusing, as they're describing algorithms, not mathematical equations. 


---
Comments


I'm not fully convinced. The figures indeed show that AdaMa uses different LR for different actor. Why is this helpful? Why should we slow down the learning of an actor if it's already good? 

How does Adam, one of the most widely used optimizer, work? 
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper studies adaptive learning rate for Actor-Critic style MARL algorithm. I have several questions:

1. How is the adaptive learning rate related to MARL? It seems this is just a general improvement for actor-critic methods?
2. This is closely related to the first question. Some optimization algorithms with adaptive learning ate are mentioned in the related work section. However, it is not clear why these methods cannot solve the problem and we need new techniques. Or, what structure in MARL makes it possible to achieve further improvement? Is this possible for, say single-agent setting?
3. I am not very familiar with the Dec-POMDP setting so I have a stupid question: shouldn't the policy a function of the state instead of the observation? Otherwise, we will need to work with the significantly larger observation space instead of the state space?

review_title: The paper studies adaptive learning rate for Actor-Critic style MARL algorithm. I have several questions:

Meta-review: This paper investigates how to deploy adaptive learning rates in multi-agent RL (MARL). In particular, the learning rates are adaptively chosen based on which directions maximally affect the Q-function, and take into account the interplay and balance between the actors and the critics. The topic is certainly of great interest when designing fast-convergent MARL algorithms. However, the reviewers point out the inadequacy and insufficiency of empirical gains in the reported experiments. Also, larger-scale experimental settings are needed in order to provide more convincing evidence about the practical benefits of the proposed scheme.   
Decision: Reject