Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 



This paper extended Q-Learning to continuous action space by the action discretization with value decomposition in MARL. The experiment results show that the proposed method is competitive with the state-of-art continuous actor-critic methods.


strength_and_weaknesses: 

Strength:

The method of the paper is very clear. The method of applying the decentralized technique to the continuous action space problem is very intuitive. The experiment results are also sufficient.



Weakness:

The proposed method is quite simple. The author only considered the average aggregation function for the decoupled Q values. As the aggregation method is widely discussed in MARL literature and there still remains some performance gap with the state-of-the-art actor-critic methods (e.g. Humanoid Stand) , I think the paper should consider using a more complex aggregation function. 

Minor typos:

- $|\mathcal{A}|$ should be $\mathop{dim}(\mathcal{A})$.





Several Questions:

- I don't get the necessity of comparing Dreamer in Section 5.3.

- Do you consider using distribution to model the discretized action, as what is done in bang-bang policy?

- In MARL, some works (e.g. QMIX) used a monotonic function to aggregate the Q values for each decentralized action. However, it's interesting to see that the author only uses the average function (eq. 4) and performs well. Do you ever try some other monotonic aggregation function?
- Can you provide the range of the values for each dimension on some of the tasks? Dies the range vary from different dimensions?
clarity,_quality,_novelty_and_reproducibility: 

The method of the paper is very clear. The experiments are solid and sufficient to verify the effectiveness of the method. The paper conduct experiments on various benchmark tasks, including the visual input task.

Applying the decentralized method in MARL is very intuitive. I'm surprised that this method is never proposed before.

The implementation of the paper is not open-sourced. Open-source code is not compulsory, however, it could help readers understand some details of the methods.




summary_of_the_review: 

The proposed method in this paper is very simple and effective. Applying the decentralized method in MARL is very intuitive. I'm surprised that this method is never proposed before.

However, as the aggregation method is widely discussed in MARL literature and there still remains some performance gap with the state-of-the-art actor-critic methods (e.g. Humanoid Stand), I think the paper should consider using a more complex aggregation function in eq. 7.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper considers the problem of evaluating the performance of simple a simple variation of DQN (Deep Q- Networks), derived by discretizing actions across different dimensions independently, on standard continuous control tasks from the continuous control RL literature. By combining different ideas like value function decomposition and cooperative multi agent RL, the author are able to achieve performance comparable to algorithms designed for continuous control accross a variety of continuous control tasks.
strength_and_weaknesses: Strengths:
- The paper is clear and easy to follow for the most part
- The authors perform a thorough empirical evaluation and empirically validate their claim that simple independent action discretization might perform as well as complicated actor-critic architectures
Weaknesses:
- Some terms used throughout the paper need to be introduced before being used, e.g. the authors continuously use the term bang-bang discretization without defining it
- The empirical evaluation is thorough but could be improved. I would have expected to see some domains where the 2 assumptions made by the authors are clearly invalid (independent action dimension and linearity of the value decomposition) and how this affects the performance of the DecDQN compared to the continuous control agents. I suspect that continuous control agents would behave also worse when action dimensions are not independent (since generally they apply diagonal covariance matrices like the authors underline) but we might have a different result when linear value decomposition is not valid, since this is an assumption heavily used by DecDQN. T
clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to follow 
It's message is clear, and empirically verified
The paper presents limited novelty
summary_of_the_review: Overall I am positive towards the paper. The claim made is simple and thoroughly investigated empirically. The paper does present limited novelty as the ideas discussed have been discussed before in the literature but In my opinion the thorough empirical evaluation adds to the papers contributions and ICLR is a perfect venue for this kind of results. Nevertheless, I would have also expected experiments in environments that explicitly break the 2 assumptions made to define DecDQN.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper attempts to solve the problem of higher dimensional actions spaces in the context of deep reinforcement learning. The authors propose DecDQN, which consists of multiple Q-networks over different sets of the action space, resulting in reduced network parameters and potentially better performance.
strength_and_weaknesses: Strengths:
- The authors intelligently framed the solution as cooperative MARL.
- Evaluation in Section 5.1 is strong and shows that  DecDQN can coordinate across Q-networks.
- 10 seeds per agent

Weaknesses:
- The baselines are not up to date with other RL agents, such as SAC and DDPG. Would there be similar observed benefits with different algorithms and replacing the Q-networks with the paper's proposed formulation?
- Figure 6, why does 21 bin Dec DQN generally do worse than 3 bin?
- Is Sec 5.3 necessary? Dreamer-v2 would be slower due to being a model-based approach which looks ahead.
-  (Nit) want to cite the library used to train other baselines (assuming it is RLlib)
- Is there a failure case where DecDQN would fail when compared to other baselines?
clarity,_quality,_novelty_and_reproducibility: I have concerns regarding novelty and originality of the work.

Suppose that I had no concern regarding that parameters used for training (such as in foundational models). Then, how would this method compare against an ensemble of Q functions. There is also prior work in distributional RL such as QR-DQN, which approximates the distributional Q-value with multiple Q-functions. This would be better to benchmark against, as opposed to vanilla DQN.


summary_of_the_review: Overall, the paper is well written and clear. The results are logical and structured. However, there are concerns regarding novelty of the idea. I am willing to improve the score as long as the authors better address methods in distributional RL (such as QR-DQN) and add it to the results.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper focuses on evaluating the power of a critic-only approach to solving continuous control problems, by combining Q-learning with value decomposition (an approach from multi-agent RL under the *centralized training but decentralized execution* paradigm). They test this approach across a large set of environments from continuous control benchmarks and show that this simple model-free approach performs on par with SOTA actor-critic methods, both for model-free and model-based methods, and based on state features and pixel observations.   
strength_and_weaknesses: **Strengths:** 
- Exploring critic-only methods (e.g. Q-Learning) for continuous control problems is interesting because: (1) they are much simpler; (2) they allow us to have a unified approach for tackling discrete and continuous action problems; ... 

- Using discretization for tackling continuous control problems is interesting because: (1) because the agent can potentially capture non-parametric multimodal policy distributions in actor-critic methods; (2) many advances for discrete-action Q-learning (for exploration and credit assignment) can be immediately adopted; (3) curriculum learning can be achieved by growing the action space gradually; ...   

- The experiments are done on a vast and appropriate set of continuous control benchmarks. 

- The baselines capture a good range of SOTA model-free and model-based methods for the chosen benchmarks.

- Exposition of the connection of the method to multi-agent RL is appropriate and useful.


**Weaknesses:**

- The approach (i.e. Decoupled Q-Learning) is not novel: it is similar to HGQN (r=1) from (Tavakoli et al., 2021) (which the authors have referenced), but where the authors replace the **sum** mixing function with a **mean** mixer to achieve less sensitivity to learning rates in domains with varying action dimensionality. This is an appropriate choice but nonetheless one that is supported under the Action Hypergraph Networks framework.  

- Due to the above reason, I'm not a fan of the completely new name for the method in this paper - but it is not a major issue.

- The abstract makes it seem like this paper is proposing or achieving the following for the first time:

   - This paper is not the first to solve and achieve competitive performance wrt. continuous-action actor-critic methods in continuous control benchmarks via discretization and a scalable version of Q-learning (e.g. Metz et al. (2017) and Tavakoli et al. (2018) are early approaches that have done so).  

   - This paper is not the first to draw a connection between discrete multi-dimensional-action-space RL and multi-agent RL to achieve scalability wrt. increasing action dimensionality (e.g. Tavakoli et al. (2018) is an early instance in the context of Deep RL). 

   - This paper is not the first to exploit value-decomposition within Q-learning in single-agent RL (Sharma et al. (2017) and Tavakoli et al. (2021) are two instances).   
clarity,_quality,_novelty_and_reproducibility: I am happy with the quality of the experiments, and I believe the results will be easily reproducible. The baseline performances also match what I would expect (based on comparing them with published results). 

The method is not original, but I like the work in that it is the first large-scale evaluation of this super-simple method and against strong baselines (including model-based and model-free). I highly believe this paper could help bring more interest to solving continuous control problems via discretization and Q-learning. 

The exposition of the ideas is clear, and the paper cites all relevant literature. Nonetheless, considering the nature of the contribution is mostly in the large-scale evaluation and not the proposed methodology/paradigm, I believe the tone in the abstract and also the intro regarding contributions should be softened.  
summary_of_the_review: Great empirical evaluation of a simple discrete-action critic-only model-free method on a large set of continuous control benchmarks (including some complex ones) against much more involved SOTA model-free and model-free actor-critic methods for continuous control; showing competitive performance. The method is not novel, but the study is interesting and insightful and the paper should be accepted in my view due to the potential impact it can have: showing strong evidence that simple Q-learning is sufficient for solving high-dimensional continuous control problems, via discretization and combining insights from multi-agent RL to achieve scalability.

My only concern is overclaiming contributions in the abstract and intro. The authors appropriately cite all related works, to the best of my knowledge, but should tone down the claims particularly in the abstract.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a critic-only approach to solving continuous control problems, by combining value decomposition and action discretization. They test this approach across a large set of environments from continuous control benchmarks and show that this simple model-free approach performs on par with SOTA actor-critic methods, both for model-free and model-based methods, and based on state features and pixel observations.
Reviewers raised concerns about the experiments. The concerns were addressed during the rebuttal period. Overall this is a solid paper, and the AC recommends acceptance. 
justification_for_why_not_higher_score: The novelty is below the bar of spotlight.
justification_for_why_not_lower_score: The results are impressive.
Decision: Accept: poster