Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper proposes a novel variant of Q-learning, called Maxmin Q-learning, to address the issue of overestimation bias Q-learning suffers from (variance of the reward of best action leading to overestimated reward). 
The idea is to keep a number of estimators each estimated using a different sub-sample, and taking the minimum of the (maximum) reward value of each. 
The paper gives theoretical analyses, in terms of the reduction in the overestimation bias, as well as the convergence of a class of generalized Q-learning methods including Maxmin Q-learning. 
The experiment section presents a thorough evaluation of the proposed method, including how the obtained rewards vary as a function of the variance of the reward function and as a function of learning steps, as compared to a number of existing methods such as the Double Q-learning method and its variants. 
The experimental results are quite convincing, and the theoretical analyses seem solid. 
Overall this is a well balanced paper which proposes a reasonable new idea, simple but effective, backed by sound theoretical analysis and well executed experimental evaluation. 
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new Q learning algorithm framework: maxmin Q-learning, to address the overestimation bias issue of Q learning. The main contributions of this paper are three folds: 1) It provides an inspiring example on overestimation/underestimation of Q learning. 2) Generalize Q learning by a new maxmin Q-learning by maintaining independent Q estimator and interact them in a max-min way for the update. 3) Provide both theoretical and empirical analyses of their algorithm.

I have two main concerns for this paper:
1) When is your algorithm useful? What's your criterion of picking the hyper-parameters (e.g. number of Q functions you want to learn).
2) Comparison to more intriguing way for jointly update of multiple Q functions, like soft Q learning.

For the first concern, the paper has shown an interesting example in Figure 2. However, it seems that we cannot decide whether overestimation or underestimation will help the exploration, since the reward function is often unknown in real world. And in both cases, maxmin Q learning is not the best algorithm than either Q learning and double q learning. On the other hand, if we use a softmax policy for Q function, e.g. $\pi(a|s) \propto \exp(\alpha Q(s,a))$, a drift for Q learning(e.g. Q(s,a) = Q*(s,a) + c) has no effect on our policy. I believe in this case we should more focusing on the inner difference between different value of Q function, rather than comparing our estimate Q function with the true Q*.

For the second concern, we can view the framework of maxmin q learning as a joint update scheme for different Q function. In experimental part, the comparison is not fair since the paper use multiple Q function to compare with single or double Q function. One reasonable baseline is to update N different Q function, and take the minimum of the final Q function as our decision policy, compare with maxmin Q learning with N different Q function. Another baseline the paper should consider is soft Q learning, where it maintain multiple Q function and jointly update Q different function to maximize the entropy while moving towards an improvement Q.

Overall, I believe the idea of the paper is novel and interesting, but further improvements should be added in order to improve the score the paper.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper tackles the problem of bias in target Q-values when performing Q-learning.  The paper proposes a technique for computing target Q-values, by first taking the min over an ensemble of learned Q-values and then taking the max over actions.  The paper provides some theoretical properties of this technique: (1) the bias of the estimator can be somewhat controlled by the size of the ensemble; (2) performing Q-learning with these target values is convergent. Experimental results show that the proposed technique can provide performance improvement on a number of tasks.

Overall, this paper is a modest contribution to the field, since variants of this technique are known and the theoretical arguments are derivatives of known arguments, which places it roughly borderline for an ICLR conference paper.

My comments:
-- The paper is very well-written. Thank you for putting the effort to provide clear writing.
-- The idea of computing a target value as the minimum of an ensemble is well-known in continuous control.  See https://arxiv.org/abs/1802.09477 as well as a number of works which follow it.
-- The method is motivated as a way to control over/under-estimation.  However, the theoretical arguments show that this depends on N, M, and tau (unknown). Are there any ways to choose N other than hyperparameter tuning?
review_title: Official Blind Review #1

Meta-review: The authors propose the use of an ensembling scheme to remove over-estimation bias in Q-Learning. The idea is simple but well-founded on theory and backed by experimental evidence. The authors also extensively clarified distinctions between their idea and similar ideas in the reinforcement learning literature in response to reviewer concerns.
Decision: Accept (Poster)