Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper focuses on addressing the delusional bias problem in deep Q-learning, and propose a general framework (ConQUR) for integrating policy-consistent backups with regression-based function approximation for Q-learning and for managing the search through the space of possible regressors. Specifically, it proposes a soft consistency penalty to alleviate the delusional bias problem while avoiding the expensive exact consistency testing. This penalty encourages the parameters of the model to satisfy the consistency condition when solving the MSBE problem. 

Pros:
The soft penalization itself is already shown to be effective in further improving any regression-based Q-learning algorithms (e.g., DQN and DDQN). When combining the soft consistency penalty and the search procedure, it is shown to significantly outperform the DQN and DDQN baselines, which is impressive. This work presents novel idea and solid supporting experiments.

Cons:
The major experimental results of the paper are in Table 4 of Appendix D. This is not a good effort to save space by moving the most important results into appendix. At least a selected set of results should be presented in the main paper rather than in appendices. One alternative approach is the authors plot a bar figure to demonstrate the performance of different algorithms on different Atari games.


Other comments:
•	Fig. 2 two is a bit hard to read due to too many curves.
•	Standard baseline results other than DQN and DDQN should also be listed, in order to demonstrate that solving delusional bias could make Q-learning more competitive than alternatives (e.g., A3C, PPO, TRPO).

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper presents a solution to tackling the problem of delusional bias in Deep Q-learning, building upon Lu et.al. (NeuRIPS 2018).  Delusional bias arises because independently choosing maximizing actions at a state may be inconsistent as the backed-up values may not be realizable by any policy. They encourage non-delusional Q-functions by adding a penalty term that enforces that the max_a in Q-learning chooses actions that do not give rise to actions outside the realizable policy class. Further, in order to keep track of all consistent assignments, they pose a search problem and propose heuristics to approximately perform this search. The heuristics are based on sampling using exponentiated Q-values and scoring possible children using scores like Bellman error, and returns of the greedy policy. Their final algorithm is evaluated on a DQN and DDQN, where they observe some improvement from both components (consistency penalty and approximate search).

I would lean towards being slightly negative towards accepting this paper. However, I am not sure if the paper provides enough evidence that delusional bias is a very relevant problem with DQNs, when using high-capacity neural net approximators. Further, would the problem go away, if we perform policy iteration, in the sense of performing policy iteration instead of max Q-learning (atleast in practice)? Maybe, the paper benefits with some evidence answering this question. To summarize, I am mainly concerned about the marginal benefit at the cost of added complexity and computation for this paper. I would appreciate more evidence justifying the significance of this problem in practice. 

Another comment about experiments is that the paper uses pre-trained DQN for the ConQur results, where only the last linear layer of the Q-network is trained with ConQur. I think this setting might hide some properties which arise through the learning process without initial pre-training, which might be more interesting. Also, how would other auxilliary losses compare in practice, for example, losses explored in the Reinforcement Learning with Auxilliary Tasks (Jaderberg et.al.) paper? 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: A recent paper by Lu et al introduced delusional bias in Q-learning, an error due to the max in the Bellman backup not being consistent with the policy representation implied by the greedy operator applied to the approximated value function. That work proposed a consistent algorithm for small and finite state spaces, which essentially enumerates over realizable policies. This paper proposes an algorithm for overcoming delusional bias in large state spaces. The idea is to add to the Q-learning objective a smooth penalty term that induces approximate consistency, and search over possible Q-function approximators. Several heuristic methods are proposed for this search, and results are demonstrated in Atari domains.

I found the topic of the paper very interesting - delusional bias is an intriguing aspect of Q learning, and the approach of Lu et al is severely limited to discrete and small state spaces. Thus, tackling the large state space problem is worthy and definitely not trivial. 

The authors’ proposed solution of combining a smooth penalty for approximate consistency and search over regressors makes sense. The implementation of the search (Sec 3.4) is not trivial, and builds on a number of heuristics, but given the difficulty of the problem, I expect that the first proposed solution will not be straightforward.

I am, however, concerned with the evaluation of the method and its practicality, as reflected by the following issues: 
1. The method has many hyper parameters. The most salient one, \lambda, the penalty coefficient, is changed between 0.25 to 2 on the consistency penalty experiment, and between 1 to 1000 in the full ConQUR experiments. I did not understand the order of magnitude change between the experiments, and more importantly, how can one know a reasonable \lambda, and an annealing schedule for it in advance. 
2. I do not understand the statistical significance of the results. For example, with the constant \lambda=0.5, the authors report beating the baseline in 11 out of 19 games. That’s probably not statistically significant enough to claim improvement. Also, only one run is performed for each game; adding more runs might make the results clearer. 
3. The claim that with the best \lambda for each game, the method outperforms the baseline in 16 out 19 games seems more significant, but testing an optimal hyper parameter for each game is not fair. Statistically speaking, *even if the parameter \lambda was set to a constant zero* for the 5 runs that the method is tested on, and the best performing run was taken for evaluation against the baseline, that would have given a strong advantage to the proposed method over the baseline….
4. For the full ConQUR, there are many more hyper parameters, which I did not understand the intuition how to choose. Again, I do not understand how the results establish any statistically significant claim. For example, what does: “CONQUR wins by at least a 10% margin in 20 games, while 22 games see improvements of 1–10% and 8 games show little effect (plus/minus 1%) and 7 games show a decline of greater than 1% (most are 1–6% with the exception of Centipede at -12% and IceHockey at -86%)” mean? How can I understand from this that ConQUR is really better? Establishing a clearer evaluation metric, and using well-accepted statistical tests would greatly help the paper. At the minimum, add error bars to the figures!
5. While evaluating on Atari shows applicability to large state spaces, it is hard to understand from it whether the (claimed) advantage of the method is due to the delusional bias effect, or some other factor (like implicit regularization due to the penalty term in the loss). In addition, it is hard to understand the different approximations in the method. For example, how does the proposed consistency penalty approximate the true consistency? These could all be evaluated on the simple MDP example of Lu et al. I strongly advise the authors to add such an evaluation, which is easy to implement, and will show exactly how the approximations in the approach deal with delusional bias. It will also be easier to demonstrate the effects of the different hyper parameters in a toy domain. 
review_title: Official Blind Review #3

Meta-review: While there was some support for the ideas presented, the majority of reviewers felt that this submission is not ready for publication at ICLR in its present form.

Concerns raised included the need for better motivation of the practicality of the approach, versus its computational cost. The need for improved evaluations was also raised.
Decision: Reject