ConQUR: Mitigating Delusional Bias in Deep Q-Learning
Authors: DiJia-Andy Su
Jayden Ooi
Tyler Lu
Dale Schuurmans
Craig Boutilier‎
Venue:  (2020)
Keywords: reinforcement learning
q-learning
deep reinforcement learning
Atari

Abstract: Delusional bias is a fundamental source of error in approximate Q-learning. To date, the only techniques that explicitly address delusion require comprehensive search using tabular value estimates. In this paper, we develop efficient methods to mitigate delusional bias by training Q-approximators with labels that are "consistent" with the underlying greedy policy class. We introduce a simple penalization scheme that encourages Q-labels used across training batches to remain (jointly) consistent with the expressible policy class. We also propose a search framework that allows multiple Q-approximators to be generated and tracked, thus mitigating the effect of premature (implicit) policy commitments. Experimental results demonstrate that these methods can improve the performance of Q-learning in a variety of Atari games, sometimes dramatically.

Under review as a conference paper at ICLR 2020
CONQUR: MITIGATING DELUSIONAL BIAS IN DEEP Q L-LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Delusional bias is fundamental source of error in approximate Q-learning. To date, the only techniques that explicitly address delusion require comprehensive search using tabular value estimates. In this paper, we develop efficient methods to mitigate delusional bias by training Q-approximatons with labels that are "consistent" with the underlying greedy policy class. We introduce simple penalization scheme that encourages Q-labels used across training batches to remain (jointly) consistent with the expressible policy class. We also propose search framework that allows multiple Q-approximators to be generated and tracked, thus mitigating the effect premature (implicit) policy commitments. Experimental results demonstrate that these methods can improve the performance of Q-learning in a variety of Atari games, sometimes dramatically.
INTRODUCTION
Q-learning WWtkins & Dayan 1992 Sutton & Barto 2018) lies at the heart of many of the recent successes of deep reinforcement learning (RL) (Mnih al. 2015 Silver al. 2016). with recent
advancements (c.g.. van Hasseli (2010); Bellemare et al. (017)); Wang et al. (0166) Hessel et al. (2017) helping to make among the most widely used methods in applied RL. Despite these successes, many properties of Q-learming are poorly understood, and it is challenging to successfully apply deep Q-learning in practice. When combined with function approximation. Q-learning can become unstable (Baird. 1995 Boyan & Moore 1995 Tsitsiklis & Roy, 1996 Sutton & Barto 2018). Various modifications have been proposed to improve convergence or approximation error (Gordon 1995 1999 Szepesvári & Smart 2004 Melo & Ribeiro 2007 Maci et al. 2010 Munos et al. 2016);
but it remains difficult to reliably attain both robustness and scalability.
Recently, Lu al. (2018) identified source of error in Q-learning with function approximation known as delusional bias. It arises because Q-learnig updates the value of state-action pairs using estimates of (sampled) successor-state values that can be mutually inconsistent given the policy class induced by the approximator. This can result in unbounded approximation error, divergence, policy cycling, and other undesirable behavior. To handle delusion, the authors propose a policy-consistent backup operator that maintains multiple Q-value estimates organized into information sets. Each information set has its own backed-up Q-values and corresponding "policy commitments" responsible for inducing these values. Systematic management of these sets ensures that only consistent choices of maximizing actions are used to update Q-values. All potential solutions are tracked prevent premature convergence on any specific policy commitments. Unfortunately, the proposed algorithms use tabular representations of Q-functions, so while this establishes foundations for delusional bias, the function approximator is used neither for generalization nor to manage the size of the state/action space. Consequently, this approach is not scalable to RL problems of practical size.
In this work, we propose CONQUR (CONsistent Q- Q-Update Regression), general framework for integrating policy-consisterr backups with regression- based function approximation for Q-learning and for managing the search through the space of possible regressors (i.e., information sets). With suitable search heuristics, our framework provides a computationally effective means for minimizing the effects of delusional bias in Q-learning while admitting scaling to practical problems.
Our main contributions are as follows. First we define novel augmentations of standard Q Q-regression to increase the degree of policy consistency across training batches. While testing exact consistency

---

Under review as a conference paper at ICLR 2020
is expensive, we introduce efficient soff-consistency penalty that promotes consistency of new labels with earlier policy commitments. Second, drawing on the information-see structure of et al.) (2018), we define search space over Q-regressors to allow consideration of multiple sets of policy commitments. Third, we introduce heuristics for guiding the search over regressors, which is critical given the combinatorial nature of information sets. Finally, we provide experimental results on the Atari suite (Bellemare et al. 2013)) demonstrating that CONQUR can offer (sometimes dramatic) improvements over Q-learning. We also show that (easy-to-implement) consistency penalization on its own (i.e., without search) can improve over both standard and double Q-learning.
2 BACKGROUND
We assume a discounted, infinite horizon Markov decision process (MDP), = (S, R. ?)- The state space S can reflect both discrete and continuous features, but we take the action space A to be finite (and practically enumerable). We consider Q-learning with a function approximator Qa to learn an (approximately) optimal Q-function (Watkins] 1989; Sutton Barto 2018), drawn from some approximation class parameterized by è (c.g., the weights of a neural network). When the approximator is a deep network, we generically refer the algorithm DQN, the method at the heart of many recent RL successes (Mnih et al. 2015 (ilver et al., 2016).
For online Q-learning. at a transition s, r, s', the Q-update is given by:
0 - 0 + a(r + max Qo a') (o((s.a))oooolss... a'eA
(1)
Batch versions of Q-learning, including DQN, are similar, but fit a regressor repeatedly to batches of training examples (Ernst et al.) 2005 Riedmiller 2005). Batch methods are usually more data efficient and stable than online Oelarning Abstractly, batch Q-learning works through a sequence of (possibly randomized) data batches D1.-. ... Dr to produce sequence of regressors Qe...... Qo, Qs. estimating the Q-function! the Q-label q = r + y max,' Qo (s', a'). We then fit Qo, to this training data using suitable regression procedure with an appropriate loss function. Once trained, the (implicit) induced policy Ty is the greedy policy w.r.t. Qo. i.c., (s) = arg maX.e Qs (s. a). Let F(O), resp. G(()) be the corresponding class of expressible Q-functions, resp. greedy policies.
| For each (s, a, r, 8') E Da. we use prior estimator Qa,_, to bootstrap
Intuitively, delusional bias occurs whenever a backed-up value estimate is derived from action choices that are not (jointly) realizable in G(O) (Lu et al. 2018). Standard Q-updates back up values for each (s, a) pair by independently choosing maximizing actions at the corresponding next states s'. However, such updates may be "inconsistent" under approximation: if no policy in G(@) can jointly express all past action choices, backed up values may not be realizable by any expressible policy. Lu eu et al.) (2018) show that delusion can manifest itself with several undesirable consequences. Most critically, it can prevent Q-learning from learning the optimal representable policy in G(O); it can also cause divergence. To address this, they propose a non-delusional policy consistent Q-learning (PCQL) algorithm that provably eliminates delusion. We refer to the original paper for details, but review the main concepts we need consider below.
The first key concept is that of policy consistency. For any SCC. S, an action assignment as : + A associates an action (s) with each s E S. We say a is policy consistent if there a greedy policy E G(() s.t. (s) = (s) for all E S. We sometimes equate a set SA of state-action pairs with an implied assignment π (s)) = a for all (8, a) E SA. If SA contains multiple pairs with the same state s, but different actions a, it is a multi-assignment (though we loosely use the term "assignment" in both cases when there is no risk of confusion).
In (batch) Q-learning, each successive regressor uses training labels generated by assuming maximiz- ing actions (under the prior regressor) are taken at its successor states. Let σ reflect the collection of states and corresponding maximizing actions taken to generate labels for regressor Qo, (assume it is policy consistent). Suppose we train Qo, by bootstrapping on Q and consider a training sample (s, a, r, s'). Q-learning generates label r + y max, Q (s', aa) for input (s, a). Notice,
We describe our approach using a straightforward form of batch Q- Q-learning. but it can accommodate many variants, e.g., where the regressor used for bootstrapping is some earlier Q Q-estimator, or the estimators generating the max-actions and the value estimates are different as in double Q-lcarning (van Hasseli 2010 Hasselt et al. 2016); indeed, we experiment with such variants.
2

---

Under review as a conference paper at ICLR 2020
however, that taking action a" a Q8, (s', a') at s' may not be policy consistent with o Thus Q-learning will estimate a value for (s, assuming the execution of a policy that cannot be realized given the limitations of the approximator. The PCQL algorithm (Lu et al., 2018) prevents this by insisting that any action assignment a used to generate bootstrapped labels is consistent with earlier assignments. Notice that this means Q-labels will often not be generated using maximizing actions relative to the prior regressor.
The second key concept is that of information sets. One will generally not be able to maximizing actions to generate labels, so tradeoff's can be made when deciding which actions to assign to different states. Indeed, even if it feasible to assign a maximizing action to state s early in training, say at batch k, since it may prevent assigning a maximizing a' to s' later, say batch k + f, we may want to consider a different assignment to 8 to give more flexibility to maximize at other states later. PCQL. doesn't it try to anticipate the trdeoffs--rthee corresponding to a different assignment to the states seen in the training data so far. Each gives rise to different Q-function estimate, resulting in multiple hypotheses. At the end of training, the best hypothesis is the one maximizing expected value w.r.f. an initial state distribution.
it maintains multiple information sets, each
PCQL provides strong convergence guarantees, but it is a tabular algorithm: the function approximator retricts the policy class, but is not used to generalize Q-values. Furthermore, its theoretical guarantees come at cost: it uses exact policy consistency tractabac for linear approximators, but not practical for large problems; and maintains all consistent assignments. As result, PCQL. cannot be used for large RL problems of the type tackled by DQN.
THE CONQUR FRAMEWORK
We develop the CONQUR framework to provide a practical approach reducing delusion in Q- learning, specifically addressing the limitations of PCQL identified above. CONQUR consists of three main components: practical soft-constraint penalty that promotes policy consistency; a search space to structure the search over multiple regressors (information sets, action assignments); and heuristic search schemes (expansion, scoring) to find good Q-regressors.
3.1 PRELIMINARIES
We assume of training data consisting of quadruples (s,,,,,''),, divided into (possibly non- disjoint) batches Di,... Dr for training. This perspective is quite general: online RL corresponds to [D, 1: off-line batch training (with sufficiently exploratory data) corresponds to single batch (i.c.. T = 1); and online or batch methods with replay are realized when the D; are generated by sampling some data source with replacement.
For any data batch D, let x(D) {s' (s, E D} denote the collection of successor states of D. An action assignment oD for D an assignment (or multi- austti-assignment) from xD) to A: this dictates which action (s'))) ) is considered "maximum" for the purpose of generating a Q-label for pair (s, a); i.e., (8, a) will be assigned training label r + 7Q(s', (ss)) rather than
r + y max ((s', a'). The set of all such assignments E(D) = Ax(D); note that grows exponentially with D).
Given Q-function parameterization O we say oD is -ccosisttent (w.rt. D) if there is some E θ s.t. no (s') = os8') for all s' E F This is simple policy consistency, but with notation that emphasizes the policy class. Let Ee(D) denote the set of --consistent assignments over D. The union o Udz of two assignments (over D1, D2, resp.) is defined in the usual way.
3.2 CONSISTENCY PENALIZATION
Enforcing strict Ô Occonsistency as regressors 01, .2.....88 Or are generated is computationally chal- lenging. Suppose assignments oI.... Okk-- 0k-1 used generate labels for Di,... Dx-1, are jointly O-consistent (let 0-k-1 denote their multi-set union). Maintaining O-consistency when generating 0g imposes two requirements. First, one must generate an assignment dı over D₃ s.L. 0gk-1 U Ja is consistent. Even testing assignment consistency can be problematic: for linear approximators this is
We suppress mention of D when clear from context implicd by the assignment under considcration.
3

---

Under review as a conference paper at ICLR 2020
linear feasibility program (Lu et al. 2018) whose constraint set grows linearly with DD ....U Dyl- For DNNs, this is complex, and much more expensive, polynomial program. Second, the regressor 8g should itself be consistent with d=k-1 U og. Again, this imposes significant constraint the regression optimization: in the linear case, this becomes a constrained least-squares problem (solvable, e.g., quadratic program); while with DNNs, it could be solved, say, using much more complicated projected SGD. However, the sheer number of constraints makes this impractical.
Rather than enforcing consistency, we propose a simple, computationally tractable scheme that "encourages" it: a penalty term that can be incorporated into the regression itself. Specifically, we add penalty function the usual squared loss to encourage updates of the Q-regressors to consistent with the underlying information set, i.c., the prior action assignments used to generate its labels.
When constructing &s, let D**= UfD; : S k). and E Ee (Dex) be the collective (possibly multi-) assignment used to generate labels for all prior regressors (including 0% itself). The multiset of pairs B {(s,, (ss))sss' xx) called a consistency buffer. The collective assignment need not be consistent (as we elaborate below), nor does the regressor O2 need to be consistent with a. Instead, we incorporate the following soft consistency penalty when constructing 0z:
Cols',a) Σ 1Qo(s' -ollaaa---Q___aoooo a'EA
Co
Σ (d'
Cols',
where [x]] max(0,
x).. This penalizes Q-values of actions at state that are larger than that of action 0 (s). We note that is O-consistent if and only if minsee Co Co(B) O. We incorporate this penalty into our regression loss for batch Dy:
B)
E (s,a,T,,")ED,
[r+
-
ACo(B).
(2)
Here Qo is prior estimator on which labels are bootstrapped (other prior regressors may be used). The penalty effectively acts "regularizerr on the squared Bellman error, where À controls the degree of penalization, allowing a tradeoff between Bellman error and consistency with the action assignment used to generate labels. It thus promotes consistency without incurring the expense of testing strict consistency. It is a simple matter to replace the classical Q-learning update (1) with one using a consistency penalty:
0₂ 1-
Σ (s,o,ro)))
afr + +500 (o(1))))))((s..))V01...
+aXVoCo(B) 0=0--1
(3)
This scheme is quite general. First, it is agnostic how the prior action assignments are made, which can be the standard maximizing action at each stage w.r.t. the prior regressor like in DQN, Double DQN (DDQN) (Hasselt al., 2016), or other variants. It can also be used in conjunction with search through alternate assignments (see below).
Second, the consistency buffer B may be populated in variety of ways. Including all max-action choices from all past training batches promotes full consistency in an attempt to minimize delusion. However, this may be too constraining since action choices early in training are generally informed by very inaccurate value estimates. Hence, B may be implemented in other ways to focus only on more recent data (e.g., with a sliding recency window, weight decay, or subsampling): and the degree of recency bias may adapt during training (c.g., becoming more inclusive training proceeds and the Q-function approaches convergence). Reducing the size of B also has various computational benefits. We discuss other practical means of promoting consistency in Sec.
The proposed consistency penalty resembles the cemporal-consittene loss of Pohlen et al. (2018). but our aims very different. Their temporal consistency notion penalizes changes in next state's Qeestimate over all actions, whereas we discourage inconsistencies in the greedy policy induced by the Q-estimator, regardless of the actual estimated values.
4

---

Under review as a conference paper at ICLR 2020
D1
D2
Q
i
2
2
@
1
Q1
2
Fig. 1: A generic search tree.
3.3 THE SEARCH SPACE
Ensuring optimality requires that PCQL track all O-consistent assignments. While the set of such assignments is shown to be of polynomial size (Lu et al. 2018), it is still impractical to track this set in realistic problems. As such, in CONQUR we recast information set tracking as a search problem and propose several strategies for managing the search process. We begin by defining the search space and discussing its properties. We discuss search procedures in Sec. 3.4
As above, assume training data is divided into batches DI.... Dr and we have some initial Q- function estimate 0, (for bootstrapping D1 's labels). The regressor 0g for D₃ can, in principle, be trained with labels generated by any assignment a E Ee(Dz) of actions to its successor states x x(Ds), necessarily maximizing actions w.r.t. 0z-1- 1. Each a gives rise to a different updated Q-estimator Og. There are several restrictions we could place on "reasonable" o-candidates: (i) a is O-consistent; (ii) a is jointly O-consistent with all J. for j < k, used construct the prior regressors on which we bootstrap -1: (iii) a is not dominated by any o' E Ee (Dx), where we say o' dominates if Q+ (s', o' (s')) 2 Q (s', (s')) for all s' E (D), and this inequality strict for at least one s Conditions (i) and (ii) are the strict consistency requirements of PCQL.. We will, however, relax these below for reasons discussed in Sec. 3.2 Condition (iii) is inappropriate in general, since we may add additional assignments (e.g., to new data) that render all non-dominated assignments inconsistent, requiring that we revert to some dominated assignment.
This gives us a generic search space for finding policy-consistent, delusion-free Q-function, as illustrated in Fig. Each node ni at depth & in the search tree associated with a regressor 8% defining Qo and action assignment of that justifies the labels used to train 0f (of can also be viewed as an information set). We assume the root no is based on an initial regression 0g. and has an empty action assignment du- Nodes at level it of the tree defined follows. For each node ni_1 at level 1 with regressor 6%__1 and O-consistent assignment oi we generate a child n for each of Ee (D(())) Di)) such that of 1 U of is O-consistent. Node ni assignment is of U of and its regressor 0f is trained using the following data set:
((s,a) a) r ~Q0=_ (s',o (s,a,r,s') E D&).
The entire search space constructed in this fashion to maximum depth of T. See Appendix B Algorithm T for pseudocode of a simple depth-fin fiistt recursive specification.
The exponential branching factor in this search tree would appear to make complete search intractable; however, since we only allow ecconsistent "collective" assignments we can bound the size of the tree- it is polynomial in the VC-dimension of the approximator.
Theorem 1.
The number of nodes in the search tree is no more than O [(TT) m) CCDDmi(()) where VCDim(-) is the VC-dimension [Vapnik 1998) of a set of hoolean-valued functions, and G is the set of boolean functions defining all feasible greedy policies under O:
g = {go(s, a, d') == l[fo(s. - fo(s, a') > 0), Vs, d' | [
(4)
linear approximator with a fixed set features induces a policy- indicator function class & with VC-dimension d, making the search tree polynomial in the size of the MDP. Similarly, a fixed ReL.U
5

---

Under review as a conference paper at ICLR 2020
DNN architecture with W weights and L layers has VC-dimension of size L W) again rendering the tree polynomially sized.
Even with this bound, navigating the search space exhaustively is generally impractical. Instead, various search methods can be used to explore the space, with the aim of reaching a "high quality" regressor at some leaf of the tree (i.e., trained using all T data sets/batches). We discuss several key considerations in the next subsection.
3.4 SEARCH HEURISTICS
Even with the bound in Theorem[]] traversing the search space exhaustively is generally impractical. Moreover, as discussed above, enforcing consistency when generating the children of a node, and their regressors, may be intractable. Instead, various search methods can be used to explore the space, with the aim of reaching a "high quality" regressor some (depth T) leaf of the tree. We outline three primary considerations in the search process: child generation, node evaluation or scoring, and the search procedure.
Generating children. Given node n there are, in principle, exponentially many action assign- ments, or children, (Dx) (though Theorem I significantly limits the number of children if we enforce consistency). For this reason, we consider heuristics for generating a small set of children. Three primary factors drive these heuristics.
The first factor preference for generating high-value assignments. To accurately reflect the intent of (sampled) Bellman backups, we prefer to assign actions to state s' E (Dx)) with larger predicted Q-values over actions with lower values, i.e., a preference for over a' if Qug. (s'.a) >
Qot a'). However, since the maximizing assignment may be --inconsistent (in isolation, or jointly with the parent's 's information set, or with future assignments), candidate children should merely have higher probability of high-value assignment. The second factor is the need to ensure diversity in the assignments among the set of children. Policy commitments at stage k constrain the possible assignments at subsequent stages. In many search procedures (e.g., beam search), we avoid backtracking, so we want the policy commitments possible in later stages. The third is the degree to which we enforce consistency.
make at stage k to offer much flexibility as
There are several ways to generate such high- high-value assignments. We focus on just one natural technique: sampling action assignments using a Boltzmann distribution. Specifically, let a denote the assignment (information set) of some node (parent) at level k - 1 in the tree. We can generate an assignment dg for D3 as follows. Assume some permutation si,..., S of (Dx).. For each sf in
turn, we sample a; with probability proportional to e 1 This can be done without regard to consistency, in which case we would generally use the consistency penalty when constructing the regressor 0g for this child to "encourage" consistency rather than enforce it. If we want strict consistency, we can use rejection sampling without replacement to ensure a; is consistent with
%/1 U 0gi-1 (we can also use a subset of oi_1 as a less restrictive consistency buffer) The temperature parameter controls the degree to which focus purely maximizing assignments versus more diverse, random assignments. While stochastic sampling ensures some diversity, this procedure will bias selection of high-value actions states s' x(Dx) that occur early in the permutation. To ensure sufficient diversity, we use a new random permutation for each child.
Scoring children. Once the children of some expanded node are generated (and, optionally, their regressors constructed), we need some way of evaluating the quality of each child as means of deciding which new nodes are most promising for expansion. Several techniques can be used. We could use the average Q-label (overall, or weighted using some initial state distribution), Bellman error, or loss incurred by the regressor (including the consistency penalty or other regularizer). However, care must be taken when comparing nodes at different depths of the search tree, since deeper nodes have a greater chance to accrue rewards or costs-simple calibration methods can be used. Alternatively, when simulator is available, rollouts of the induced greedy policy can be used evaluate the quality of a node/regressor. Notice that using rollouts in this fashion incurs considerable computational expense during training relative to more direct scoring based on properties on the node, regressor, or information set.
Notice that at least one action for state of must be consistent with any previous (consistent) information

---

Under review as a conference paper at ICLR 2020
Search Procedure. Given any particular way of generating/sampling and scoring children, variety of different search procedures can be applied: best-first search, beam search, local search, etc. all fit very naturally within the CONQUR framework. Moreover, hybrid strategies are possible- one we develop below is a variant of beam search which we generate multiple children only at certain Q-regression at the intervening levels. This reduces the size of the search tree considerably and, when managed properly, adds only constant-factor (proportional beam size) slowdown standard Q-learning methods like DQN.
levels of the tree, then do "deep dives" using consistency-penaliie
3.5 A CONCRETE INSTANTIATION OF THE CONQUR FRAMEWORK
We now outline a specific instantiation of the CONQUR framework that can effectively navigate the large search space that arises in practical RL settings. We describe a heuristic, modified beam- search strategy with backtracking and priority scoring. Pseudocode is provided in Algorithm (see Appendix B ; here we simply outline some of the key refinements.
Our search process grows the tree breadth-firss manner, and alternates between two phases. In an expansion phase, parent nodes are expanded, generating one or more child nodes with action assignments sampled from the Boltzmann distribution, For each child, we create target Q-labels, then optimize the child's regressor using muusistnccyppnnaliie Bellman error (Eq, 2 our loss. We thus forego strict policy consistency, and instead "encourage" consistency in regression. In a dive phase, each parent generates one child, whose action assignment is given by the usual max-actions selected by the parent node's 's regressor as in standard Q-learning. No additional diversity is considered in the dive phase, but consistency is promoted using consistency-penaliie regression.
From the root, the search begins with an expansion phase to create e children-ct is the splitting factor. Each child inherits its parent's s consistency buffer from which add the action assignments that were used to generate that child's Q-labels. To limit the size of the tree, we only track a subset of the children, the frontier nodes, selected using one of several possible scoring functions. We select the top Ê-nodes for expansion, proceed to a dive phase and iterate.
It is possible move beyond this "beam--ike" approach and consider backtracking strategies that will return to unexpanded nodes at shallower depths of the tree. We consider this below as well.
3.6 RELATED WORK
Other work has considered multiple hypothesis tracking in RL. One particularly direct approach has been use ensembling, where multiple Q-approximators are updated in parallel (Faußerr & Schwenker 2015 Oshand et al. 2016; Anschel et al.) 2017) then combined straightforwardly to reduce instability and variance. An alternative approach has been to consider population-basee methods inspired by evolutionary search. For example, Conti et al. (2018) combine novelty-search and quality diversity technique to improve hypothesis diversity and quality in RL. Khadka & Tumer (2018) consider augmenting an off-policy RL method with diversified population information from an evolutionary algorithm. Although these techniques do offer some benefit, they do not systematically target an identified weakness of Q-learning, such delusion.
EMPIRICAL RESULTS
We experiment using the Atari test suite (Bellemare et al., 2013) to assess the performance of CONQUR. We first assess the impact of using the consistency penalty in isolation (without search) as a "regularizer" that promotes consistency with both DQN and DDQN. We then test the modified beam search described in Appendix B to assess the full power of CONQUR.
4.1 IMPACT CONSISTENCY PENALIZATION
We first study the effects of introducing the soft-policy consistency in isolation, augmenting both DQN and DDQN with the consistency penalty term. We train our models using an open-source implementation (Guadarrama al.) 2018)) of both DQN and DDQN (with the same hyperparameters) We call these modified algorithms (QN(A) and DDQ. respectively, where is the penalty coefficient defined in Eq. Note that X recovers the original methods. This is lightweight
7

---

Under review as a conference paper at ICLR 2020
Grodarr
Allen nidees
Alle 16 moles
In CandJ
CondiRip-2d
Solars 5 rodes
Solarb nodes
11:M Conqueaane))
201
112
Fig. 2: Varying penalization X (no search procedure).
Fig. 3: Effects of increasing the number of nodes.
Selabb
Alann
30
25
2.
Allenck
14
72
0.0
Fig. 4: Policy value of nodes, sorted.
Fig. 5: Training curves of 8 (of 16) sorted nodes.
Fig. 6: Effects of A.
modification that can be applied readily to any regression-based Q-learning method, and serves to demonstrate the effectiveness of soft-policy consistency penalty. Since we don't consider search (i.e., don't 't track multiple hypotheses), we maintain a small consistency buffer using only the current data batch by sampling from the replay buffer-this prevents getting "trapped" by premature policy constraints. As the action assignment is the maximizing action of some network, o(s') can be computed easily for each batch. This results in a simple algorithmic extension that adds only an additional penalty term to the original TD-loss.
We train and evaluate (Q) and (DQN(A) for = {0.25, 0.5, 1, 2, on 19 Atari games. In training, À is initialized to 0 and slowly annealed to the desired value avoid premature commitment to poor action assignments. Without annealing, the model tends fit to poorly informed action assignments during early phases of training, and thus fails to learn a good model.
The best À is generally different across games, depending on the nature of the game and the extent of delusional bias. Though single > 0.5 works well across all games tested, Fig. illustrates the effect of increasing À on two games. In Gravitar, increasing À generally results in better performance for both DQN and DDQN, whereas in SpaceInvaders, but performance starts to degrade for À = 2.
0.5 gives improvement over both baselines,
We compare the performance of the algorithms using each À value separately, as well as using the best X for each game. Under the best X, (A) and DDQN(A) outperform their "potentially delusional" counterparts on all except 3 and 2 games, respectively. In 9 of these games, each of DQN (A) and DDQN(X) beats both baselines. With constant À = 0.5, each algorithm still beats their respective baseline in games. These results suggest that consistency penalization (independent of the general CONQUR model) can improve the performance of DQN and DDQN by addressing the delusional bias that is critical to learning a good policy, Moreover, we see that consistency penalization seems to have different effect on learned Q Q-values than double Q-learning, which addresses maximization bias. Indeed, consistency penalization, when applied to DQN, can achieve gains that are greater than DDQN (in 15 games). Third, in games (DQN(A) provides additional performance gains over (QN()).
detailed description of the experiments and further results can be found in Appendix
8

---

Under review as a conference paper at ICLR 2020
Rollouts
Bellman Consistency Penalty
BattleZone
33796.30
32618.18
BeamRider
9914.00
10341.20
Boxing
83.34
83.03
Breakout
379.21
393.00
MsPacman
5947.78
5365.06
Seaquest
2848.04
3000.78
Spacelnvader
3442.31
3632,25
StarGunner
55800.00
56695.35
Zaxxon
11064.00
10473.08
Table I: Results of CONQUR with (split 2) nodes on games using the proposed scoring function compared to evaluation using rollouts.
4.2 CONQUR RESULTS
We test the full CONQUR framework using the modified beam search discussed above. Rather than training a full Q-network, for effective testing of its core principles, we leverage pre-trained networks from the Dopamine package Castro al. (2018) These networks have the same architecture as in Mnih et al. (201)) and are trained on 200M frames with sticky actions using DQN. We use CONQUR to retrain only the last (fully connected) layer (implicitly freezing the other layers), which can be viewed as a linear Q-approximator over the features learned by the CNN. We run CONQUR using only 4M addtional frames train our Q-regressors)
We consider splitting factors e of 2 and 4; impose limit on the frontier size of 8 or 16; and an expansion factor of or 4. The dive phase is always of length 9 (i.e., 9 batches of data), giving an expansion phase every 10 iterations. Regressors are trained using the loss in Eq. 2 and the consistency buffer comprises prior action assignments. (See Appendix D for details, hyperparameter choices and more results.)
We run CONQUR with À = {1, 10, 100, 1000} and select the best performing policy. We initially test two scoring approaches, policy evaluation using rollouts and scoring using the loss function (Bellman error with consistency penalty). Results comparing the two on a small selection of games shown in Table [ While rollouts, not surprisingly, tend to give rise better- peitterpprformingg policies, consistent-Bellmann scoring is competitive. Since the latter much less computationally intense, and does not require sampling the environment, we use it throughout our remaining experiments.
We compare CONQUR with the value of the pre-trained DQN. We also evaluate a "multi-DQN" baseline that applies multiple versions of DQN independently, warm-starting from the same pre- trained DQN. It uses the same number of frontier nodes as CONQUR, and is otherwise trained identically as CONQUR but with direct Bellman error (no consistency penalty). This gives DQN the same advantage of hyytthessss tracking as CONQUR but without policy consistency.
We test on 59 games, comparing CONQUR with frontier size 16 and expansion factor 4 and split- ting factor 4 (16-4-4) with backtracking (as described in the Appendix D) resulted in significant improvements the pre-trained DQN, with an average score improvement of 125% (excluding games with non-positive pre-trained score). The only games without improvement are Montezuma's 's Revenge, Tennis, PrivateFye and BankHeist. This demonstrates that, even when simply retraining the last layer of highly tuned DQN network, removing delusional bias has the potential to offer strong improvements in policy performance. It is able exploit the reduced parameterization to obtain these gains with only 4M frames of training data. Roughly, half-dozen games have outsized score improvements, including Solaris (1 times greater value), Tutankham (6.5 times) and WizardOFWor (5 times) 6
Compared to the stronger multi-DQN baseline (with 16 nodes), CONQUR wins by at least a 10% margin in 20 games, while 22 games see improvements of 1-10% and 8 games show little effect
*See https: /
//gittttubbb cwn/ googl e dopamine
*This approach is simply to reduce the computational and memory footprint our experiments. The framework is not limited to this approach.
"This may he in part, but not fully, due the sticky-action training of the pre-trained model.

---

Under review as a conference paper at ICLR 2020
(plus/minus 1 1%) and 7 games show decline of greater than 1% (most are 1-6% with the exception of Centipede at -12% and IceHockey at -86%). Results are similar when comparing CONQUR and multi-DQN each with 8 nodes (8-2-2): 9 games exhibit 10%+ improvement, 21 games show 1-8% improvement, 12 games perform comparably and 7 games do worse under CONQUR. A table of complete results appears in Appendix D.3 Table 4 and training curves (all games, all A) in Fig.
Increasing the number of nodes from 8 to 16 generally leads to better performance for CONQUR, with 38 games achieving strictly higher scores with 16 nodes (16-4-4): 16 games with 10%+ improvement, 5 games tied and the remaining 16 games performing worse (only a few with 5%+ decline). Fig. shows the (smoothed) effect of increasing the number of nodes for a fixed = 10. The y-axis represents the rollout value of the best frontier node (i.e.. the greedy policy of its Q-regressor) as a function of the training iteration. For both Alien and Solaris, the multi-DQN (baseline) training curve is similar with both 8 and 16 nodes, but CONQUR improves Alien from 3k to 4.3k while Solaris improves from 2.2k to 3.5k.
Fig. and Fig. 5 (smoothed, best frontier node) shows node policy values and training curves, respectively, for Solaris. When considering nodes ranked by their policy value, comparing nodes of equal rank generated by CONQUR and by multi-DQN (baseline), we see that CONQUR nodes dominate their multi-DQN counterparts: the three highest-ranked nodes achieve a score improvement of 18%. 13% and 15%, respectively, while the remaining nodes achieve improvements of roughly 11-12%. Fig. 6 (smoothed, best frontier node) shows the effects of varying X. In Alien, increasing from to 10 improves performance, but it starts decline for higher values of 100 and 1000. This is similar to patterns observed in [.1 and represents a trade-off between emphasizing consistency and not over-commiting to action assignments. In Atlantis, stronger penalization tends to degrade performance. In fact, the stronger the penalization, the worse the performance.
5 CONCLUDING REMARKS
We have introduced CONQUR, a framework for mitigating delusional bias in value-based RL that relaxes some of the strict assumptions of exact delusion-free algorithms to ensure scalability. Its two main components are (a) a tree-search procedure used to create and maintain diverse, promising Q-regressors (and corresponding information sets); and (b) consistency penalty that encourages "maximizing" actions to be consistent with the FA class. CONQUR embodies elements of both value-based and policy-based RL: it can be viewed using partial policy constraints to bias the value estimator or as a means of using candidate value functions to bias the search through policy space. Empirically, we find that CONQUR can improve the quality of existing approximators by removing delusional bias. Moreover, the consistency penalty applied on its own, directly in DQN or DDQN, itself can improve the quality of the induced policies.
Given the generality of the CONQUR framework, there remain numerous interesting directions for future research. Other methods for nudging regressors to be policy-consistent include exact consistency (constrained regression), other regularization schemes that bias the regressor to fall within the information set, etc. Given its flexibility, more extensive exploration of search strategies (e.g., best first), child-generation strategies, and node scoring schemes should be examined within CONQUR. Our (full) experiments should also be extended beyond those that warm-start from a DQN model, as should testing CONQUR in other domains.
Other connections and generalizations are of interest as well. We believe our methods can be extended to both continuous actions and soft max-action policies. We suspect that there is a connection between maintaining multiple "hypotheses" (i.c., Q-regressors) and notions in distributional RL. which maintains distributions over action values Bellemare et al. (2017).
REFERENCES
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning. arXiv: 611 0192019990 929, 2017.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Proceedings of the International Conference on Machine Learning (ICMI.955, pp. 30-37, 1995.
10

---

Under review as a conference paper at ICLR 2020
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, June 2013.
Marc Bellemare, Will Dabney, and Remi Munos. A distributional perspective reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML-17), 2017.
Justin A. Boyan and Andrew W. Moore. Generalization reinforcement learning: Safely approx- imating the value function. In G. Tesauro, D. Touretzky, and T. K. Leen (eds.), Advances in Neural Information Processing Systems 7 (NIPS-94), MIT Press, Cambridge, 1995.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: research framework for deep reinforcement learning. arxiv: 1812.06110
[cs. 2018.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via 2018.
population of novelty-seeking agents. arXiv:1712.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
Stefan Faußer and Friedhelm Schwenker. Neural network ensembles reinforcement learning.
Neural Processing Letters, 2015.
Geoffrey Gordon.
Approximation Solutions to Markov Decision Problems. PhD) thesis, Carnegie Mellon University, 1999.
Geoffrey J. Gordon. Stable function approximation in dynamic programming. In Proceedings of the Twelfth International Conference on Machine Learning (ICML-95), pp. 261-268. Lake Tahoe, 1995.
Sergio Guadarrama, Anoop Korattikara, Pablo Castro Oscar Ramirez, Ethan Holly, Sam Fishman, Ke Wang, Ekaterina Gonina, Neal Wu, Chris Harris, Vincent Vanhoucke, and Eugene Brevdo. TF-Agents: A library for reinforcement learning in tensorflow. https://sg http///ttttuucccooo com/ tensorflow/aeee agentss 2018. htRlss/tss:/// / www.///esoooll/ooooocoooo agent.s
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement with double q-learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 16,' pp. 2094 2100. AAAI Press, 2016. URL acmp:///d.ac...om http://dal..ac.ocrretaaaaaaa..oo 3016191
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski. Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. arxiv: 02298, 2017.
Shauharda Khadka and Kagan Tumer. Evolution-guidee policy gradient in reinforcement learning. In Advances in Neural Information Processing Systems 31 (NeurIPS-18), Montreal, 2018.
Tyler Lu, Dale Schuurmans, and Craig Boutilier. Non-delusional Q-learning and value iteration. In Advances in Neural Information Processing Systems 31 (NeurIPS-Je) Montreal, 2018.
Hamid Maei, Csaba Szepesvári. Shalabh Bhatnagar, and Richard Sutton. Toward off-policy learning control wtih function approximation. In International Conference on Machine Learning, Haifa, Israel, 2010.
Francisco Melo and M. Isabel Ribeiro. Q-learning with linear function approximation. In Proceedings of the International Conference Computational Learning Theory (COLT)) pp. 308-322 2007.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Marc Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Science, 518:529-533. 2015.
11

---

Under review as a conference paper at ICLR 2020
Remi Munos, Thomas Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems 29 (NIPS-16), Barcelona, 2016.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in Neural Information Processing Systems 29 (NIPS-16), 2016.
Tobias Pohlen, Bilal Piot, Todd Hester, Mohammad Gheshlaghi Azar, Dan Horgan, David Bud- den, Gabriel Barth-Maron, Hado van Hasselt, John Quan, Mel Vecerík, Matteo Hessel, Rémi Munos, and Olivier Pietquin. Observe and look further: Achieving consistent performance on atari. CoRR, htss/8805.1199,, 2018. URL http::/aaxx....zz///////B/
arXiv: 1805 1159.
Martin Riedmiller.
Neural fitted iteration first experiences with data efficient neural reinforce- ment learning method. In Proceedings of the 16th European Conference on Machine Learning, pp. 317-328, Porto, Portugal, 2005.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwiesser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game Go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 2018.
Csaba Szepesvári and William Smart. Interpolation-based Q-learning. In Proceedings of the International Conference on Machine Learning (ICML-04), 2004.
John H. Tsitsiklis and Benjamin Van Roy. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42:674-690, 1996.
Hado van Hasselt. Double q-learning. In Advances in Neural Information Processing Systems 23 (NIPS-1O), pp.
Vancouver, BC, 2010.
Vladimir N. Vapnik. Statistical Learning Theory. Wiley-Interscience, September 1998.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Mare Lanctot, and Nando de Freitas. Dueling network architectures for deep reinforcement learning. In Proceedings of the International Conference on Machine Learning (1CML-16), 2016.
Christopher C. H. Watkins and Peter Dayan. Q-learning. Machine Learning, 8:279-292, 1992.
Christopher John Cornish Hellaby Watkins. Learning from Delayed Rewards. PhD thesis, King's College, Cambridge, UK, May 1989.
A AN EXAMPLE OF DELUSIONAL BIAS
We describe example, taken directly from (Lu et al. 2018), to show concretely how delusional bias causes problems for Q-learning with function approximation. The MDP in Fig. Fig.Zsillsstaatt the phenomenon: [u et al. (2018) linear approximator over specific set of features in this MDP to show that:
(a) No π E G(O) can express the optimal (unconstrained) policy (which requires taking 02 at each state);
(b) The optimal feasible policy in G(O) takes a at 81 and 02 at 84 (achieving a value of 0.5).
(c) Online Q-learnin (Eq. I with data generated using e-greedy behavior policy must converge to a fixed point (under a range rewards and discounts) corresponding to a "compromise" admissible policy which takes di both 8 and (value of 0.3).
12

---

Under review as a conference paper at ICLR 2020
$1
02
82
83
84
02
R 02)
a1 prob. q
01 prob. 1 R()).a))
a1
a1
a1
Fig. 7: A simple MDP (Lu et al. 2018
Algorithm CONQUR SEARCH (Generic, depth-first)
Input: Data sets DR: DR+1.... Dr; regressor QK-1; and assignment over Dek-1 Uisisk-1 D2 reflecting prior data; policy class O.
1: Let Ee.. = fox E Ee(D,) oR U consistent]
2: for all of Ea.a do
3: Training set t 0
4: for all (s, r,s') E D2 do
5:
g+ ((=1((s, =-SSU(((sa))))
6:
7:
end for
Train Q using training set S
8:
9:
if = 7' then
10:
Return Q terminate else
12: Return SEARCH(DE#1.... Dr: Q/s of Uo; θ) recurse
end if
14:
end for
Q-learning fails to find reasonable fixed-point because of delusion. Consider the backups at (82, a2) and (83, a2). Suppose 0 assigns a "high" value to (#3, 02), so that Qg(83, a2) >> Qg(83, 01) required by Tg-- They show that any such also accords "high" value to (s2, a2). But Q5(82; 02) > (2(s2,a)) a1) is inconsistent the first requirement. As such, any update that makes the Q-value of (#2, higher undercuts the jastification for it to be higher (i.e., makes the "max" value of its successor state ($3, 02) lower). This occurs not due to approximation error, but the inability of Q-learning find the value of the optimal representable policy.
B ALGORITHMS
The pseudocode (depth-first) version of the CONQUR search framework listed in Algorithm 1 As discussed in Sec. 3.5 a more specific instantiation of the CONQUR algorithm is listed in Algorithm.
C ADDITIONAL DETAIL: EFFECTS OF CONSISTENCY PENALIZATION
C.I DELUSIONAL BIAS IN DQN AND DDQN
Both DQN and DDQN uses a delayed version of the Q-network Qs- but in different way. In DON, (s' a') is used for both value estimate and action assignment (s)N(s') = argmax, Qo (s', .a), whereas in DDQN, Qo (s',a') is used only for value estimate and the action assignment is computed from the current network ODDON (s') argmax, Qo, (s', a').
(ssss''''' aa) for label generation,
With respect to delusional bias, action assignment of DQN consistent for all batches after the latest network weight transfer, ds))(s') is computed from the same Q (s', a') network. DDQN, on the other hand, could have very inconsistent assignments, since the action is computed from the current network that is being updated every step.
13

---

Under review as a conference paper at ICLR 2020
Algorithm 2 Modified Beam Search Instantiation of CONQUR Algorithm
Input: Search control parameters: m, L, d, T
1: Maintain list data batches DI,...; Di. initialized empty
2: Maintain candidate pool P of at most nodes, initialized P {no}
3: Maintain frontier list F of L nodes
4: Maintain for each node ni, regressor lg and ancestor assignment oj
5: for each search level k < T do
6: Find top scoring node n' P
7: Use e-greedy policy extracted from Qp to collect next data batch D₃
8: if is expansion level then
9: Select top & scoring nodes n',...,"' n' E P for each selected node n² do Generate с children n²'1 ,.... n using Boltzmann sampling on De with Qu:
11:
for each child n² do
13:
Let assignment history g. be of U [new assignment) Determine regressor Ahd by applying update (3) from end for
15:
16:
Score and add child nodes to the candidate pool P Assign frontier nodes to set of child nodes, F' {n²
17:
18:
[P| > m then
19:
evict bottom scoring nodes, keeping top in P
20:
end end for
21:
22:
end if
23: if is a refinement ("dive") level then
for each frontier node n² e do
25:
Update regressor (i by applying update (3 to 0* end for
27: end if
28: Run d "dive" levels after each expansion level
29:
end for
C.2 TRAINING METHODOLLGGY AND HYPERPARAMETERS
We implement consistency penalty on top of the DQN and DDQN algorithm by modifying the open- source TF-Agents library (Guadarrama al. 2018). In particular, we modify existing DqnAgent and DdqnAgent by adding consistency penalty term to the original TD loss.
We TF-Agents implementation of DQN training Atari with the default hyperparameters, which are mostly the same as that used in the original DQN paper (Mnih et al. (015). For conveniece to the reader, some important hyperparameters are listed in Table 2 The reward is clipped between [-1, 1, 1] following the original DQN.
C.3 EVALUATION METHODOLOGY
We empirically evaluate our modified DQN and DDQN agents trained with consistency penalty on 15 Atari games. Evaluation is using the training and evaluation framework for Atari provided in TF-Agents without any modifications.
C.4 DETAILED RESULTS
Fig. shows the effects of varying À on both DQN and DDQN. Table 3 summarizes the best
penalties for each game and their corresponding scores. Fig. showss the training curves of the best penalization constants. Finally, Fig. 10 shows the training curves for a fixed penalization of À = 1/2. The datapoints in each plot of the aforementioned figures obtained by taking windows of size 30
14

---

Under review as a conference paper at ICLR 2020
Hyper-parameter
Value
Mini-batch size
32 million transitions
Replay buffer capacity
Discount factor
0.99
Optimizer
R.MPProp
Learning rate
0.00025
Convolution channel
32, 64, 64
Convolution filter size Convolution stride
(8 x 8). (4 x 4), (3 3) 4, 2,
Fully-connecteo hidden units
512
Train exploration Eraiin
0.01
Eval exploration Eeval
0.001
Table 2: Hyperparameters for training DQN and DDQN with consistency penalty on Atari.
Atlamt
Deam
Bextng
2.4
1.
1-4
4.5
022
1x2
1ed
3.4
3.
-6.6
2.0
65
-14
60
Narms7Haaamm
Ozert
Semquent
Spacenvaters
Strruuner
2.0
2.0
TineFict
12
Tutendfam
VideoPinball
Zasoon
167
2006
M
3008/011.0
359
535
325
359
5.5
359
3.5
335
Fig. 8: DQN and DDQN training curves different penalty constant
steps, and within each window, we take the largest policy value (and over =2-5 multiple runs). This done to reduce visual clutter.
D ADDITIONAL DETAIL: CONQUR RESULTS
Our results use a frontier queue of size (P)) 8 or 16 (these are the top scoring leaf nodes which receive gradient updates and rollout evaluations during training). To generate training batches, we select the best node's s regressor according to our scoring function, from which we generate training samples (transitions) using e-greedy. Results are reported in Table and [ and related figures where number of nodes are 8 or 16. We used Bellman error plus consistency penalty as our scoring function. During the training process, also calibrated the scoring account for the depth difference between
15

---

Under review as a conference paper at ICLR 2020
DQN
Abest
DQN (best)
BDQN
best
DDQN( Xeas
Assault
2546.56
1.5
3451.07
2770.26
1
2985.74
Atlantis
995460.00
0.5
1003600.00
940080.00
1.5
999680.00
BattleZone
67500.00
2
55257.14
47025.00
2
48947.37
BeamRider
7124.90
0.5
7216.14
5926.59
0.5
6784.97
Boxing
86.76
0.5
90.01
82.80
0.5
91.29
Breakout
220.00
0.5
219.15
214.25
0.5
242.73
Enduro
1206.22
0.5
1430.38
1160.44
1
1287.50
Gravitar
475.00
1.5
685.76
462.94
1.5
679.33 1
JourpeyEscape
1020.599
0.25
-696.47
-794.71
-692.35
MsPacman
4104.59
2
4072.12
3859.64
0.5
4008.91
NameThisGame
7230.71
9013.48
9618.18
0.5
10210.00
Qbert
13270.64
0.5
14111.11
13388.92
1
12884.74 74
Seaquest
5849.80
1
6123.72
12062.50
7969.77
SpaceInvaders
2389.22
0.5
2707.83
3007.72
0.5
4080.57
StarGunner
40393.75
0.5
55931.71
55957.89
0.5
60035.90
TimePilot
4205.83
2
7612.50
6654.44
2
7964.10
Tutankham
222.76
1
265.86
243.20
0.25
247.17
VideoPinball
569502.19
0.25
552456.00
509373.50
0.25
56296
1.50
Zaxxon
5533.33
10520.00
7786.00
0.5
10333.33
Table 3: Consistency penalty ablation results on best penalty constants for DQN and DDQN.
Assaul
AlMani
BoamBiike
Biainn
33
3.5
CA
3.0
0.2
2.4
-2.4
15
-2.5
1.
3.0
150
NarmeTHaaee Hanee
1044
Ozart
Seazuent
140
Spaceinaccera
144
StarGumer
3.0
655
0.0
0.2
TimeFRit
Tutarkaa
Vides
Z2x00n
3.a
3.8
2.0
6.6
5.4
Inokowe
MM
Fig. DQN and DDQN training curves for the respective best and baseline.
the leaf nodes at the frontier versus the leaf nodes in the candidate pool. We calibrated by taking the mean of the difference between scores the current nodes in the frontier with their parents. We scaled this difference by multiplying with constant of 2.5.
16

---

Under review as a conference paper at ICLR 2020
145
140
143
1.5
22
8.0
0m
2.0
2.0
3.
Breakous
Enntum
Grawiar
00
juumeyEscape
1.5
Hohaeman
-0.
3.0
2.
6.6
14
C2
10
3.0
Qper
Seanuest
3.0
Spacernwaeers
StarGumerr
6.0
2.0
66-
3.5
Timetket
Tutorkham
VidesFittal!
000
635
359
075
3.00
335
0.00
035
359
075
3.0
325
0.00
633
359
075
3.00
305
800
635
859
075
1.00
335
20080-1200
Fig. 10: DQN and DDQN training curves for À = 0.5 and the baseline.
In our implementation, we initialized our Q-network with pre-trained DQN. We start with the expansion phase. During this phase, each parent node splits into children nodes and the Q-labels are generated using action assignments from the Boltzmann sampling procedure, in order to create high quality and diversified children. We start the dive phase until the number of children generated is at least F. In particular, with F 16 configuration, and first iterations, and then at every tenth iteration starting at iteration 10, then at 20, and so on until ending at iteration 90. In the F 8 configuration, the expansion phase occurred the zero-th and first iterations, then at every tenth iterations starting at iterations 10 and 11, then at iterations 20 and 21, and so on until ending iterations 90 and 91, All other iterations execute the "dive" phase. For every fifth iteration, Q-labels are generated from action assignments sampled according to the Boltzmann distribution. For all other iterations, Q-labels are generated in the same fashion as the standard Q-learning (taking the max Q-value). The generated Q-labels along with the consistency penalty are then converted into gradient updates that applies one or more generated children nodes.
performed the expansion phase the zero-th
D.1 TRAINING METHODOLOGY AND HYPERPARAMETERS
Each iteration consists of 10k transitions sampled from the environment. Our entire training process has 100 iterations which consumes 1M transitions or 4M frames. used RMSProp as the optimizer with learning rate of 2.5 x 10 One training iteration has 2.5k gradient updates and used a batch size of 32. We replace the target network with the online network every fifth iteration and reward is clipped between [-1, 1]. We discount value of 0.99 and e-greedy with 0.01 for exploration. Details of haper-parameter settings can be found in Table ( 16 nodes) and Table (for 8 nodes).
17

---

Under review as a conference paper at ICLR 2020
D.2 EVALUATION METHODOLOGY
We empirically evaluate our algorithms on 59 Atari games (Bellemare et al. 2013). and followed the evaluation procedure as in Hasselt al. (2016). We evaluate agents every 10th iterations (and also the initial and first iteration) by suspending our training process. We evaluate on 500k frames, and we cap the length of the episodes for 108k frames. We used e-greedy as the evaluation policy with 0.001.
D.3 DETAILED RESULTS
shows training curves of CONQUR with 16 nodes under different penalization strengths Each plotted step of each training curve (including the bascline) shows the best performing node's policy value as evaluated with full rollouts. achieved for all 59 games for CONQUR and the baseline under 8 and 16 nodes. Table Sjowows similar summary, but without no-op starts (i.e. policy actions are applied immediately). Both the baseline and CONQUR improve overall, but CONQUR'S advantage over the baseline is amplified. This may suggest that for more deterministic MDP environments, CONQUR may have even better improvements. The results on 16 and 8 nodes use splitting factor of 4 and 2, respectively.
Fig. I1
shows the summary of the highest policy values
18

---

Under review as a conference paper at ICLR 2020
CONQUR (8 nodes)
CONQUR (16 nodes)
Baseline (8 modes)
Baseline (16 nodes)
Checkpoint
AirRaid
11365.00
11627.68
9578.
6962.21
Alien
3585.00
4340.42
3327.33
3213.07
2440.52
Amidar
552.90
632.63
655.54
668.477
207.08
2005.68
1971.44
2002.59
2050.48
Asterix
6094.21
6395.56
5589.39
4984.82
2476.85
Asteroids
1313.21
1395.48
1224.42
1366.25
686.19
952920.00
974880.00
9007400.0
BankHeist
873.18
873.16
894,25
897.09
873,18
BattleZone
3.618.18
35178.57
30966.
34796.30
26000.00
BeamRider
10341.20
9426.00
8643.85
10021.75
6131.14
Berzerk
716.00
797.60
630.8
637.37
$31.26
Bowling
33.26
47.86
25.41
43.64
25.03
Boxing
83.03
84.21
82.39
82.57
80.12
Breakout
393.00
394.31
354.31
367.17
326.83
Camival
5080.02
5121.81
4921.79
4907.37
4725.68
Centipede
2773.36
3914.76
4305.33
4383.07
745.20
Chopper Command
7683.89
13812.50
6971 .43
8866.67
2.500.00
CrazyClimber
141784.00
139857.14
138336.84
125383.34
68222.22
Attack
13270.77
14469.40
11544.17
11496.00
6436.40
DoubleDunk
-10.49
-9.58
-1.78
-14.89
-17.14
ElevatorAction
120.69
100.00
83.33
0.00
Enduro
938.64
1056.00
1006. 18
1002.70
566.56
Derby
21.85
14.59
14.45
14.32
11.88
Freeway
32.60
32.63
32.71
32.65
32.47
Frostbite
317.78
333,25
220.8
224.29
167.21
Gopher
12782.42
12780.00
7948.5
8436.00
5066.66 69
Gravitar
Hero
20797.14
20816.77
20803.64
20799.20
20612.46
IceHockey
-2.98
-3.40
-3.07
-188
-8.49
Jamesbond
876.12
725.76
727.54
626.12
JourneEsscape
1622.41
3094.22
1523.12
2798.84
-1099.43
Kanguroo
12564.87 87
13717.65
10979.07
10972.09
10629.55
Krull
9496.24
9803.63
9239.06
9443.49
4002.90
MontezumaRevenge
0.00
0.00
0.00
0.00
0.00
MsPacman
5365 06
$697.31
5108.51
5658.80 89
4185.28
Name ThieeThimmamee
9220.43
9462.17
9024.78
8944.78
5497.81
Phoenix
5581.67
5486 00
5472.22
5624.29
4645.71
Pitfall
0.00
0.00
0.00
0.00
-00.00
Pong
21.00
21.00
21.00
21.00
20.92
Pooyan
6636.03
6688.46
5358.33
5444.67
4551.30
PrivateFye
100.00
100.00
100.00
100
100.00
Qbert
17637.16
1.751.71
15239.14
15319.86
8626.09
Riveraid
15434.68
16.388.99
14743.12
15570.32
10808.57
RoadRunner
51097.73
50872.34
48465.45
47603.45
47603.45
Robotank
64.59
64.56
60.95
62.94
50.19
Seaquest
3000.78
3399.02
2416.07
3110.88
1050.27
Skiing
9409.16
9282.99
9315.588
9456.21
2.01
Solaris
3013.33
3548.89
1917.14
2208.89
293.96
S SpaceInvaders
3632.25
3873.55
3658.81
3520.00
28.44557
SturGunner
56695.35
58729
53477.78
55084.09
$3477.78
Tennis
0.20
0.00
0.00
0.00
0.00
TimePilot
7755.89
10743.18
7118.33
5043.21
3833.54
Tutankbam
257.00
292.00
231.63
233.83
39.07
UpNDown
29612.31
32601.54
28862.76
31623.93
5789.64
Venture
202.00
254.55
35,29
22.86
0.00
VideoPinball
501510.50
4807330.5
325911.72
434650.41
283719.04
WizardOrWor
9932.61
12197.73
$747.37
8771.11
2013.22
Revsgeeenge
26152.97
27801.22 24
28064.77
27007.31
25297.36
Zanxon
10473.08
11154.17
10316.36
12422.00
Table Summary of scores with e-greedy (e = 0.001) evaluation with up to 30 no-op starts.
19

---

Under review as a conference paper at ICLR 2020
1.3
2.3
20
2.5
1.0
Alirits
Bonctcot
Sollscare
Bcamtoo
0.5
65
0.2
Bezzer
122
Rrealout
CiegperCarmate
CazyOlmber
Demorskaacc
Dousk63ukk
-22
14
082
b1iu9
F:Sinjuorty
50
0.5
275
250
0.2
-30
Gpphct
[amcsbon]
-20
langeise
Khul
MorthoumsSevrrr
McPacman
1.2
139
1.1
-.15
6.7
-158
094
102
-022
23.9
-00
284
Qbert
Kxerall
Sosckummr
Nabrranc
1.0
1.0
1.2
636
0.5
3.0
363
394
140
162
244
35
2.9
Temts
TimeP
UpRDow'
Vest.re
000
500
0.2
7+5
AddaRrbal
WicandoWaar
1:4
Tarshowogge
busi basoine
Intaal checkrotnt
Conc RIA.
Condl 1083)
155
110
30
331
Fig. 11: Training curves on 16 nodes with up to 30 no-op starts.
20

---

Under review as a conference paper at ICLR 2020
CoNQUR(8 nodes)
Baseline (8 nodes)
Checkpoint
AirRaid
21100.00
17200.00
4825.00
Alien
6480.00
3970.42
2990.00
Amidar
742.00
762.00
188,00
Assault
1872.77
1971.55
1678.14 14
Asterix
68300.00
7700.00
2605.97
Asteroids
2350.00
1180.00
430.22
Atlantis
969700.00
940300.00
922700.00
BankHeist
940.00
980.00
860.00
BattleZone
65000.00
48000.00
30000.00
BeamRider
16694.00
13012.11
9026.86
Berzerk
1223.33
850.48
450.00
Bowling
33.00
25.03
25.03
Boxing
98.39
90.38
87.25
Breakout
415.00
3,00
2.40
Carnival
5776.36
5331.79
4310.00
Centipede
6192.00
4486.31
1952.50
ChopperCommand
16544.19
6900.00
1197.14
CrazyClimber
156500.00
144500.00
22100.00
DemonAttack
35611.54
1110000
7877.30
DoubleDunk
0.00
-2.00
-6.00
ElevatorAction
0.00
0.00
0.00
Enduro
1399.00
1065.00
197.95
FishingDerby
43.67
25.82
16.58
Freeway
33.00
33.00
33.00
Frostbite
220.00
449.88
112.47
Gopher
18040.00
11320.00
3720.00
Gravitar
850.00
700.00
598.8
Hero
20845.00
20780.44
20610.04
IceHockey
-0.77
-1.97
-9.10
Jameshond
750.00
750.00
600.00
JourneyEscape
9300.00
1007.0
-2687.65
Kangaroo
14700.00
10700.00
10700.00
Krull
9753.00
9459.00
687.92
MsPacman
5660.00
6310.00
4180.00
Name ThisGaee
12470.00
11670.00
4,750.00
Phoenix
5990.00
5770.00
4815.00
Pitfall
0.00
0.00
0.00
Pong
21.00
21.00
21.00
Pooyan
8910.00
7945.00
5320.00
Qbert
15650.00
15375.00
7221.43
Riverraid
20280.00
17000.00
7840.00
RoadRunner
63500.00
63500.00
42924.14
Robotank
75.00
72.00
$2.65
Skiing
9,044.00
-30,000.00
-30,000.00
Solaris
912.00
582.86
0.00
StarGunner
64900.00
61000.00
53800.00
Tennis
0.00
0.00
-0,20
TimePilot
7937.10
4692.75
3801.84
Tutankham
252.00
203.00
30.00
UpNDown
71220.00
25470.00
4260.00
Venture
200.00
0.00
0.00
VideoPinball
778319.00
502803.00 00
23520.80
WizardOfWor
11700.00
4900.00
2000.00
YarsRevenge
34199.42
28605.70
13098.00
Zaxxon
17800.00
14300.00
5797.18
Table Summary of scores with e-greedy 0.001) evaluation, without no-op starts.
21

---

Under review as a conference paper at ICLR 2020
Hyperparameters
Description
Value
Splittigg factor e
Controls the number of children created from a par-
4
ent node
Candidate pool size w
Pool of candidate leal nodes for selection into the
46
dive expansion phase
Maximum frontier nodes F.
Maximum number of child leaf nodes for the dive phase
16
Top nodes to expand
Select the top / nodes from the candidate pool for the expansion phase.
4
Dive levels to run
We run di levels of diving phase after each expansion phase
9
Boltzmann iteration
Every module this number iteration/level, Q-
5
labels generated from Boltzmann distribution in order to create diversified node.
Online network target net- work swap frequency Evaluation frequency
Iteration (Frequency) which the online network parameters swap with the target network
5
Iteration (Frequency) which perform rollout operation (testing with the environment).
10
Learning rate Optimizer
Learning for the optimizer.
2.5 10
Optimizer for training the neural network.
RMSProp
Iteration training data transi- tion size
For each iteration, we generate this number tran- sitions and use it as training data.
10k
Training step trequency
For each iteration, we perform (iteration training data transition size / training step frequency) number of gradient updates.
Mini-batch size
Size of the mini batch data used to train the Q-
32
network.
Etrain
e-greedy policy for exploration during training.
0.01
Eeval
s-greedy policy for evaluating Q-regressors.
0.001
Training calibration parame-
Calibration to adjust the difference between the
2.54
nodes from the candidate pool which didn't
lected during both the expansion nor the dive phases. The calibration is performed based on the average difference between the frontier nodes and their par- ents. We denote this difference A.
Discount factor y
Discount factor during the training process.
0.99
Table 6: Hyperparameters for CONQUR training and evaluation.
Hyperparameters
Description
Value
Splitting factor
Controls the number children created from a par- ent node
Candidate pool size m
Pool of candidate leaf nodes for selection into the dive or expansion phase
38
Maximum frontier nodes F
Maximum number of child leaf nodes for the dive
phase
Top nodes expand I
Select the top nodes from the candidate pool for the expansion phase.
Table 7: Different hyperparameters for CONQUR (8 nodes) training and evaluation.
22


Reviews:
[{"review_id": "H1gFcoFBYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "A recent paper by Lu et al introduced delusional bias in Q-learning, an error due to the max in the Bellman backup not being consistent with the policy representation implied by the greedy operator applied to the approximated value function. That work proposed a consistent algorithm for small and finite state spaces, which essentially enumerates over realizable policies. This paper proposes an algorithm for overcoming delusional bias in large state spaces. The idea is to add to the Q-learning objective a smooth penalty term that induces approximate consistency, and search over possible Q-function approximators. Several heuristic methods are proposed for this search, and results are demonstrated in Atari domains.\n\nI found the topic of the paper very interesting - delusional bias is an intriguing aspect of Q learning, and the approach of Lu et al is severely limited to discrete and small state spaces. Thus, tackling the large state space problem is worthy and definitely not trivial. \n\nThe authors’ proposed solution of combining a smooth penalty for approximate consistency and search over regressors makes sense. The implementation of the search (Sec 3.4) is not trivial, and builds on a number of heuristics, but given the difficulty of the problem, I expect that the first proposed solution will not be straightforward.\n\nI am, however, concerned with the evaluation of the method and its practicality, as reflected by the following issues: \n1. The method has many hyper parameters. The most salient one, \\lambda, the penalty coefficient, is changed between 0.25 to 2 on the consistency penalty experiment, and between 1 to 1000 in the full ConQUR experiments. I did not understand the order of magnitude change between the experiments, and more importantly, how can one know a reasonable \\lambda, and an annealing schedule for it in advance. \n2. I do not understand the statistical significance of the results. For example, with the constant \\lambda=0.5, the authors report beating the baseline in 11 out of 19 games. That’s probably not statistically significant enough to claim improvement. Also, only one run is performed for each game; adding more runs might make the results clearer. \n3. The claim that with the best \\lambda for each game, the method outperforms the baseline in 16 out 19 games seems more significant, but testing an optimal hyper parameter for each game is not fair. Statistically speaking, *even if the parameter \\lambda was set to a constant zero* for the 5 runs that the method is tested on, and the best performing run was taken for evaluation against the baseline, that would have given a strong advantage to the proposed method over the baseline….\n4. For the full ConQUR, there are many more hyper parameters, which I did not understand the intuition how to choose. Again, I do not understand how the results establish any statistically significant claim. For example, what does: “CONQUR wins by at least a 10% margin in 20 games, while 22 games see improvements of 1–10% and 8 games show little effect (plus/minus 1%) and 7 games show a decline of greater than 1% (most are 1–6% with the exception of Centipede at -12% and IceHockey at -86%)” mean? How can I understand from this that ConQUR is really better? Establishing a clearer evaluation metric, and using well-accepted statistical tests would greatly help the paper. At the minimum, add error bars to the figures!\n5. While evaluating on Atari shows applicability to large state spaces, it is hard to understand from it whether the (claimed) advantage of the method is due to the delusional bias effect, or some other factor (like implicit regularization due to the penalty term in the loss). In addition, it is hard to understand the different approximations in the method. For example, how does the proposed consistency penalty approximate the true consistency? These could all be evaluated on the simple MDP example of Lu et al. I strongly advise the authors to add such an evaluation, which is easy to implement, and will show exactly how the approximations in the approach deal with delusional bias. It will also be easier to demonstrate the effects of the different hyper parameters in a toy domain. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BkxgvOhitH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a solution to tackling the problem of delusional bias in Deep Q-learning, building upon Lu et.al. (NeuRIPS 2018).  Delusional bias arises because independently choosing maximizing actions at a state may be inconsistent as the backed-up values may not be realizable by any policy. They encourage non-delusional Q-functions by adding a penalty term that enforces that the max_a in Q-learning chooses actions that do not give rise to actions outside the realizable policy class. Further, in order to keep track of all consistent assignments, they pose a search problem and propose heuristics to approximately perform this search. The heuristics are based on sampling using exponentiated Q-values and scoring possible children using scores like Bellman error, and returns of the greedy policy. Their final algorithm is evaluated on a DQN and DDQN, where they observe some improvement from both components (consistency penalty and approximate search).\n\nI would lean towards being slightly negative towards accepting this paper. However, I am not sure if the paper provides enough evidence that delusional bias is a very relevant problem with DQNs, when using high-capacity neural net approximators. Further, would the problem go away, if we perform policy iteration, in the sense of performing policy iteration instead of max Q-learning (atleast in practice)? Maybe, the paper benefits with some evidence answering this question. To summarize, I am mainly concerned about the marginal benefit at the cost of added complexity and computation for this paper. I would appreciate more evidence justifying the significance of this problem in practice. \n\nAnother comment about experiments is that the paper uses pre-trained DQN for the ConQur results, where only the last linear layer of the Q-network is trained with ConQur. I think this setting might hide some properties which arise through the learning process without initial pre-training, which might be more interesting. Also, how would other auxilliary losses compare in practice, for example, losses explored in the Reinforcement Learning with Auxilliary Tasks (Jaderberg et.al.) paper? ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BklDpulB9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on addressing the delusional bias problem in deep Q-learning, and propose a general framework (ConQUR) for integrating policy-consistent backups with regression-based function approximation for Q-learning and for managing the search through the space of possible regressors. Specifically, it proposes a soft consistency penalty to alleviate the delusional bias problem while avoiding the expensive exact consistency testing. This penalty encourages the parameters of the model to satisfy the consistency condition when solving the MSBE problem. \n\nPros:\nThe soft penalization itself is already shown to be effective in further improving any regression-based Q-learning algorithms (e.g., DQN and DDQN). When combining the soft consistency penalty and the search procedure, it is shown to significantly outperform the DQN and DDQN baselines, which is impressive. This work presents novel idea and solid supporting experiments.\n\nCons:\nThe major experimental results of the paper are in Table 4 of Appendix D. This is not a good effort to save space by moving the most important results into appendix. At least a selected set of results should be presented in the main paper rather than in appendices. One alternative approach is the authors plot a bar figure to demonstrate the performance of different algorithms on different Atari games.\n\n\nOther comments:\n•\tFig. 2 two is a bit hard to read due to too many curves.\n•\tStandard baseline results other than DQN and DDQN should also be listed, in order to demonstrate that solving delusional bias could make Q-learning more competitive than alternatives (e.g., A3C, PPO, TRPO).\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While there was some support for the ideas presented, the majority of reviewers felt that this submission is not ready for publication at ICLR in its present form.

Concerns raised included the need for better motivation of the practicality of the approach, versus its computational cost. The need for improved evaluations was also raised.
Decision: Reject