Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learning
Authors: Fan Zhou
Zhoufan Zhu
Qi Kuang
Liwen Zhang
Venue:  (2021)
Keywords: Non-decreasing Quantile Function
Distributional Reinforcement Learning
Distributional Prediction Error
Exploration

Abstract: Although distributional reinforcement learning (DRL) has been widely examined in the past few years, there are two open questions people are still trying to address. One is how to ensure the validity of the learned quantile function, the other is how to efficiently utilize the distribution information. This paper attempts to provide some new perspectives to encourage the future in-depth studies in these two fields. We first propose a non-decreasing quantile function network (NDQFN) to guarantee the monotonicity of the obtained quantile estimates and then design a general exploration framework called distributional prediction error (DPE) for DRL which utilizes the entire distribution of the quantile function. In this paper, we not only discuss the theoretical necessity of our method but also show the performance gain it achieves in practice by comparing with some competitors on Atari 2600 Games especially in some hard-explored games.

Under review as a conference paper at ICLR 2021
NON-DECREASING QUANTILE FUNCTION NETWORK WITH EFFICIENT EXPLORATION FOR DISTRIBUTIONAL REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Although distributional reinforcement learning (DRL) has been widely examined in the past few years, there are two open questions people are still trying to ad- dress. One how to ensure the validity of the learned quantile function, the other is how to efficiently utilize the distribution information. This paper attempts provide some new perspectives to encourage the future in-depth studies in these two fields. We first propose non-decreasing quantile function network (NDQFN) to guarantee the monotonicity of the obtained quantile estimates and then design general exploration framework called distributional prediction error (DPE) for DRL which utilizes the entire distribution of the quantile function. In this pa- per, we not only discuss the theoretical necessity of our method but also show the performance gain it achieves in practice by comparing with some competitors on Atari 2600 Games especially in some hard-explored games.
INTRODUCTION
Distributional reinforcement learning (DRL) algorithms (Jaquette 1973 Sobel 1982 White 1988
Morimura al., 2010 Bellemare et al., 2017), different from the value based methods (Watkins 1989 Mnih et al. 2013) which focus on the expectation of the return, characterize the cumulative reward as random variable and attempt to approximate its whole distribution. Most existing DRL methods fall into two main classes according to their ways to model the return distribution. One class, including categorical DQN (Bellemare al. 2017), C51, and Rainbow (Hessel et al., 2018), assumes that all the possible returns are bounded in known range and learns the probability of each value through interacting with the environment. Another class, for example the QR-DQV (Dabney cet al. 2018b), tries to obtain the quantile estimates at some fixed locations by minimizing the Huber loss (Huber 1992) of quantile regression (Koenker) 2005). To more precisely parameterize the entire distribution, some quantile value based methods, such IQN (Dabney et al. 2018a) and FQF (Yang et al. 2019), are proposed to learn a continuous map from any quantile fraction T E (0, 1) to its estimate the quantile curve.
The theoretical validity of QR-DQN (Dabney et al., 20185), IQN (Dabney et al., 2018a) and FQF Yang al. 2019) heavily depends on prerequisite that the approximated quantile curve non- decreasing. Unfortunately, since no global constraint is imposed when simultancously estimating the quantile values multiple locations, the monotonicity can be ensured by using any existing network design. At early training stage, the crossing issue is even more severe given limited training samples. Another problem be solved is how design an efficient exploration method for DRL. Most existing exploration techniques are originally designed for non-distributional RI. (Bellemare et al., 2016; Ostrovski et al., 2017 Tang al. 2017 Fox et al., 2018 Machado et al. (2088), and very few of them can work for DRL. Mavrin et al. (0019) proposes DRL-based exploration method, DLTV, for QR-DQN by using the left truncated variance as the exploration bonus. However, this approach not be directly applied to quantile value based algorithms since the original DLTV method requires all the quantile locations to be fixed while IQN or FQF resamples the quantile locations at each training iteration and the bonus term could be extremely unstable.
To address these two common issues in DRL studies, we propose a novel algorithm called Non- Decreasing Quantile Function Network (NDQFN), together with an efficient exploration strategy,

---

Under review as a conference paper at ICLR 2021
distributional prediction error (DPE), designed for DRL. The NDQFN architecture allows to approximate the quantile distribution of the return by using a non non-deceaasing piece-wise linear func- tion. The monotonicity of N + fixed quantile levels is ensured by incremental structure, and the quantile value at any 7 E (0, 1) can be estimated as the weighted sum of its two nearest neighbors among the N 1 locations. DPE uses the 1-Wasserstein distance between the quantile distributions estimated by the target network and the predictor network as an additional bonus when selecting the optimal action. We describe the implementation details of NDQFN and DPE and examine their performance on Atari 2600 games. We compare NDQFN and DPE with some baseline methods such IQN and DLTV, and show that the combination of the two can consistently achieve the op- timal performance especially in some hard-explored games such as Venture, Montezuma Revenge and Private Eye.
For the rest of this paper, we first go through some background knowledge of distributional RI. in Section 2. Then in Sections 3, we introduce NDQEN, DPE and describe their implementation de- tails. Section 4 presents the experiment results using the Atari benchmark, investigating the empiri- cal performance of NDQEN, DPE and their combination by comparing with some baseline methods.
2 BACKGROUND AND RELATED WORK
Following the standard reinforcement learning setting, the agent-environment interactions are mod- eled as Markov Decision Process, MDP, (X', A, R. P, P.)) 7) (Puterman 2014). X' and A denote finite set of states and a finite set of actions, respectively. R X X R is the reward function, and y [0, 1) is a discounted factor. P: X' A X' [0, is the transition kernel.
On the agent side, a stochastic policy л maps state x to a distribution over action a ~ <<- (x) regardless of the time step t. The discounted cumulative rewards is denoted by random variable 2" (z. a) = Di= 7 R( ar), where To = I, 4n = a, Te ~ P(- az-1) and ag ~ TT-12,)) The objective is find an optimal policy π maximize the expectation of 2 (xx, a). EE2" (x)
which is denoted by the state-action value function Q (xc a). A common way to obtain x is to find
the unique fixed point Q* = =Q" of the Bellman optimality operator T (Bellman 1966):
Q (x, a) - ((x.a)=TQ*( (z, a) = E[R(, a)] + 7EE max (x',
(I)
The state-action value function Q can be approximated by parameterized function Qa (c.g. a neural network). Q -learning (Watkins 1989) iteratively updates the network parameters by minimizing the squared temporal difference (TD) error
57 re max Qo (0( - Qo(xx, a.) a.)] )²
a'EA
on sampled transition (Ie, oe, Fe, 2e41), collected by running an e-greedy policy over Qo-
Distributional RL algorithms, instead of directly estimating the mean EZ* a). focus on the dis- tribution zT (a, a) to sufficiently capture the intrinsic randomness. The distributional Bellman oper- ator, which has a similar structure with (I. is defined as follows,
TZ" (I, a) 2 R(z, + (X'
where X' ~ P(-|z, a) and A' ~ 7 (-IX). A 2 B denotes the equality in probability laws.
3 NON-MONOTONICITY IN DISTIRBUTIONAL REINFORCEMENT LEARNING
In Distributional RL studies, people usually pay attention to the quantile function F7 FattiionFF755
= inf(= e R 7 < Fz(2)} of total return Z, which is the the inverse of the cumulative distribution function F2(2) = Pr(Z < z). In practice, with limited observations at each quantile level, it is much likely that the obtained quantile estimates Fz (r) at multiple given locations [T1, ..,..., TN } for some aaatenaction pair (z, are non-monto as Figure [ (a) illustrates. The key reason behind this phenomenon is that the quantile values at multiple quantile fractions are separately estimated without applying any global constraint to ensure the monotonicity. Ignoring the non-decreasing
2

---

Under review as a conference paper at ICLR 2021
property of the learned quantile function leaves theory- practicc gap, which results in the potentially non-optimal action selection in practice. Although the crossing issue has been broadly studied by the statistics community (He [997] Chemozhukov al. 2010 Dette & Volgushev, 22088,, how to ensure the monotonicity of the approximated quantile function in DRL, still remains challenging, especially some quantile value based algorithms such IQN and FQF, which do not focus fixed quantile locations during the training process.
(T.... Tab
IQN
(a)
p (p) Pp.
11:13
(r)
15400
6200
Incremental Structure
(b)
r'
45409
p
NQDEN
(C)
Figure 1: Quantile estimates obtained by (a) IQN, (b) mere incremental structure and (c) NDQFN under 50M training on Breakout, T and 7' denote two sets of sampled quantile fractions from two different training iterations.
4 NON-DECREASING QUANTILE FUNCTION NETWORK
To address the crossing issue, we introduce a novel Non-Decreasing Quantile Function Network (NDQFN) with two main components: (1). incremental structure estimate the increase of quantile values between two pre-defined nearby supporting points p;-1 E (0, 1) and pt E (0, i.... (1 a: p) = FD)==( (px) Fz (pi-)), and subsequently obtain each Fz (p.) for E {o...., N} as the cumulative sum of Ass (2). piece-wide wisee-wise//e linear function which connects the N + supporting
points to represent the quantile estimate () (T) any given fraction E (0, 10. Figure
describes the whole architecture of NDQFN.
We first describe the incremental structure. Let p {Po.... ,PN] } be set of N + 1 supporting points, satisfying << Pi S Pi+1 < for each i E (1.2.... .1.......f N 1}. Thus, Z(x, a) can be parameterized by mixture of N Diracs, such that
N-1
Ze a) = Σ (pi+1 - (.)))
where each Og denotes the quantile estimation at 9. putpi+1, and 0 (fu....,0N.. 11 Let X' - d and Φ [0 + Rd represent the embeddings of state and quantile fraction т, respectively. The baseline value Ao(x, increments (1 for i E [1, can be represented by
ar) (Po) Po and the following N non-negative
An(r, p) = (x, a; p) = f()()))
(, a)) (w a: = g ((((()), o(p)). d(p.1))aa i 1, .N.
where f: .Rd + RId| and parameters of the network.
2d + [0, 12l1 are two functions to be learned. w includes all the
3

---

Under review as a conference paper at ICLR 2021
Then, use A. to denote projection operator that projects the quantile function onto piece- wise linear function supported by p and the incremental structure A. = (Au...". , above. For any quantile level 7 E (0, 1), its projection-bate based quantile estimate is given by
N-1
F P. (r) p.A. Z(2,0)
C...A.ZZura (z. (T) (0, ax, +
Σ (r,,,,;
(4)
N
where A. could be regarded as a linear combination of quantile estimates at two nearby support- ing points satisfying
G (T = Σ (x,
+ T Pi Pi+1 p:
(r, (pl S P++1).
By limiting the output range of Ja in 3 to be [0, oo), the obtained N + 1 quantile estimates are non den-eecra.sing. Under the NDQFN framework, the expected future return starting from (xx, a), also known as the Q-function can be empirically approximated by
Qu / po
F F-l.p.Aw
(F)
(p++1 P.) I.p.A. ( + Pit1)
2
(pit1 pı))
(Z)(0))
(pttt))ppttt)) (Di+1) +
F.I.p.u (p:)
F(z,a)
V
Actions
Actione
ber
who Liwer
Pi p (Ps)
Incremental Structure
Figure 2: The network architecture of NDQFN.
A...,,.A. For notational simplicity, we let (r, = (( (7), given the support p. Following the
idea of IQN, two random sets of quantile fractions independently drawn from uniform distribution U0, 1) at each training iteration. In this case, for each [1....,Ny N and each j E (2). the corresponding temporal difference (TD) error (Dabney al. 2018a) with n-step updates (Zt, Ti, ,Tttn-1, I(tn) is computed as follows,
{Ti,' .N, }. +' {-í, Tx,
61,5 = Σ r'reti + + Pr/aa (Ze-ns arg max Qw (x)+ a')) P Pre (r,, at),
(5) where wa and w/ denote the online network and the target network, respectively. Thus, we can train the whole network by minimizing the Huber quantile regression loss (Huber 1992) as follows,
a'EA
Na
C(xi, ats at, Tt,"" 1,4m-1,3e4n)
Ps (ōe.),
(6)
where
Lx (5ir)
pf (81.g) = -|T - -
1881 -II
< <0)
2 80
| S K
Lx (81.) =
1
1 (15
=
otherwise
2
4

---

Under review as a conference paper at ICLR 2021
is a pre-defined positive constant, and | denotes the absolute value of scalar. Remark:
As shown by Figure (c), the piece-wise linear structure ensures the monotonicity within the union of any two quantile sets and r' from two different training iterations regardless of whether they are included in the p or not. However, as Figure (b) demonstrates, directly applying similar incremental structure onto IQN without using the piece-wise linear approximation may result in the non-monotonicity of T U U although each of their own monotonicity not violated.
To investigate the convergence of the proposed algorithm, which can be seen an extension of Proposition 2 in Dabney et al. (0018))
introduce the following theorem,
Theorem 1. Let p. be the quantile projection defined as above with non-decreasing quantile function Fz I..... For any two value distribution Zi, Z2 E 2 for an MDP with countable state and action spaces and enough large N,
doo (T..... I. P... TF.... m S ydoc (FZ) (F-ID,,
F F.l.p.A.))
(8)
where T denotes the distributional beliman operator, de Z1 Fzz) == supn,a W4 (F7,, F a)(z,)) and Wa(ss-) denoles the k- t-Wasserstein metric.
(w,a))
By Theorem 1, we conclude that A. T have a unique fixed point and the repeated application of P..._. T converges to the fixed point. With da I doc. the convergence occurs for all k E (1). It ensures that we can obtain consistent estimator for Fz denoted as Fz u. by minimizing the Huber quantile regression loss defined in (6).
Now discuss the implementation details of NQDFN. For the support p used in this work, let p; = i/N for E {1, {1...... -N- 1}, Po = 0.001 and pN = 0.999. NDQEN models the state embedding (b(x)) and the r-embedding ((r) in the same way IQN. The baseline function f in K3 consists of two fully-connected layers and a sigmoid activation function, which takes v((x) as the input and returns unconstrained value for An(r, a: p). The incremental function g shares the same structure with f but uses the ReLU activation instead to ensure the non-negativity of all the N increments (I, a; p)'s. Let @ denote the element-wise product and g take vxx) ((p)) and ((P.) ((pi-1) 1) as input, such that
(z, a) (vv()) @ ((p:). ((pi-1)) i i= N,
which to some extent captures the interactions among (xx)), p(p)) and p(p--1). Although (xx) @ ( may be another potential input form, the empirical results show that the combination of (1r) @ @(:;) and @(p;) ((p:-1) is more preferred in practice considering its outperformance in Atari games. More details are provided in Section C of the Supplement file.
EXPLORATION USING DISTRIBUTIONAL PREDICTION ERROR
To further improve the learning efficiency of NDQFN, we introduce novel exploration approach called distributional prediction error (DPE), motivated by the Random Network Distillation (RND) method (Burda et al. 2018), which can be extended to most existing DRL frameworks. The proposed method, DPE, involves three networks, online network, target network and predictor network, which have the same architecture but different network parameters. The online network and the target network w' use the same initialization, while the predictor network w" is separately initialized. To be more specific, the predictor network are trained on the sampled data using the quantile Huber loss defined as follows:
N
Cloe, oz) Nº i=1
(62.),
where 67, Pr. w (Ze, ae) P an (Ie, az) and pf(() are detined in On the other hand, follow- ing Hassel 2010) and Pathak al. 2019 the target network of iPE periodically synchronized
by the online network. DPE employs the 1- Wasserstein metric W1(-.)) -) to measure the distance be- tween the two quantile distributions associated with the predictor network and the target network. And its empirical approximation based on NDQFN is defined as follows,
PN
i(x1, az)
Wi (Prar (Kt, at) - Pr.w a:)) dr.
(10)
5

---

Under review as a conference paper at ICLR 2021
Since the minimum of [(2,, 0z) is contraction of the 1-Wasserstein distance (Bellemare et al. 2017). the prediction error would be higher for an unobserved state-action pair than for a trequently visited one. Therefore, i(z,, ae) can be treated as the exploration bonus when selecting the optimal action, which encourages the agent to explore unknown states. With i( ar) being the exploration bonus, the optimal action is determined by
04 = arg max (1 a) ce((ss)))
(11)
where is the bonus rate. As might be expected, more reasonable quantile estimate would highly enhance the exploration efficiency according our exploration setting. To more clearly demonstrate this point, we compare the performance of DPE based on NDQFN and IQN in Section 6.2
On the other hand, we compared the exploration efficiency of distributional prediction error and value-based prediction error in Section D of supplement, where value-based prediction error is de- fined as i' (x1,, az) = 1Qw- (Iz, at) - Qu ).)]. As might be expected, DPE enhances the explo- ration efficiency by utilizing more distributional information.
6 EXPERIMENTS
In this section, we evaluate the empirical performance of the proposed NDQFN with DPE on 55 Atari games using the Arcade Learning Environment (ALE) (Bellemare al.) 2013). The baseline algorithms we compare include IQN (Dabney et al. 2018a). QR-DQN (Dabney et al.) (018b), C51 (Bellemare et al. 2017). prioritized experience replay (Schaul al. 2015) and Rainbow (Hessel et al. 2018). The whole architecture of our method is implemented by using the IQN baseline in the Dopamine framework (Castro et al., 2018). Note that the whole method could be built upon other quantile value based baselines such as FQF.
Our hyper-parameter setting is aligned with IQN for fair comparison. Furthermore, we incorporate n-step updates (Sutton, 1988), double Q learning (Hasselt 2010), quantile regression (Koenker 2005) and distributional Bellman update (Dabney et al.] 2018[008) into the training for all compared methods. The size of p and the number of sampled quantile levels, ; and +', are 32, i.e. N N1 N2 = 32. We use e-greedy policy with = 0.01 for training. At the evaluation stage, we test the agent for every 0.125 million frames with 0.001.
For the DPE exploration, we fix = 1 without decay (which is the optimal setting based on experiment results). As a comparison, we extend DLTV (Mavrin et al. 2019), an exploration ap- proach designed for DRL algorithms based on fixed quantile locations such as QR-DQN, to quan- tile value based methods by modifying the original left truncated variance used in their paper to of = S (F (7) - Fz ())] All training curves in this paper are smoothed with a moving average of 10 to improve the readability. With the same byper-parameter setting. NDQFN with DPE is about 25% slower than IQN at training stage due to the added predictor network.
6.1 NDQFN vs IQN
In this part, compare NDQFN with its baseline IQN (Dabney al. 2018a) to verify the improve- ment it achieves by employing the non-decreasing structure. The two methods are fairly compared without employing any additional exploration strategy.
Figure 3 visualizes the training curves of eleven randomly selected easy explored Atari games. NDQFN significantly outperforms IQN in most cases, including Berzerk, KungFu Master, Ms. Pac- Man, Qbert. River Raid, Up and Down, Video Pinball and Zaxxon. Although NDQFN achieves similar final scores with IQN in Battle Zone and Tennis, it learns much faster. This tells that the crossing issue, which usually happens in the early training stage with limited training samples, can be sufficiently addressed by the incremental design in NDQFN to ensure the monoticity of the quantile estimates. More related results could be seen in Section E of the supplement.
6.2 EFFECTS OF NDQFN ON DPE
In this section, we show how the NDQFN design can help improve the exploration efficiency of DPE compared using the IQN baseline. We pick three hard explored games, Montezuma Revenge,

---

Under review as a conference paper at ICLR 2021
battlezone
berzerk
breakout
Barum
fillions of frrmnes
Willerr of frames
Million framees
kungfumaster
mspacman
obert
Recurn
Millions 101 13
Millirs frames tennis
Millions frames
riverraid
upndown
Pilltins
Viilions
Nillions
videopinball
1:000
4:000
ION NDOFN
Pattions of tramees
Aittons of framess
Figure 3: Training curve on Atari games for NDQEN and IQN.
Private Eye, Venture, and three easy explored games, Ms. Pac-Man, River Raid and Up and Down. The main results are summarized in Figure
montezumarevenge
privateaye
venture
Millions
1P AP DP 10
Millons
Us Tor 120
Nitlons frames
mspacman
riverraid
upndown
MooF
Millinns
frames
framess
Figure 4: Comparison between NDQFN+DPE and IQN+DPE.
As Figure 4 Aillustrates, DPE is effective exploration method which highly increases the training performance of both IQN and NDQFN in three hard explored games. the three easy-explored games, DPE still performs slightly better than the baseline methods, which to some extent demon- strates the stability and robustness of DPE. On the other hand, we find that NDQFN + DPE sig- nificantly outperforms IQN + DPE especially in some hard-explored games. This agrees with our conclusion in Section 5 that NDQFN obtains a more reasonable quantile estimate by adding the non-decreasing constraint which helps to increase the exploration efficiency.
7

---

Under review as a conference paper at ICLR 2021
6.3 DPE VS DLTV
We do some more experiments to show the advantage of DPE over DLTV when applying both of the two exploration approaches quantile value based DRL algorithms. To be specific, we evaluate the performance of NDQFN+DPE and NDQFN+DLTV on the six games examined in the previous subsection. Figure 5 shows that DPE achieves much better performance than DLTV especially in the three hard explored games. DLTV does not achieve significant performance gain over the baseline NDQFN without doing exploration.
montezumarewenge
privateeye
venture
Pillions of trames
Millons of frames
Millions trames
mspacman
riverradd
upndown
Herurn
Millions Us Ls to
Willions
36 11
Nillion frames
Figure 5: Comparison between DPE and DLTV.
This result tells that DLTV, which computes the exploration bonus based on some discrete quantile locations, performs extremely unstable for quantile value based methods such as IQN and NDQFN since the quantile locations used to train the model are changed each time. As contrast, DPE focuses on the entire distribution of the quantile function, which relies less on the selection of the quantile fractions, and thus performs better in this case.
6.4 FULL ATARI RESULTS
In this part, we evaluate the performance of the whole network which incorporates NDQFN with DPE in all the 55 Atari games. Following the IQN setting (Dabney al. 2018a), all evaluations start with 30 non-optimal actions to align with previous distributional RL works. 'IQN' in the table tries to reproduce the results presented in the original IQN paper, while 'IQN* corresponds the improved version of IQN by using n-step updates and some other tricks mentioned above.
Mean
Median
Human
DQN
221%
79%
24
PRIOR.
580%
124%
39
C51
701%
178%
40
RAINBOW
1213%
227%
42
QR-DQN
902%
193%
41
IQN
1112%
218%
39
IQN*
1207%
224%
40
NDQEN+DPE
1959% %
237%
42
Table 1: Mean and median of scores across 55 Atari 2600 games, measured percentages of human baseline. Scores of previous work are referenced from Castro et al. (2018) and Yang et al. (2019).
Table presents the mean and median human normalized scores by the seven compared methods across 55 Atari games, which shows that NDQEN with DPE significantly outperforms the IQN baseline and other state-of-the-art algorithms such Rainbow. In particular, Figure 6 plots the
8

---

Under review as a conference paper at ICLR 2021
relative improvement of NDQFN+DPE over IQN score for all the 55 Atari games. We observe that our method consistently outperforms the IQN baseline in most situations, especially for some hard games with extremely sparse reward spaces, such as Private Eye and Montezuma Revenge. More comparison results including the training curves for all the 55 games and the raw scores provided in Sections F and G in the supplement.
in testing
NDQFN+DPE gain
NDOFN
28 gamess IQN level
13 games at IQN level
Logh
14 games below IQN level
-100%
0%
200%
400%
600%
23000%
8
Figure 6: Cumulative rewards performance comparison between our method and IQN with 200M training frames. The bars represent relative gain/loss of NDQFN+DPE over IQN.
7 CONCLUSION AND DISSCUSION
In this paper, we make some attempts to address two important issues people care about for distribu- tional reinforcement learning studies. We first propose a Non-decreas Quantile Function Network (NDQFN) to ensure the validity of the learned quantile function by adding the monotonicity con- straint to the quantile estimates at different locations via a piece-wise linear incremental structure. We also introduce DPE, general exploration method for all kinds of distributional RL frameworks, which utilizes the information of the entire distribution and performs much more efficiently than the existing methods. Experiment results more than 50+ Atari games illustrates NDQFN can more precisely model the quantile distribution than the bascline methods. On the other hand, DPE perform better in both hard-explored Atari games and easy ones than DLTV. Some ablation studies show that the combination of NDQFN with DPE achieves the best performance than all the others.
There some questions remain unsolved. First, since lots of non-negative functions available, is Relu a good choice for function g? Second, will a better approximated distribution affect agent's policy? If so, how does it affect the training process as NDQFN can provide potentially better dis- tribution approximation. More generally, how to analytically compare the optimal policies NDQFN and IQN converge to. Third, how much does the periodically synchronized target network affect the efficiency of the DPE exploration.
For the future work, we first want to apply NDQFN to other quantile value based methods such as FQF to see if the incremental structure can help improve the performance of all the existing DRL algorithms. Second, as there exists some other kind of DRL methods which naturally ensures the monotonicity of the learned quantile function (Yue et al.] (020) but requies extremely large computation cost, it may be interesting to integrate our method with these methods to obtain an improved empirical performance and training efficiency.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in neural information processing systems, pp. 1471-1479, 2016.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal Artificial Intelligence Research, 47: 253-279, 2013.
Mare G Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 449-458, 2017.
Richard Bellman. Dynamic programming. Science, 153(3731):34-37, 1966.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference Learning Representations, 2018.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare.
Dopamine: A Research Framework for Deep Reinforcement Learning. 2018. URL http:
Victor Chernozhukov, Iván Fernández-Val, Val, and Alfred Galichon. Quantile and probability curves without crossing. Econometrica, 78(3):0093-1125 2010.
Will Dabney, Georg Ostrovski. David Silver, and Remi Munos. Implicit quantile networks for distributional reinforcement learning. In International Conference on Machine Learning, pp- 1105, 2018a.
Will Dabney, Mark Rowland, Mare G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In AAAI, 2018b.
Holger Dette and Stanislav Volgushev. Non-crossing non-parametric estimates of quantile curves. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 70(3):609-627, 2008.
Lior Fox, Leshem Choshen, and Yonatan Loewenstein. Dora the explorer: Directed outreaching reinforcement action-selection. In International Conference on Learning Representations, 2018.
Hado Hasselt. Double q-learning. In Advances neural information processing systems, pp-
2613-2621, 2010.
Xuming He. Quantile curves without crossing. The American Statistician, 51(2):186-192 1997.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018.
Peter Huber. Robust estimation of a location parameter. In Breakthroughs in statistics, pp. 492- 518. Springer, 1992.
Stratton Jaquette. Markov decision processes with a new optimality criterion: Discrete time. The Annals of Statistics, pp. 496-505, 1973.
Koenker. Quantile regression. Cambridge University Press, 2005.
Marlos C Machado, Marc Bellemare, and Michael Bowling. Count-based exploration with the successor representation. arXiv preprint 1rXiv:1807.11622 2018.
Borislav Mavrin, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional re- inforcement learning for efficient e exploration. In International Conference on Machine Learning, pp. 4424 4434, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 131.::1312.5600 2013.
10

---

Under review as a conference paper at ICLR 2021
Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Nonparametric return distribution approximation for reinforcement learning. JCML, 2010.
Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International Conference on Machine Learning, pp. 2721-2730, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. 2019.
arXiv preprint
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John
Wiley & Sons, 2014.
Tom Schaul, John Quan, Toannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv: /5/ 15..0555,, 2015.
Matthew Sobel. The variance of discounted markov decision processes. Journal of Applied Prob- ability, 19(4):794-880) 802, 1982.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9-44, 1988.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schul- man, Filip DeTurek, and Pieter Abbeel. exploration: A study of count-based exploration for deep reinforcement learning. In Advances in neural information processing systems, pp. 2753- 2762, 2017.
Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. 1989.
DJ White. Mean, variance, and probabilistic criteria in finite markov decision processes: A review. Journal of Optimization Theory and Applications, 56(1):1-29, 1988.
Derek Yang, Li Zhao, Zichuan Lin, Tao Qin, Jiang Bian, and Ye- Liu. Fully parameterized quantile function for distributional reinforcement learning. In Advances in Neural Information Processing Systems. pp. 6193-6202, 2019.
Yuguang Yue, Zhendong Wang, and Mingyuan Zhou. Implicit distributional reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.


Reviews:
[{"review_id": "vH5Myh4Gr9Q", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper Summary: This paper mainly contributes in two parts: 1). A non-decreasing structure for quantile values in quantile-based distributional RL. 2). A curiosity-based intrinsic reward using distribution disagreement.\n\nClarity: \n- Some mathematical expressions, while correct, are hard to interpret, e.g. G_{i,\\delta}, H_{i,\\omega', \\omega^*}. It would be nice if the authors could provide a brief explanation in words. For example, G_{i,\\delta} is just linear interpolation given fixed support.\n\n- The paper stops at experiment results. There are no further discussions or conclusion section.\n\n- The relation between RND and proposed DPE is not as close as the authors suggest. In RND, the target network is a fixed network for the predictor network to mimic. In DPE, both the target network and the predictor is trained with the same objective, i.e. minimize TD error. The intrinsic reward is evaluated by the disagreement between the predictor and the target network. Such disagreement-based intrinsic reward had been studied in other lines of work, e.g. [1].\n\nPros: \n- Resolves existing crossing quantile issue in most quantile-based distributional RL algorithms.\n- Proposed method ensures monotonicity on all possible quantile sets. Using linear interpolation for distribution approximation feels natural and straightforward.\n- Distribution based curiosity is novel.\n\nCons:\n- The significance of DPE is not properly evaluated. In fact, the paper only provide detailed results for NDFQN+DPE. It is hard to tell exactly how much of the performance gain is credit to NDFQN or DPE. To evaluate the significance of DPE, I would expect the authors to compare across different curiosity-driven exploration methods on the same baseline, say IQN. At least, the authors should answer how much distribution-based disagreement outperforms value based disagreement.\n\n- Same reason as above, the performance gain of NDFQN itself cannot be inferred from just 6 games. Judging from section 4.1 and 4.2, I would presume that the performance gain is mainly credit to DPE instead of NDFQN.\n\n- The issue in figure 1(c) is not entirely resolved. Different from the issue in figure 1(b), this issue directly comes from using incrementals instead of values. The authors partially resolves it by using a fixed set of support, but if the support is changeable the issue remains.\n\n\nQuestions:\n- Is the embedding of p* really necessary? Or, is p* necessary for network input? As the support p* is fixed, I do not believe that they should be part of the input. From my own point of view, using modified QR-DQN's structure sounds more reasonable. If the authors find out that an additional, fixed input does impact the performance significantly, some explanations or insights are required.\n\nPost rebuttal:\n- The authors have addressed most of my main concerns and the additional experiment looks good to me. Therefore, I increase my score to 6. However, some experiment results are still missing and the paper still needs some editing before published, especially the experiment section. For example figure 6. is misleading since the comparison is not fair.\n\nReferences:\n[1] Pathak, Deepak, Dhiraj Gandhi, and Abhinav Gupta. \"Self-Supervised Exploration via Disagreement.\" International Conference on Machine Learning. 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "u_ue0wxoD4k", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper provides a method to learn the quantile function in distributional reinforcement learning that guarantee its monotonicity. Moreover, an exploration strategy for distributional reinforcement learning is also presented.\n\nReasons for score:\n\nThe paper claims two contributions: a better way to model quantile functions and an exploration strategy for distributional reinforcement learning. The former is a trivial extension of previous work, and the latter is not properly explained. \n\nPros:\n\n1. The experiment results look good.\n\n2. The DPE exploration strategy seems to be effective in the examples.\n\n\nCons:\n\n1. The NDQFN is a trivial extension of (3) through piece-wise linear interpolation. The explanation of Figure 1 is misleading.\n\n2. There is another class of methods in distributional reinforcement learning which uses generative network to model the Z function, which doesn’t suffer from lack of monotonicity as in quantile function learning (see e.g. Y. Yue, Z. Wang, M. Zhou ``Implicit Distributional Reinforcement Learning”). These type of methods need to be properly discussed and also compared in the experiments.\n\n3. Theorem 1 could be problematic. The quantile projection defined in this paper doesn’t seem to be contractive. \n\n4. Section 3.4 is not well presented. \n\nQuestions:\n\nPlease address and clarify the cons above.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VUHWLnvyHS5", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "###  Summary\nThis paper proposes to use a monotonic quantile function for distributional RL by: (i) estimating the quantile values at the supported quantiles via a cumulative sum of non-negative incremental value, and (ii) interpolate the quantile values at the unsupported quantiles via linear combination of the nearest supported quantiles. The paper then also proposes to estimate the exploration bonus for each state using random network distillation. It then conducts experiments in Atari games to verify some conclusions.  \n\n### Strong points \n-\tClarity: The paper is well structured. \n-\tTechnical novelty: immediate. The idea of piece-wise linear function to approximate quantile function in Eq. (4) seems novel though very natural. The idea of using random network distillation to measure the exploration bonus seems novel and useful. \n-\tEmpirical significance: The combination of DPE with NDQFN performing well in some hard-explored games (Fig. 3) looks very encouraging and promising. In my opinion, I think this is the most interesting and significant empirical result reported in this paper in the current form.\n\n###  Weak points\n-\tThe experimental comparison/conclusion with IQN in the present form is unfair. \nIn particular, after the Table 1, the authors conclude that “NDQFN with DPE significantly outperforms the IQN baseline”. I think this is an unfair conclusion as the result of IQN reported in Table 1 is merely the performance for distributional part, i.e., the IQN result in Table 1 does not include any orthogonal improvements such as n-updates, double networks and any advanced exploration method beyond epsilon-greedy while NDQFN incorporates DPE exploration method (and n-step updates  and double Q update if I infer correctly from the paper). To be fair, I think the paper should include to IQN any orthogonal improvements that NDQFN has.  Even though Figure 2 has a fair comparison between NDQFN and IQN where both use n-step updates and without DPE exploration, I think only 6 games are not sufficient to make a reliable conclusion that the proposed non-decreasing quantile function is more helpful than the original quantile function in IQN. \n-\tIt is unclear from the experimental results how helpful are each non-decreasing quantile function and DPE individually. \nExcept a nice result in Figure 3, I am not fully convinced if the proposed non-decreasing quantile function is actually more helpful than the original quantile function as a distributional component. For example, I think it is more useful to report the result of a mere NDQFN (i.e., without DPE or any exploration methods rather than epsilon-greedy, without any orthogonal improvements such as n-step updates and double Q value) in the full Atari games to see how much it improves over the original IQN (reported in Figure 1). Regarding DPE, since it is a new exploration method, I think it is more helpful and reliable to experimentally compare DPE with some other distributional exploration methods such as  DLTV (Mavrin et al. 2019). I think the current experimental results are hard to make any reliable conclusion about each NDQFN and DPE individually except a nice observation that a combination of NDQFN and DPE can help in some hard explored games.   \n\n###  Questions for the authors. \n-\tThough I believe that IQN indeed does not guarantee the monotonicity in quantile values it estimates, the illustrative figure for this (Fig 1) seems a hypothetical one instead of a real experimental result.  Can we show that empirically by a simulation? \n-\tDoes the NDQFN+DPE in Table 1 also have n-step update and double Q network update style? \n\n###  My initial recommendation\nFor the main reasons in the weak points section, I vote for rejecting for this current form.\n\n### My final recommendation \n\nThe authors have attempted to address some of my points but these points require more time to fully address as they require to run more experiments. For the current form, I remain my inital score and recommend rejection for this time. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LUrOqKbxe6w", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies distributional RL and proposed two extensions. One is a method to enforce a non-decreasing ordering of quantile functions by a linear and non-negative increments. The other is extends the idea of DLTV which adds exploration bonus in action selection by using the random network distillation method, which in particular, using a measure of inconsistency between target network and predictor networks as a frequency measure of sampled states. \n\nHow do you convince us enforcing a non-decreasing ordering of the learned quantile functions is helpful? \nI understand your arguments, but there is no evidence in the paper showing that doing so is helpful. \n\nComparison with DLTV is missing. \nThe paper argues that DLTV is not applicable to continuous quantiles. However, it would be to include this comparison especially they have results on Atari games as well. \n\nThe empirical results are not very strong, with 13 and 14 ties and losses with/to IQN. It appears the advantage of DLTV over QRDQN is larger than your advantage over IQN. \n\nThe technical quality and presentation of the paper can still be much improved. \n\nAbstract: \ntwo important problems still remain unsolved. \nthe other is how to design\nan efficient exploration strategy to fully utilize the distribution information\n--> Later you showed this is false argument by introducing DLTV (Mavrin et. al. 19)\n\nWe describe the implementation details of the two architectures with\nwhat are they?\nyou have two \"architectures\"?\n\n (b)(c) a simple incremental structure proposed in(3):\nthis sentence is confusing. \n\nWhat is the circle dot operator? (.)\n\nEq 4 is just interpolation to ensure positive increments. \nWhy do you need to show (3) since it's not good?\n\nTh 1: \nThe definition of the \\Pi operator isn't clear. \n\nBefore Sec 3.4:\n\nIs Relu is a good choice? What is your thought on other functions? How to choose g in practice?\n\n\nThe prediction error would be high\n->The prediction error would be higher\n\neq 11:\nThis is similar to Mavrin's idea: using exploration bonus -- UCT style. \n \n\n\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a non-decreasing quantile functional form for distributional RL, and secondly propose using the distributional error as a means of exploration. The experimental results are very exciting. The paper, however, needs further work before acceptance: the reviewers raised concerns about Theorem 1: a full proof is not included (nor written convincingly during discussion), and while several encouraging experiments were added during the discussion to the paper addressing the reviewers concerns, they fell short (understandably, given the time available).

Thus on this basis, I recommend rejection at this time, but think it likely that with these adjustments the paper will be accepted in future.
Decision: Reject