Distributional Reinforcement Learning with Monotonic Splines
Authors: Yudong Luo
Guiliang Liu
Haonan Duan
Oliver Schulte
Pascal Poupart
Venue: ICLR 2022 Poster (2022)
Keywords: Distributional RL

Abstract: Distributional Reinforcement Learning (RL) differs from traditional RL by estimating the distribution over returns to capture the intrinsic uncertainty of MDPs. One key challenge in distributional RL lies in how to parameterize the quantile function when minimizing the Wasserstein metric of temporal differences. Existing algorithms use step functions or piecewise linear functions. In this paper, we propose to learn smooth continuous quantile functions represented by monotonic rational-quadratic splines, which also naturally solve the quantile crossing problem. Experiments in stochastic environments show that a dense estimation for quantile functions enhances distributional RL in terms of faster empirical convergence and higher rewards in most cases.

Published as a conference paper at ICLR 2022
DISTRIBUTIONAL REINFORCEMENT LEARNING WITH MONOTONIC SPLINES
Yudong Luo Guiliang Liui.1, HHHonan Duan Oliver Schulte?, Pascal Poupart University of Waterloo, Uniiversity of Toronto, SSimon Fraser University, Veecoor Institute {yudong. 1uo, guiliang. 1iu, p euwaterloo. haonandecs aebronto.edu, oschulte@cs.sfie sa
ABSTRACT
Distributional Reinforcement Learning (RI.) differs from traditional RL by esti- mating the distribution over returns to capture the intrinsic uncertainty of MDPs. One key challenge in distributional RL lies in how to parameterize the quantile function when minimizing the Wasserstein metric of temporal differences. Ex- isting algorithms we propose to learn smooth continuous quantile functions represented by mono- tonic rational-quadratic splines, which also naturally solve the quantile crossing problem. Experiments in stochastic environments show that a dense estimation for quantile functions enhances distributional RL in terms of faster empirical con- vergence and higher rewards in most cases.
step functions or piecewise linear functions. In this paper,
INTRODUCTION
A fundamental problem in traditional value-based RL is to estimate the expectation of future re- turns (Mnih al. 2013 Van Hasselt al.) 2016). Distributional RL differs from this by also taking into account the intrinsic randomness of returns within MDPs (Morimura et al.] 2010 Bellemare al., (017). To do so, distributional RL algorithms characterize the total return as a random vari- able and estimate its underlying distribution. In contrast, traditional value-base RL algorithms focus only on the mean of the random variable.
Distributional RL offers several advantages over value-based RL that computes only expected re- turns. The distribution of returns enables risk-sensitive RL. by facilitating the optimization of other statistics than just the mean of the returns (Dabney et al.) 2018a Martin et al. 2020). Even when we stick maximizing the mean of the returns, the distribution offers more reliable and robust way of computing the expectation, which has led to series of records
the Atari benchmark
among value-based non-distributed RL techniques (Bellemare et al., 2017| Dabney et al. 2018a b) Hessel et al. 2018 Yang et al. 2019: Zhou et al. [020 Nguyen et al. 2021p. Intuitively, while is sufficient to represent an expected return by single mean value, errors due to finite samples and function approximations can be reduced by "canceling" each other when multiple sample returns quantile values are used. This is similar to the benefits of ensemble learning techniques although, strictly speaking, distributional RL been combined with ensemble learning and truncated critic predictions to mitigate overestimation bias in continuous control (Kuznetsov al. 2020).
an ensemble RL technique. In fact, distributional RL has
One key aspect of distributional RL algorithms is the parameterization of return distributions. In Categorical DQN (C51) (Bellemare et al., 2017). the return distributions are limited to categorical distributions over a fixed set of discrete values. It is also shown that the distributional Bellman operator a contraction under the maximal form of the Wasserstein metric, but practice, C51 optimizes the cross-entropy loss with a Cramér-minimizin, projection (Rowland et al.) 2018). To bridge the gap between theoretical analysis and algorithmic implementation. quantile regression (QR)-based distributional RL algorithms (Dabney al. 2018a Yang et al.) 2019 Zhou et al.) 2020) estimate a finite number of quantile values instead of the distribution of returns since quantile regression can easily the Wasserstein metric as the objective. In fact, the Wasserstein metric is approximately minimized by optimizing the quantile Huber loss (Huber 1992) between the Bellman updated distribution and the current return distribution.

---

Published as a conference paper at ICLR 2022
Although with an infinite number of quantiles, the step quantile function in those quantile regres- sion based methods will approximate the full quantile function arbitrarily closely, in practice, it is infeasible to have infinite quantiles in most existing architectures. In addition, the quantile crossing issue, recently pointed out and solved by (Zhou et al. 2020), was ignored by previous distributional RL techniques. The issue is that if no global constraint is applied, the quantile values estimated by a neural network at different quantile levels are not guaranteed to satisfy monotonicity, which can distort policy search and affect exploration during training (Zhou et al. 2020).
In this work, we propose to learn a continuous representation for quantile functions based on mono- tonic rational-quadratic splines (Gregory Delbourgo 1982). The monotonic property of these splines naturally solves the quantile crossing issue described above. Furthermore, unlike step func- tions or piecewise linear functions that provide a crude approximation in each bin, monotonic splines provide a more flexible and smooth approximation. With sufficiently many knots, splines can approximate any quantile function arbitrarily closely. We compare empirically our spline-based technique with other quantile-based methods in stochastic environments. We demonstrate that our method offers greater accuracy in terms of quantile approximation, faster convergence during train- ing and higher rewards at test time.
DISTRIBUTIONAL REINFORCEMENT LEARNING
In standard RL settings, agent-environment interactions are modeled MDP, represented tuple (S, A, R, P. 7) (Puterman 2014). S and A denote state and action spaces. P(-|s, a) defines the transition.
the state and action dependent reward, and y (0, 1) discount factor.
For a policy ग,, the discounted sum of returns is denoted as random variable z (s.ssa) = Ei=o R(s,, at), where 8o 5, ag a, St+1 P(-|st, dz), and a, TT-[se). The Q-value (state-action value) is defined as Q" a) = Eiz*(ss (s. a)]. The optimal Q-value, ("(s.al max, Q (s, a), is the unique fixed point of the Bellman optimality operator T (Bellman 1966
Q' (Ts.a)TQQCGG as) [E[(s,, a)] Ep max max (5',
o'
In most deep RL studies, Q is approximated by neural network. To update Q, Q-learnin trains the network iteratively minimize the squared temporal difference (TD) error
t [re 1 y max Qx- (s', Qo(s,a))
where o- is the target network which is updated periodically with the most recent Φ).
Instead of learning the scalar Q(s,a), distributional RL considers the distribution returns (the law of 2) to capture the aleatoric uncertainty (intrinsic stochasticity in the environment). A similar distributional Bellman operator for Z can be derived as (Bellemare al. 2017
Z a)
D R(s.a) +Z(("'
(3)
with P(-(s, a), A' T(-IS') and X indicates that random variables X and Y follow the same distribution. In theory, Bellemare et al. (207)) proved the distributional Bellman operator is contraction in the Wasserstein metric
W (x( = (L
(4)
where F-1 is the quantile function (inverse cumulative distribution function). Following this theory, a series of distributional RL algorithms have been proposed based on quantile regression estimate F-1 at precisely chosen quantile fractions, such that the Wasserstein metric is minimized.
2.1 QUANTILE REGRESSION FOR DISTRIBUTIONAL RL
In QR-DQN (Danney et al. 201 8b), the random return is approximated by a uniform mixture of N
Diracs
1 N
Zo(s, a) = 50.(s.a)
(5)
N
1=1
2

---

Published as a conference paper at ICLR 2022
with each 9 set to fixed quantile fraction, f بي for 1 < i I N, and i/N. The quantile estimation is performed by minimizing the quantile Huber loss, with threshold 17
P (61z)
(6)
on the pairwise TD enror 8ig = - -0, , a²)
0:(s, a), where
p7( = |7 - -Iócol EycolLn000 (6), with
(,) { 2002
|6| n
(7)
(1(8 1)) otherwise.
Based on QR-DQN, Dabney et al. (2018a) proposed to sample quantile fractions from a base distri- bution, e.g. E ((0 1J) rather than fixing them. They built implicit quantile network (IQN) to learn mappings from sampled probability embeddings to corresponding quantile values. FQF (Yang)
al., 2011 further improves IQN by learning a function to propose 7's. However, the quantile values generated by neural networks may not satisfy the non-decreasing property of F-1 (known as the quantile crossing issue). This was recently solved by SC-QR-DQN (Zhou al.) 2020), by applying softmax, followed by cumulative sum of the output logits of the neural network $2, and then rescaling by multiplying a non-negune non-negative factor (s,, a) and adding an offset o))
0g(s, a) aa(s,) a) x bi, + 3(8, a), with
(8)
4,a Xj.a: and Xj,a = softmax(11(s)).
jm0
One recent method NDQFN (Zhou et al. 2021) further combines the ideas of NC-QR-DQN and IQN to learn monotonic function for F by connecting the neighboring two monotonic quantile data points with line segments. Different from NC-QR-DQN, NDQFN generates monotonic quantile values by first learning baseline value and then adding non-negative increments.
2.2 OTHER DISTRIBUTIONAL METHODS
Other recent methods investigate different metrics for the distributional Bellman operator. Moment matching, generally parameterized as the maximum mean discrepancy (MMD) between two sample sets in a reproducing Hilbert kernel space (Gretton et al. 20122), is adopted by Nguyen et al. (2021) propose moment matching DQN (MM-DQN). The MMD loss with kernel k derived as:
Σ K(UL, Us) + N2
1
Σ <(()))))) M
2 NM
Σ
x(v,,
where (ui)h1 (ss a) and (w)) R(s, a) 7Z'S' A').
is worth noting that the theoretical analysis by Nguyen et al. (021 shows the distributional Bellman operator under MMD
contraction with commonly used Gaussian kernels exp- prod kernels. It is contraction only when the kernel function is shift invariant and scale sensitive.
Categorical distributional RL was also combined with policy gradient obtain the Reactor algo- rithm (Gruslys et al. 2018) for discrete control and the Distributed Distributional Deep Determin- istic Policy Gradient (D4PG) algorithm (Barth-Maron et al. 2018) for continuous control. Subse- quently. Singh et al. (2020) replaced categorical return distributions by samples in Sample-base Distributional Policy Gradient (SDPG), yielding improved sample efficiency. The return distribu- tion can also be represented by generative network trained by adversarial training (in the same way as GANs) to minimize temporal differences between sampled returns (Doan et al., 2018 Freirich et al. 2019). While most distributional RL techniques compute state-action return distributions, Li & Faisal (202)) proposed the Bayesian Distributional Policy Gradient (BDPG) algorithm that computes state return distributions and uses inference derive curiosity bonus. In another line of work, Tessler et al. (2019) introduced the Distributional Policy Optimization (DPO) framework in which agent's policy evolves towards distribution over improving actions.

---

Published as a conference paper at ICLR 2022
Knots Legit Natwork
Feeture Extractor
Sefiphas
monatinnic splines
Sottmec
FC
Ti.a
state
Cork
Zonv Fo/Reuu
FolResu
[
Sofmes
FC/ RuLU
eep
quantile function
Bin Scale
Figure 1: System Flow of SPL-DQN architecture
OUR ALGORRIHM
Our method is originally motivated by NC-QR-DON, where special architecture is designed for the last layer of the neural network satisfy the monotonicity of F-1. The output represents the estimated quantile values at chosen quantile fractions. One drawback of discretization is that precise approximation for F-1 may need infinite fractions. But in practice, one can only use finite quantile fractions to estimate quantile values for decision making. In this work, we propose to learn a dense approximation for F-1 using monotonic antional-quadenn splines (Gregory & Delbourgo 1982) as a building block.
3.1
QUANTILE APPROXIMATION WITH MONOTONIC RATIONAL QUADRATIC SPLINES
Monotonic splines produce a monotonic interpolant to a set of monotonic data points (called knots) ((r,, Vi k= which has been recently used as transformation function in normalizing
flows (Durkan al. 2019; Dolatabadi et al. 2020). Furthermore, denote (dæ k==y set of posi-
tive numbers, as the derivative of each knot. The monotonic rational-quadratic splines aim to find
rational-quadratic functions with form f& (z)
Oz(z)
Px(x] to fit the points and derivatives in each inter-
val (called bin) [zx; 1k]1]. where Oş and Pk. are quadratic functions (with form ax² + + c).
1982) suggested to construct O₂ and Ps. follows. Denote 9v (yk+1 (x)/(=x+1 - IR) and hy ( = (x #x)/(xx41 - x&) for x E [æ,, r2+1]. The expressions for the quadratic (x(h((=)) and (x) for the 4 bin is defined by (use hę for short of h4 (x:)):
Gregory & Delbourgo
(a(h)) 9 h (y
(4+1)/+( h&) + 91 h2)²
(10)
(hx))) 9* + (dx+1 + d2 2gk) 10--ha hx)
Then, the aational-quadratte function for the ttth bin is computed by the quotient of O₂ and Ps Oz((ha) (hx) Pa(h)) yk + (yk+1 noglghh + d (1 h&)]
(11)
9k + (dx+1 + dN 2gN )h (1 hx)
Equation 11 is proven be monotonic and continuously differentiable, while passing through the knots and satisfying the given derivatives at the knots (Gregory & Delbourgo 1982).
The monotonicity of the above splines fits the non-decreasing property of F-1, Let F-1 ((..)) (7) be the quantile function for the random variable of the discounted total return Z(s, a) with T E [0. 1]. Given the number of bins K, the aims of the spline approximator for Fz are threefold. First, propose partition for the domain of definition [0, 1] with To <... < Tk and тк = 1. Second, estimate the corresponding quantile values yo < ... < qs <... < qK. Third, assess the derivatives at those points with do. ..., de...., dg. We give small fixed positive value for do and dæ as they are assigned with endpoints. After the generation these three sets of statistics, the monotonic spline of each bin is given by Equation [T replacing I& by т and yk by qc).
< TK. Here
0
3.2 MODEL IMPLEMENTATION
We now show how to learn the monotonic splines for quantile functions in distributional RL by neural networks, and name the technique spline DQN (SPL-DQN). As shown in Figure T the
4

---

Published as a conference paper at ICLR 2022
SPL-DQN consists of three major components, including Feature Extractor which extracts latent features from a state, a Knots Logit Network which, for each action, generates the logits of the widths and heights for K bins, and derivatives for K - inner knots, and a Bin Scale Network which recovers the heights in [0, 1] to the original quantile range. Here we describe the model for a discrete action space of size |A]. To use monotonic splines in continuous control, the model can be modified by taking state-action pairs as input and only producing knots for that state-action pair.
The Feature Extractor F is usually made up of multiple convolutional layers with subsequent fully- connected layers for image-like inputs or stacked fully-connected layers for non-image inputs. It produces the feature embedding F(s) (s) E I2 of state s. Then the Knots Logit Network W maps F(s) to unconstrained logits with dimension |A| x (3K - 1) using a fully-connected layer. The vector Ua for each action a is partitioned as Pa W Va .. where w/ and vf have length K, and ve has length K - 1. Instead of directly learning Tkta and yk,a associated with each monotonic knot, we propose to learn the normalized width and height of each bin. Here, vectors " and v are each passed through softmax function and are interpreted as the normalized widths and heights. Vector ue is regarded as the derivatives, and is passed through softplus function to satisfy monotonicity.
With the width and height of each bin, Tk, and Qs.a of each knot can be easily calculated by a cumulative sum. Since the values of wW and w. fall into [0, 1], each Tka (k > 0) is computed by
Viaa k 1, K; 1,..,,14 IA
(12)
without rescaling as the domain of quantile function is [o, 1] (To,a = 0 by definition). To compute each gs.a. another transformation is required to rescale v.l to a range corresponding to the true quantile values. Inspired by NC-QR-DON, we introduce the Bin Scale Network to generate two adaptive scale factors and 33 by applying a fully connected layer C: Rd - R|4x2 to the state embedding F(s). We compute the exponential of a to ensure the total bin height is positive. Then q0,a Pe and for k > 0, Qk.a is computed by
tk.a = exp((aa) x
11 1.0
+ Bay k K; 1,..,,1A
(13)
3.3 APPROXIMATE WASSERSTEIN METRIC MINIMIZATION
When using continuous approximations of the quantile functions, there are several choices to com- pute the integral of the Wasserstein metric between two quantile functions. We can try to calculate the integral directly, but this is not straightforward for rational-quadratie functions since the integral rarely has a closed form. Alternatively, we can calculate the Riemann integral, but this leads to loss function analogous to the Ll-norm, which may cause instability in training. Thus, in this work, perform quantile regression (Koenker & Hallock 2001) in projected space approximately minimize the Wasserstein metric.
Let = (fo .... FN) be fixed sequence of non-decreasing quantile fractions (note that the set is different from the valuuu of knots in Section 3.1 to partition the [0, 1] domain, which are learned by the neural network. In our experiments, we let be uniformly fixed), we project the monotonic spline quantile function ff to quantile distribution space Zq by computing
Zg(3, a)
Σ
i F 1
(14)
where each (s, a) is the corresponding quantile value at the quantile fraction f. f given by f(f,) with < N. To compute f(f;), we first search which bin f, lies in. Then the value is returned by the corresponding spline function given (f;) as input. In this case, the optimal value distribution Z is achieved by minimizing the 1- W-Wasserstein metric with Zg
W (()
1o (w) Cs ldu,
(15)
F
which is equivalent to finding a projection operator Iw, such that 1w Z == arg min (Z, 2g).
(16)
ZgE2Q
5

---

Published as a conference paper at ICLR 2022
Quantil fanctions cyan equarr slade
Quantiiu fiarstoons of yullow di stadt
(e)
[0]
MC on
Ouamie value
10
Quande velue
1
-15
NC.OR
HD07H
-30
maves
OR
2.0
08 1.0
0.5
0.2
0.4
0.0
0.5
Figure 2: (a) Windy Gridworld, with wind strength shown along bottom row. (b) & (c) The quantile functions for value distribution of the cyan square state and yellow circle state by MC, SPL-DQN (SPL), NC-QR-DON (NC-QR), NDQFN, and QR-DQN (QR).
Furthermore, Dabney et al.
2018b) shows that the unique minimizer of this operator is given by
F az (ft) = (1(8,a), f 4-1 + T
(17)
2
Proposition 1. [Proposition 2 in Dabney et al. [018b)] Let Iw, be the quantile projector defined above. When applied to value distributions, it gives a projection for each state-value distribution. For any two value distributions 21, ZZ E Z for MDP with countable state and action spaces,
dx (Tw,, T Z1, N1, T Z2) S gd (7, 7z2,,
(18)
where d,(2,, 22) = sup,,a W (Z (s, a), (2(5, a)) and Z is the space of action-value distributions with finite moments.
Proposition 1 suggests that after projecting f to Zg. the operator Iw, T is a y-contraction under the measure doe and the repetition of this operator converges to a fixed point in space Zo.
Based on Proposition 1, the ultimate goal to estimate quantile values in Equation 17 for Z(e.o) using quantile regression in each training batch. In our implementation, we uniformly fix 4 = (fo...., FN) to be consistent with QR-DQN and NC-QR-DON, which leads to the same quantile Huber loss as shown in Equations 6 and 7 However, the advantage of our method over QR-DQN and NC-QR-DQN is that we can freely enrich the density of F to get better estimation of the quantile function without increasing the size of the model architecture, while QR-DQN and NC- QR-DQN must enlarge the output dimension of their models to get more quantile estimates. Since we can freely query quantile values at any quantile fraction, quantile fraction embedding as done in IQN and FQF is no longer necessary in our method.
Although one recent method, NDQFN, also learns continuous monotonic quantile func- tions, our method is different from NDQFN in three aspects. First, the z-values of those monotonic knots, i.e., To, ..., TK. are uniformly fixed in NDQEN, while they are trainable in our method. Sec- ond, by also learning the derivatives at each knot, we get a smooth interpolant over the entire domain, while NDQFN connects those knots with line segments, which has limited approximation ability. Third, to get the increments of y-values of those knots, i.c., 90. ..., QK. NDQFN learns a function taking the quantile fraction embeddings, i.e., the embeddings of corresponding 7s, input, while we do not calculate increments but use a scale network as discussed in Section 3.2
Remark:
To demonstrate the monotonicity and approximation strength of our method in stochastic environ- ments, we plot the quantile functions learned by SPL-DQN. NC-QR-DQQ NDQFN, and QR-DQN in variant of the classic Windy Gridworld domain (Sutton & Barto 2018 In Figure 2a the agent starts at the yellow circle state and makes standard moves in a gridworld to reach the red flag. A reward of is earned at each step. Some columns are affected by some wind blowing from bottom to top. The orange line shows the optimal trajectory without stochasticity. We set each state tran- sition to have probability 0.1 of moving in a random direction without any wind effect, otherwise the transition is affected by the wind, which pushes the agent northward. All methods here use the same training settings and similar network architectures as discussed in Appendix We compute the ground truth value distribution for optimal policy (learned by policy iteration) each state by performing one thousand Monte-Carlo (MC) rollouts and recording the observed returns as an empirical distribution. Then we transform the empirical distribution to the quantile function as the baseline. Here we show case the learned quantile functions at cyan square state and yellow circle state (start state) shown in Figures 2b and 2c

---

Published as a conference paper at ICLR 2022
All these four methods eventually learn the optimal policy, however their quantile approximations are quite different. Without constraints, quantile functions given by QR-DQN clearly violate the monotonic property, which is known as the quantile crossing issue (Zhou et al. (020). Although NC-QR-DQN applies monotonic constraints, the estimated quantile range is biased towards smaller values according to the quantile functions given by MC, and we observe that the quantile functions learned by NC-QR-DON are straight lines for some states, c.g. cyan square state, which means that fails to learn the value distribution in those states, and in turn this leads to a biased estimation when rescaling
for the start state. The reason for this biased estimation is that in NC-QR-DOOS the quantile range in Equation a ReLU function imposed to the coefficient a ensure it is non-negative. However, this often sets to zero and the quantile distribution will only depend on the shift parameter 3 (which leads to straight line). In this case, the value distribution cannot be precisely captured. For NDQFN, its quantile approximation at the goal nearing state (cyan square state) is close to SPL-DQN, but it overestimates the quantile range at the start state. We also observe the overestimation issue of NDQFN at another state in the middle of the orange line trajectory as shown in Figure in the appendix. Though still exhibiting estimation errors, the quantile functions learned by SPL-DQN are often the closest to the ground truth.
4 EXPERIMENTS
Cartpole
While most previous distributional RL algorithms were eval- uated with Atari games from the Arcade Learning Environ- ment (ALE), it was noted that the ALE is deterministic (Belle-
SPL1
SPL
mare et al., 2017) and therefore questionable as a benchmark to evaluate distributional algorithms that are designed to cap- ture environment stochasticity when there is none. However, note that sticky actions can be used in Atari games to intro- duce stochasticity in policies (Machado et al. 2018) and this regime was used to evaluate IQN and FQF (Yang al. 2019). When the environment is deterministic, value distributions still arise due to stochastic policies, stochastic approximations and random parameter initialization, but the resulting value distri- butions tend be simple and close to deterministic. is also well-known that deterministic environments possess optimal policies that are open-loop and therefore ignore observations (Machado et al., 2018: Koul et al. 2019). In practice, it is often desirable to train controllers with simulators in which noise is injected to increase the robustness of the learned policies in case of discrepancies between the simulator and the real world. Hence, in this work, we modify several robotics environments by adding stochas- ticity, including one discrete environment from OpenAl Gym (Brockman et al.) 2016) and nine continuous environments from PyBulletGym (Ellenberger 2018- 2019-2019). We compare with QR-DQN, IQN, FQF, NC-QR-DON, MM-DQN, and NDQFN. For MM-DQN, we used the (parameter taken from Nguyen et al. (202I))
Running Score
200
NDOFN
NO-GR
150
FDF
100
MMM
30
100
200
300
400 Episade
500
600
700
800
Figure 3: Performance comparison in stochastic Cartpole. Each curve is averaged over 5 seeds.
method
unrectified Kernel N. (zs, y) = || y|1a with instead of the Gaussian kernel recommended by the authors when they tested on Atari games since the unrectified kernel gave better results in the robotics benchmarks used in this paper. summary of how different QR- based methods compute the QR loss is provided in Appendix BB] For a fair comparison, we made sure the same Feature Extractor architecture was used in different models. To simplify acronyms, we omit DQN when referring to a method in what follows.
4.1 DISCRETE CONTROL IN CARTPOLE
We begin our experimental results in a stochastic environment with a discrete action space modified from Cartpole (Florian 2007). The system is controlled by foree of +1 or - applied to the cart. A reward of +1 is returned if the pole remains upright. We set each state transition to have probability 0.05 of moving
neighboring state make the environment stochastic. The QR-based methods use N = 8 quantiles to compute the QR loss. MM-DQN uses N = M = 8 samples. More training details are provided in Appendix A.2
As the episode rewards may vary significantly due to stochasticity, to better reflect the training process, we define the running score as soft update of episode rewards:
running score = 0.99 x running score + 0.01 x episode rewards
(19)
7

---

Published as a conference paper at ICLR 2022
Figure 3 shows the running score curves for stochastic Cartpole. In general, SPL learns much faster (faster empirical convergence) than its counterparts. As discussed before, SPL can freely increase the number of quantiles when performing quantile regression without enlarging the output dimension of the model. We further increase the number of quantiles to 24 to compute the QR loss while keeping the number of bins unchanged (K 8 and N 24), yielding the curve labeled 'SPL11 in Figure 3 This curve shows that approximately minimizing the Wasserstein metric with more quantiles leads better quantile approximations and increases the learning speed and performance of SPL. As NDQFN also learns continuous quantile functions, we do the same experiment (K" 8 and N 24) for NDQFN, whose training curve is labeled by 'NDQFN1' in Figure 7 in the appendix. Although its training performance improves, SPL with N = 24 is still better.
4.2 CONTINUOUS CONTROL IN PYBULLETGYM
PyBulletGym provides RoboSchoo | which is free port of MuJoCa The state of these environ- ments contains joint information of a robot and an action is a multi-dimmensonnal continuous vector. We take nine environments from RoBoSchool and make them stochastic by introducing Gaussian noise M(,, NoiseN((u, .)) to both the location and velocity of each part of the robot, with μ = 0 and a varying in different environments. We choose reasonable for each environment such that robots won't 't ex- hibit unrealistic motion. That is, for noise sensitive environments, such as Walker2D and Humanoid, use smaller a, and for relatively easy tasks, like Inverte/Pendullm one. The noise setting for different environments is shown in Table 3 in the appendix.
we choose bigger
To evaluate continuous control tasks, combine distributional RL with DDPG (Lillicrap al. 2016) by modifying the critic, as done by Zhang & Yao (201)) Instead of learning Q, the critic learns the distribution Z directly. To handle continuous actions, the critic takes state-action pairs as input. As an exception, for the Humanoid environment, we combine distributional RL with SAC (Haarnoja et al. 2018) due to the fact that DDPG is not as good as SAC for this environment. To update the actor in DDPG and SAC, the expectation of Q values is computed as the expectation of quantile samples given by the distributional critic. We refer to the original papers for hyperparameter settings, which are discussed in Appendix A.3 We also include raw DDPG and SAC as baselines.
Figure shows the rumning score curves given by Equation 19 for these stochastic environments. Generally, the training performance varies among different approaches in different environments, however, in most cases, the quantile regression based methods who learn monotonic quantile rep- resentations are better than those whose quantile representations have monotonicity guarantee, which clarifies that the quantile crossing issue can distort policy learning as pointed out by Zhou it always
et al. 2020). Especially, for SPL, apart from Reacher and wwweenddeenuuuuSnnnnnuu SPL performs comparably to NC-QR. Although NDQFN also learns continuous monotonic quantile functions, its performance is even worse than NC-QR in most cases, because NDQEN queries linear functions for quantile samples when computing QR loss, but the approximation ability of piecewise linear func- tion is very limited. For methods with no monotonic quantile guarantee, we notice that although IQN is the best in Reacher, it performs worse InvertedPendullm and Humanoid during training.
converges faster and performs better during training. For InvertedPendulumSSwnnge
To further demonstrate the ability of our method to handle uncertainty of the enviromment, we slightly increase the noise in HalfCheetah to (0, 0.008) (labeled by HalfCheetahl) and (O. 0.01) (labeled by HalfCheetah2). The training curves in these two environments are shown in Figure On average, QR and MM for DDPG behave poorly in these three HalfCheetah variants. The enhanced randomness of environments degrades the training performance of SPI., but SPI. is generally faster and better than NC-QR and IQN, thanks to more precise quantile approximations.
During training, when selecting actions to induce exploration in DDPG. At the evaluation stage, the methods are executed with only exploitation (without action noise). We test the best models we get after training for each method, and the testing score across different environments are shown in Table 6 in Ap- pendix A.3 Apart from Reacher, SPL outperforms its counterparts in all other domains. For the first two environments, although the training performances vary significantly among different methods, the testing scores of their best models are close each other. For most remaining environments, the
-hhlehhch noise Ou o') (Uhlenbeck & Ornstein 1930) is utilized
hhtps://wopnai.ccommmhgggooooooo hhtp://www.mujcco.rgg
8

---

Published as a conference paper at ICLR 2022
InvertedPendul..m
Inverte@DDudeeeenddlu
InvertedPend Swinggng
800
SPL
NC-OR MJGGF
5000
NC
800
Ruiting Scate
Running Spore
4000
KIN
NDOFF
Rurning Stote
FDF
2000
CR
200
DRH
0:3P0
D0PG
-00
205
+000
200
800
1003
Epleccie
Reacher
Walker20
HallCheetah
500
NC
Running Score
12.5
Rumning Score
400
ICH
Rumning Boore
-00
ION
7.5
200
FOF
CR
900
OR
FDF
6.2
200
MM
100
100
0.0
2000
200n #000
3020
1000
1000
2000
3200 Epreode Hopper
4500
5002
805
1500
1550 2000
2600 3000
3500
Epsiodo
Epleode
Ant
Humenokk
8PL
800
SPL
8PL
NZ
NCS-01
NDCIT
NDCFN
Running Stone
Runming Score
PQF
600
FOF
Running
OR
QR
MW
200
DDPG
200
400
000
800
1000
4000
5000
2000
4000
0000
8000
10000
Eptsode
Episade
Epicode
Figure 4: Performance comparison in stochastic RoboSchool. Each curve is averaged by 7 seeds. The first eight environments solved with DDPG, The last one is assigned to SAC.
HalfCheetaht
HallCheetah2
600
S00
800
NJDG-N
NJGORN
400
ION
-20
ION
FOF
Rurning
220
FOF
1200
100
2000
Eptsode
Figure 5: Performance comparison in two stochastic HalfChectahs with enhanced randomness.
testing scores of SPL and NC-QR significantly better than other methods. Appendix provides additional results for SPL and NDQFN when trained with uniformly spaced quantile fractions and random quantile fractions sampled from U(0,, 1)).
Limitations: Our technique assumes full state observability. This is a limitation that could be re- moved in future work by extending it to partially observable domains. Similar previous work in distributional RL, there is computational cost associated with representing the value distribution. Hence our technique does not scale well non-distributional RL techniques. While we demon- strated our spline approximation in combination with DQN, DDPG and SAC, our distributional RL technique is limited to RL techniques that include a critic.
5 CONCLUSION
Based previous works in distributional RL, propose more general and precise approximation for quantile functions using monotonic rational-quadratic splines when minimizing the Wasserstein metric. With monotonic continuous representation of the quantile function, the quantile value at every quantile level is accessible during training. yielding greater accuracy. One direction for future work is to investigate other monotonic function approximators for quantile estimation.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We acknowledge the funding from the Canada CIFAR AT Chairs program, the Natural Sciences and Engineering Research Council of Canada (NSERC) and support from SportLogiq. Resources used in this work were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute httpppp://eeecoornsstiuuus ai ai/partners/
ETHICS STATEMENT
We confirm that our work has no ethics issue.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we have discussed the detailed experiment settings in Appendix A The code for the main experiments released in the supplementary material.
REFERENCES
Gabriel Marth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, TB Dhruva, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Mare G Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), pp. 449- 458, PMLR, 2017.
Richard Bellman. Dynamic programming. Science, 153(3731):34-317 1966.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In Proceedings of the International conference machine learning (ICML), pp. 1005. PMLR, 2018a.
Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings the AAAI Conference on Artificial Intelligence (AAAI), volume 32, 2018b. 8b.
Thang Doan, Bogdan Mazoure, and Clare Lyle. Gan q-learning, arXiv preprint 2018.
Hadi Mohaghegh Dolatabadi, Sarah Erfani, and Christopher Leckie. Invertible generative modeling using linear rational splines. In International Conference on Artificial Intelligence and Statistics (AISTAT), pp. 4236-4246. PMLR, 2020.
Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. In
Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
Benjamin Ellenberger. Pybullet gymperium.
https:///ithub: wtt.:a..ccobbb/eeooooo
pybullet-gym 2018-2019.
Razvan V Florian. Correct equations for the dynamics of the cart-pole system. Center for Cognitive and Neural Studies (Coneural), Romania, 2007.
Dror Freirich, Tzahi Shimkin, Ron Meir, and Aviv Tamar. Distributional multivariate policy eval- uation and exploration with the bellman gan. Proceedings of the International Conference on Machine Learning (ICMI., pp. 1983-1992. PMLR, 2019.
10

---

Published as a conference paper at ICLR 2022
JA Gregory and R Delbourgo. Piecewise rational quadratic interpolation to monotonic data. IMA Journal of Numerical Analysis, 2(2):123-130) 1982.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola.
A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723- 72033373, 2012.
Audrunas Gruslys, Will Dabney, Mohammad Gheshlaghi Azar, Bilal Piot, Mare Bellemare, and Remi Munos. The reactor: A fast and sample-efficient actor-critic agent for reinforcement learn- ing. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the International Conference on Machine Learning (ICML), pp- 1861-1870. PMLR, 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bital Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 32, 2018.
Peter Huber. Robust estimation of location parameter. In Breakthroughs in statistics, pp. 492- 518. Springer, 1992.
Roger Koenker and Kevin Hallock. Quantile regression. Journal of economic perspectives. 15(4): 143-156, 2001.
Anurag Koul, Sam Greydanus, and Alan Fern. Learning finite state representations of recurrent policy networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overesti- mation bias with truncated mixture of continuous distributional quantile critics. In Proceedings of the International Conference on Machine Learning (ICML), pp. 5556-5566. PMLR, 2020.
Luchen Li and A Aldo Faisal. Bayesian distributional policy gradients. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Inter- national Conference on Learning Representations (ICLR), 2016.
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
John Martin, Michal Lyskawinski, Xiaohu Li, and Brendan Englot. Stochastically dominant dis- tributional reinforcement learning. In Proceedings of the International Conference Machine Learning (ICML), pp. 6745-6754. PMLR, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski. et al. Human-level control through deep reinforeement learning. nature. 51 518(7540):529-533 -332995-333333 2015.
Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Parametric return density estimation for reinforcement learning. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), pp. 368-378--77 068-375,200 2010.
Thanh Tang Nguyen, Sunil Gupta, and Svetha Venkatesh. Distributional reinforcement learning via moment matching. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
11

---

Published as a conference paper at ICLR 2022
Mark Rowland, Mare Bellemare, Will Dabney, Rémi Munos, and Yee Whye Teh. An analysis of categorical distributional reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pp. 29-37. PMLR, 2018.
Rahul Singh, Keuntack Lee, and Yongxin Chen. Sample-based distributional policy gradient. arXiv preprint arXiv:2001.02653 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Chen Tessler, Guy Tennenholtz, and Shic Mannor. Distributional policy optimization: An alternative approach for continuous control. Advances in Neural Information Processing Systems (NeurIPS), 32:1352-1362, 2019.
George E Uhlenbeck and Leonard S Ornstein. On the theory of the brownian motion. Physical review, 36(5):823, 1930.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double qr learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 30, 2016.
Derek Yang, Li Zhao, Zichuan Lin, Tao Qin, Jiang Bian, and Tieyan Liu. Fully parameterized quantile function for distributional reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 6190-6199 2019.
Shangtong Zhang and Hengshuai Yao. Quota: The quantile option architecture for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 33, pp. 5797-5804, 2019.
Fan Zhou, Jianing Wang. and Xingdong Feng. Non-crossing quantile regression for distributional reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems (NeurIPS), volume 33, pp. 15909- 15919. Curran Associates, Inc., 2020.
Fan Zhou, Zhoufan Zhu, Qi Kuang, and Liwen Zhang. Non-decreasing quantile function network with efficient exploration for distributional reinforcement learning. In Proceedings of Interna- tional Joint Conference on Artificial Intelligence (IJCAI), pp. 3455-3461. International Joint Con- ferences on Artificial Intelligence Organization, 2021.
12

---

Published as a conference paper at ICLR 2022
A EXPERIMENT DETAILS
A.I WINDY GRIDWORLD
Training hyperparameters We train SPL-DQN with NC-QR-DON, NDQEN, and QR-DQN in the stochastic Windy Gridworld. The common parameter settings summarized in Table [ The model inputs are the coordinates of the state. The c-greedy parameter decreases by a half every two thousand episodes. To make the results comparable, use the same Feature Extractor for the methods. For SPL-DQN and NDQFN, the number of bins is K = 30.
Table I: Common hyperparameters for SPL-DQN, DR--RR-DN,, NDQFN, and QR-DQN.
Hyperparameter Optimizer
Value
Adam
Learning rate
0.001
Batch size
50
Discount factor (7)
1
Initial e-greedy
0.3
Minimal e-greedy
0.01
Training episodes
30000
Sampling quantiles number
30
Feature Extractor hidden size
[20, 40, 80]
Additional results The learned quantile functions for the green square state are shown in Figure 6 NDQFN overestimates the quantile range.
To further demonstrate the approximation ability of SPL compared with NDQFN, we train SPL and NDQFN with 10 random seeds in windy gridworld, and check the quantile functions learnt by these two methods for the states on the orange line trajectory (apart from the goal state). There are 15 states on trajectory times 10 seeds, which yield 150 quantile functions. For each state, we can use shortest path to determine an upper bound on the return for that state. Whenever a quantile function goes above that upper bound, we have a clear overestimation. For NDQFN, 36. 36.7% (55/150) of the quantile functions yield an overestimation. In contrast, for SPL, only 4% (6/ (500) of the quantile functions yield an overestimation.
Quantile functions green square state
Quanillle valuue
-15
-20
SPL
NC-OR
NDOFN
moves
Domain
Figure 6: The learned quantile functions the green square state
A.2 CARTPOLE
Training hyperparameters We train SPL-DQN with QR-DQN, IQN, FQF, NC-QR-DON, MM- DQN, and NDQFN in the stochastic Cartpole. The common parameter settings are summarized in Table 2 The model input is a vector of length 4, which contains the cart position, the cart velocity, the pole angle, and the pole angular velocity. The e-greedy parameter decreases by 0.00005 every time step. To make the results comparable, use the same Feature Extractor for the methods.
13

---

Published as a conference paper at ICLR 2022
Specifically, for SPL-DQN and NDQFN, the number of bins is K = 8. For FQF, it contains quan- tile fraction proposal network, whose learning rate is 2.5e and the optimizer is RMSProp Yang et al. 2019).
Table 2: Common hyperparameters across SPL-DQN, QR-DQN, IQN, FQF, NC-QR-DON, MM- DQN, and NDQFN
Hyperparameter
Value
Optimizer
Adam
Learning rate
0.001
Batch size
32
Discount factor (7)
0.99
Initial e-greedy
0.3
Minimal e-greedy
0.1
Training episodes
800
Sampling quantiles number (QR based methods)
Samples number (MM-DQN)
8
Feature Extractor hidden size
[128, 128]
Additional results As NDQEN also learns continuous quantile functions, the number of quantile
samples can be enlarged when computing the QR loss. Here we train NDQFN with bin number K" = 8 and quantile number N = 24. The training curve is labeled by "NDQFN1' as shown in Figure 7 The curves labeled by 'SPLI', 'SPL', and 'NDFQN" are taken from Figure 3
Cartpole
250
SPL1
SPL
Running Score
NDOFN
190
100
100
200
300
400
500
800
700
800
Episode
Figure 7: Performance comparison in stochastic Cartpole
A.3 PYBULLETGYM
Noise setting for different environments We introduce different noise levels in PyBulletGym's environments while ensuring that the robots won't exhibit unrealistic motion. The noise settings are shown in Table
Table 3: Noise settings for different environments in PyBulletGym
Environments
Noise
InvertedPendulum
NO 0.02)
InvertedDoubleeendduun
(00,
enertedPendulumSSinn
(((00.05)
Reacher
((0,0.01)
Walker2D
0.000.005
HalfCheetah
NO 0.005)
HalfCheetahl
N(O, 0.008)
HalfCheetah2
(O, 0.01)
Ant
(00,
Hopper
N( 0.003)
Humanoid
NO 0.003)
14

---

Published as a conference paper at ICLR 2022
Training hyperparameters Hyperparameters for DDPG and DDPG based models are summarized in Table 4 The critic also uses an Lg weight decay of 10 10-2. The soft target update coefficient is 0.001. Ainttin-Uhlenheed noise (OU(,', o')) (Uhlenbeck & Ornstein, 1930) is combined with
actions for exploration DDPG, where we use p' 0 and o² 0.1.
Hyperparameters for SAC and SAC based models are summarized in Table 5 The soft target update coefficient is 0.005.
For the critic implemented by SPL-DQN and NDQFN, the number of bins is K 32. For the critic implemented by FQF, the learning rate for quantile fraction network is 2.5e and the correspond- ing optimizer RMSProp.
Table 4: Hyperparameters for DDPG and DDPG based methods
Hyperparameter
Value Adam
Optimizer
Actor learning rate
10
Critic learning rate
10-3
Batch size
64
Discount factor (?)
0.99
Training frames
one million
Sampling quantiles number
Actor hidden size
[400, 300] [400, 300]
Critic's Feature Extractor hidden size
Table 5: Hyperparameters for SAC and SAC based methods
Hyperparameter
Value Adam
Optimizer
Actor learning rate
3 10-3
Critic learning rate
3 x 10-3
Entropy learning rate
3 x 10-3
Batch size
64
Discount factor (?)
0.99
Training frames
three million
Sampling quantiles number
32
Actor hidden size
[256, 256]
Critic's 's Feature Extractor hidden size
[256, 256]
Additional results We test the best models given by different methods with four random seeds. The
results are shown in Table 6 We test all DDPG based agents without Ornstein- -hilehhch noise for 0.525 million frames, and SAC based agents for 2.5 thousand episodes. We treat DDPG and SAC scores as baselines and scale other methods' scores by them, i.e.
method scaled test score method.rawe
(20)
DDPG/SAC.rrau.ees.
B COMPUTING QR LOSS
We summarize how different QR-based distributional RL methods sample quantile values when computing the QR loss. Since the QR loss is computed in TD manner. will need N current quantile samples ((g)), = I,... N) and N' target quantile samples (Lg)) = I,..,,N' corre- sponding two quantile fraction sets ((r: }, 1,... NN) and ((7²). 1,... N'). Without loss of generalization, we consider = N'. Here we discuss the case with discrete actions, and denote the action space by [A].
For discrete quantile approximations, including QR-DQN, NC-QR-DON, IQN, and FQF, in order get N quantile samples for each action, the output dimension of the model is |A] x N for input
15

---

Published as a conference paper at ICLR 2022
Table seeds.
Scaled testing scores across different stochastic environments. Scores are averaged over
Environments
MM
QR
FQF
IQN
NDQEN
NC-QR
SPL
InvertedPendullum
0.911
0.940
0.953
0.970
0.992
0.969
0.999
InvertedDoobbleeendduumm
0.814
0.978
0.975
0.967
0.990
0.993
1.019
Inverte@PendulunSSiinuuu
0.461
0.945
0.223
0.944
1.091
1.145
1.179
Reacher
-1.501
0.412
-10.546
4.269
3,416
4.241
2.972
Walker2D
0.503
0.661
0.585
1.375
0.776
1.732
3.142
HalfCheetah
0.731
1.084
0.809
2.122
1.039
2.932
3.004
HalfCheetahl
0.763
0.897
0.859
1.764
1.156
2.158
2.633
HalfCheetah2
0.834
0.855
0.773
1.741
1.231
1.812
2.21
Ant
0,871
2.283
0.345
2.403
2.388
3.045
3.321
Hopper
0.689
0.868
0.671
0.960
0.893
1.405
1.609
Humanoid
1.077
1.409
0.035
0.044
1.108
1.558
1.640
state. For QR-DQN and NC-QR-DQN, {-]}} and {T.² are assumed to be uniformly spaced. For IQN, { and (T() } are independently drawn from a uniform distribution U([0, 1)). For FQF, (T/}
and [7,] are proposed by quantile fraction network.
For methods that learn continuous approximation of the quantile function, including SPL-DQN and NDQFN, the output (for an input state) consists of knots with shape |A] x (K+1) when the domain is divided into bins. For SPL-DQN, it leanrs the æ, y values, and derivatives of those knots. A smooth continuous function with closed form is obtained in each bin. When sampling quantile values to compute the QR loss, SPL-DQN uniformly fixes {T!} and {T²). and {g. } and { are obtained by querying the closed form with {T(} and {T²} as inputs. For NDQFN, it only learns the y-values of those knots, and the x-values of the knots are uniformly spaced. The continuous function is constructed by connecting neighboring knots with linear functions. When sampling quantile values to compute the QR loss, NDQFN draws {T: } and {-,²} from a uniform distribution U ([0, 1|) independently, and {q. } and {ạe) are obtained by querying the linear functions in each bin.
C COMPARISON OF SPL AND NDQFN WITH TWO TRAINING REGIMES
Figure g shows additional results that compare SPL and NDQFN when trained with uniformly spaced points versus points sampled uniformly at random. In this figure, SPL corresponds train- ing with uniformly spaced quantile fractions as described in the main paper. NDQFN corresponds training with quantile fractions sampled from ( 1J) described (Zhou et 2021). To iron out this difference in the training, we also report SPL-rnd where training is done with quantile fractions sampled from ([0, 1]) and NDQFN-um where training is done with uniformly spaced quantile fractions. We observe that SPL outperforms NDQFN in both training regimes.
D DDPG AND SAC WITH DISTRIBUTIONAL CRITIC
We summarize the algorithm when using distributional critic for DDPG and SAC in this section. For FQF critic, has update the quantile fraction proposal network separately, it is individually described. Similarly, for MM critic, it uses moment matching instead of quantile regression, and individually described as well. For SAC based methods, readers can refer the original paper for how to compute gradient for policy and state value.
16

---

Published as a conference paper at ICLR 2022
Inverte@Pendikumm
Inverte@@oulleeenccee
540
800
3PL-md
Rurning Store
NDOFFH N
NDDFNAA Nilii
Rurning Stote
3000
Rutning Boxe
600
DOPI
3000
-000
200
200
1903
1500 2003 Epreode
200
4000
020
800
1003
Egiiods
Eplaada
Reacher
Walker2D
HailCheetah
SPL
SPL
Rumnin Score
NDGFN
Burning Score
400
000
300
DIPO
Rurning
DOPG
2010
200
100
Episode
1000
2000
Episode
3000
4200
5000
50C
1000
1900
2000 Episode
2500 3500
3000
Ant
Hopper
53L
000
BPL
SPL-m3
620
SPLim
NINGTN
NDOIF
1250
OCP3
800
Ramnng
3000
800
500
250
9000
1000
2000
3000
4000
5000
Epieade
Episode
Figure 8: Performance comparison of SPL and NDQFN when trained with uniformly spaced quan- tile fractions or random quantile fractions sampled from ([0, 1]) in eight environments with DDPG as the baseline
Algorithm DDPG with QR-based distributional critic (apart from FQF)
Require:
Initialize critic network ((s, albb) with weights wh: Initialize actor network (us(0) with weights 0; Initialize target network Z' and p' with weights v/ v. 0' + 0; Initialize reply
buffer D
1: for each episode do
2: Initialize random process OU for action exploration
3: for each time step do
4: ae uts; ()) Oul,
5:
81++1 ~ pl p( 81, ax)
6:
D D {sL, 0t, ((t, 0z) (e), 8t+1)]
7:
Sample minibatch of N transitions (st, a,, Ti, 81+1) from D
8:
Choose current quantile fractions {7} according critic's strategy
9:
Compute corresponding current quantiles {q:} < [(s,, a,/1)
Choose target quantile fractions { according to critic's strategy
11:
Compute corresponding target quantiles {Ø++1} } + r, + (s#11,
12:
Update critic by minimizing QR loss
13:
Compute expectation of quantiles Qa,a < E(((ssal(3))
14:
Update actor by
15:
Vp. z N Σ, V.Q.
8. V. (.00 Voplssss VoH(s)u))2.
16:
Update target networks d < σy (1 )
17:
18:
- 08 + (1 - σ 0
end for
20:
end for
17

---

Published as a conference paper at ICLR 2022
Algorithm 2 DDPG with QR-based distributional critic (FQF)
Require: Initialize critic value network Z(s, alv) with weights s;; Initialize actor network p)s|0)
with weights 0; Initialize target network and p with weights of < v O' + 0: Initialize
reply buffer D
1: Initialize critic fraction proposal network P(s,alw) with weights w and its target P'(s, (8,)
with weights w'
2: for each episode do
3: Initialize random process Ou for action exploration
4: for each time step do
5: Oe ~ u1s;, (0) OU,
6:
81+11 p0 ax)
7:
D D DU (8,, at, ((, 01),8 8141)}
8:
Sample minibatch of N transitions (s,, a, Ta, 81+1) from D
9:
Compute current quantile fractions {T} f- P(s,, a,Wa
10:
Compute corresponding current quantiles {q:} < ((s;, aj (b)
11:
Choose target quantile fractions [7] <- p( (8i+1 (w)
Compute corresponding target quantiles { r; (5 (011))))
13:
Update critic fraction proposal network by
14:
27(r:] Z [r.] - 11. F4T1
DF.
15:
Update critic value network by minimizing QR loss Compute expectation of quantiles Qa,a (s,
17:
Update actor by
VaJ= Update target networks
Σ, s...slsss..amatt VoH(s)H))&.
19:
v/ < oyh (1 //
21:
0' P o0 (1 0
w' -- ow (1 o'w
23:
end for
24:
end for
Algorithm DDPG with MM distributional critic
Require: Initialize critic network (ss, alı²) with weights tे; Initialize actor network m(s(0) with
weights 0; Initialize target network Z' and p' with weights w/ vb. 0' <- 0; Initialize reply
buffer D
1: for each episode do
2: Initialize random process OU for action exploration
3: for each time step do
4: Ot u(ss (6) OU
5:
87411 ~ p( 8t, a.)
6:
D DU (st,, at, (8t, (e), 1t)
7:
Sample minibatch of N transitions (8i, U;, Ti, 81+1) from D
8:
Compute current Q samples {qi} Z(s a; ())
9:
Compute target Q samples {9++1} <- + 7Z' ($141, ((
Update critic by minimizing MMD loss
11:
Compute expectation of Q values Qua + E[Z(s,
12:
Update actor by
13:
VpJ s NV E. Update target networks
di < σy (1 o Vou(ss))))) Vou(s)u))::
14:
15:
16:
- ol (1 - o)
17:
end for
18:
end for
18

---

Published as a conference paper at ICLR 2022
Algorithm 4 SAC with QR-based distributional critic (apart from FQF)
Require: The learning rates An Åz, and λy for functions nT. Zw and Ve; Initialize parameters' 8, th di; Initialize reply buffer D
1: for each iteration do for each time step do
2:
3:
Q4 ~ [T((s)))
4:
81+1 p(82411411 ax) D (ss,, az, r(t,, oz), 8t41)}
5:
6:
end for
7:
for each gradient update step do
8:
Choose current and target quantile fractions according critic's strategy
9:
Compute current and target quantile values (for computing Jz)
v 4 to \v V Jv
11:
Wi
Az V (z(zz((s)) for E [1,2) (Jz is QR loss)
12:
0+0- A9 2 (0)
13:
16 4 oy + σ )
14: end for
15:
end for
Algorithm SAC with QR-based distributional critic (FQF)
Require: The learning rates Ag. λz, and xy for functions ne. Zw and Vy: Initialize parameters 0, w. 4/, vù Initialize reply buffer D; The learning rate Xp for fraction proposal network Pa: Initialize parameter e
1: for each iteration do
2: for each time step do
3:
Ot ~ TT()))
4:
81+1 p(8 at) D + DU [8,, a,, r(8,, oz), 8141)}
5:
6:
end for
7:
for each gradient update step do
8:
Compute current and target quantile fractions using Ps Compute current and target quantile values (for computing J2)
9:
va - to Av V. Jybb
11:
<- V ( (Jr is quantile fraction loss)
12:
wi t wi Az V Vzw,)((;) for E {1, (Jz QR loss)
13:
- 0 An (0)
14:
v + ou + (1 u)
15:
end for
16:
end for
Algorithm 6 SAC with QR-based distributional critic (MM)
Require: The learning rates An Åz, and λy for functions Ta. Zw. and Ve; Initialize parameters 8 th, yi; Initialize reply buffer D
1: for each iteration do
2: for each time step do
3:
Oe ~ (5{)
4:
81+1 p(s at)
5:
D - D D {(*,, at, r8t,, 0z), 8t41)]
6:
end for
7:
for each gradient update step do
8:
Compute current and target Q value samples (for computing Jz)
9:
sb + t - >v V Jv
wi w; >o
(w;) for E {1,2} (Jz MMD loss)
11:
0 - 0 VoJ=
12:
ve + oy + (1
13:
end for
14: end for
19


Reviews:
[{"review_id": "w6dJQ6m-i9o", "reviewer": "Reviewer_Gqyh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Novelty**: The idea of using monotonic rational-quadratic splines to interpolate an inverse cdf function is novel and interesting. I enjoyed reading this paper. \n\n**Correctness**: The methods are sound and derivations are correct. \n\n**Technicity**: Though the idea is novel, the technicity in the paper is quite low as the paper applied existing techniques to the quantile-crossing problem and combined several existing elements. The most significant technicity in the paper is perhaps the neural architecture to model the monotonic rational-quadratic splines, though this modelling is quite natural. \n\n**Clarity**: The paper is well written and organized. One comment for improving the clarity is that in defining $f_k$ in Eq. (11), it is more clearer to say that $h \\in [x_k, x_{k+1}]$, and replacing $h$ in the RHS of Eq. (11) by $(h - x_k) / (x_{k+1} - x_k)$. The reason is that in the paragraph after Eq. (14), the paper talked about computing $f(\\hat{\\tau_i})$ and searching for the bin of $\\hat{\\tau_i}$. In the original form, $\\hat{\\tau_i} = h(x)$ which is not possible to search for a bin with this representation. \n\n**Experimental significance**: The paper demonstrated that their method, when used in actor-critic framework for continuous control, better approximates the inversed CDF and better performance than the other distributional RL methods in most experiments. What I am concerned about is that why didn't the authors test their algorithm on Atari as most distributional RL methods did, so that the comparison is more standard? Of course, the Atari testbed has its own problem as the paper also discussed but it is a nice testbed for distributional RL because only valued-based RL methods are compared, this performance is more reflective of whether the value estimation is good. In the actor-critic framework, the final performance also depends on the actor; thus if SPL+actor > IQN + actor, it is not clear if SPL alone is still better than IQN alone in value-based RL setting. ", "summary_of_the_paper": "This paper proposes to use monotonic rational-quadratic splines to interpolate an inverse cdf function for distributional RL. They proposed a neural architecture to implement such interpolation and achieved a simplified yet effective solution to the quantile crossing problem in distributional RL. ", "summary_of_the_review": "The paper makes a novel contribution in terms of using monotonic rational-quadratic splines to interpolate an inverse cdf function for distributional RL. The empirical result is very promising. I suggest weak acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "spfPVfiuDg", "reviewer": "Reviewer_zyuH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- the proposed parameterization is continuously differentiable, in contrast to previous piecewise linear approaches\n- paper is clearly written and well positioned with respect to prior work\n- experimental results show significant improvements over previous approaches in many environments with added noise to assess robustness \n\nWeaknesses:\n- novelty is low with respect to NDQFN, which uses piecewise linear interpolation instead\n- it is a bit disappointing that it ends up considering a fixed set of quantile levels for the loss computation, which makes the whole approach equivalent to a fancy parameterization of a fixed level quantile network \n- it lacks some kind of analysis of the origin of the improvement over NDQFN. \n\n\n### Detailed comments:\nIt is claimed that this approach \"offers greater accuracy in terms of quantile approximation\" and is \"a more general and precise approximation for quantile functions\" .\nThese claims seem to be based on the visual assessment of Figure 2. I think they should be more strongly supported with some approximation result comparing splines to piecewise linear functions, with an equivalent number of degrees of freedom.\nHowever, it is not clear, how a better capability to approximate an arbitrary function is useful since only the quantile values at the fixed quantile levels matter when optimizing the quantile loss.\nRegarding the improvement over NDQFN: How much comes from differentiability ? How much comes from the representation itself (splines vs piecewise linear)? How much comes from the different choice of quantile levels  (fixed in SPL-DQN, and uniformly sampled in NDQFN)?  Also, a greater number of trainable parameters in SPL could also explain the increased performance. \nAlso, why isn't  SPL always superior (cf. Reacher experiment)?\n\n\n### Minor issues/comments:\nEq (4) : large parentheses should be used\n\n\"During training, ${\\cal OU}(\\mu';\\sigma')$ noise...\" : this notation should be introduced\n\nThe combination with DDPG/SAC should be described (at least in the appendix).\n", "summary_of_the_paper": "This paper proposes a new neural network design to represent quantile functions for distributional reinforcement learning, based on smooth rational-quadratic splines. This representation has the advantage of being continuously differentiable.\nThe loss is computed by evaluating the quantile loss on a set of uniformly spaced quantile levels.\nThe benefits are shown empirically on standard continuous control environments, modified with noise to make them stochastic, by using the proposed distributional representation as critic in DDPG/SAC algorithms.  ", "summary_of_the_review": "The method is a natural \"upgrade\" of NDQFN and thus its novelty is relatively low. \nThe experimental results on continuous control environments with noise show better results in most environments.\nHowever, it lacks a real analysis of the causes of this improvement.\n\nFor these reasons, I tend to vote for rejection in this current form.\n\n\n**================== After rebuttal ==================**\n\nAlthough I find the spline approach and the experimental results interesting, \nI still feel that the paper lacks theory to explain the improvement and, also, that novelty is relatively low.\nMy concern regarding the fixed quantile levels used for training has been addressed with the uniform sampling and the new experiments.\n\nTherefore, I increase my score to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ry398P67gkq", "reviewer": "Reviewer_2kiV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and clearly presented. Related works are surveyed in details. Although some existing works have already solved the quantile crossing problem, the paper carefully discusses the difference of such methods and the proposed algorithm, thereby justifying its novelty. In empirical evaluation, some of the standard experimental environment have been modified with randomness injected, and the paper provides clear motivation for such modification (more suitable for evaluating distributional RL algorithms). The extensive experiment results show that the proposed algorithm outperforms the existing ones in most of the test cases and therefore justify its effectiveness beyond novelty.", "summary_of_the_paper": "The paper studies the problem of distributional reinforcement learning. It augments the traditional quantile-based algorithms with monotonic rational-quadratic splines. Such augmentation provides a natural solution to the quantile crossing issue, which exists for many other quantile-based algorithms. Extensive experiment results are performed to verify the effectiveness of the proposed method.", "summary_of_the_review": "The paper proposes a novel algorithm with strong empirical results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "kN3QNc41TMt", "reviewer": "Reviewer_uP3K", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: SPL-DQN provides an alternative way of parameterizing the quantile function with monotonicity in distributional RL. Specifically, using an architecture (feature extractor + logit network + bin scale network) similar to NC-QR-DQN, SPL-DQN substantiates the non-crossing quantile logit network by using the rational-quadratic splines.\n\nWhile the idea of using rational-quadratic splines is interesting, I do find that the proposed method needs to be better motivated:\n- The benefits of using monotonic rational-quadratic splines as quantile approximators need to be better justified. In Section 3.3, empirical comparison of the four QR-DQN-based methods is provided, and it is mentioned that NC-QR-DQN could suffer from underestimation and degenerate quantile functions (e.g., quantile function being a straight line) in some states, and NDQFN could suffer from overestimation. While I appreciate the above observations, it remains unclear to me whether these observations are indeed fundamental systematic issues caused by the algorithm design. More empirical evidence or qualitative explanation (e.g., which design leads to the observed estimation bias) is required to strengthen these arguments.\n- On the other hand, I think a more detailed comparison between IQN and SPL-DQN is needed, given that IQN could already represent a general class of quantile functions without the quantile crossing problem. Is there any qualitative benefit of using SPL-DQN compared to IQN (besides comparing them vis-a-vis in the experiments)?\n\n\nSignificance: The experimental results show that combining SPL with DDPG improves over the existing benchmark distributional RL methods in multiple tasks of Roboschool. Having said that, in many tasks (e.g., Walker2D, Halfcheetah, Hopper, and Reacher) the improvement of SPL over other monotonic QR-DQN counterparts appears somewhat marginal compared to the benchmarking results (e.g., https://github.com/araffin/rl-baselines-zoo/blob/master/benchmark.md). One possibility is that such performance differences could partially result from the additional stochasticity added to the environment. To clarify this, it could be helpful to add other stronger baselines like TD3 and SAC as a reference.  \n\nOn the other hand, to strengthen the empirical significance of SPL-DQN per se (as SPL as a critic can perform quite differently compared to SPL-DQN), it would be really helpful to provide an empirical comparison in more complex environments with discrete actions (e.g., Atari or MinAtar) in spite of the issue of stochasticity in the first paragraph of Section 4. This could better validate if the original SPL-DQN is indeed strong compared to other distributional RL methods.\n\nClarity: The overall writing and organization of the paper are good. The proposed method is in most places well-explained. There are only a few cases where additional details are provided in the appendix.\n", "summary_of_the_paper": "This paper proposes SPL-DQN, a quantile-based distributional RL method that uses monotonic rational-quadratic splines to approximate the quantile function of the cumulative return (starting from each state-action pair), and the features of using this technique are: (i) The approximated quantile function is ensured to be monotonic; (ii) The monotonic rational-quadratic splines are continuously differentiable and can provide a more flexible class of smooth approximators for the quantile function (compared to piecewise linear or step functions).\nMoreover, the feature (i) resolves the quantile crossing problem of several existing distributional RL methods (e.g. QR-DQN), under which the quantiles output from the neural network may not be monotonic. By conducting experiments with a toy example, this paper also empirically discovers a few potential issues with the relevant prior works that enforce monotonicity (e.g., NC-QR-DQN and NDQFN). Experimental results in both tasks with discrete (Cartpole) and continuous action spaces (Roboschool) are provided to demonstrate the performance of SPL-DQN.", "summary_of_the_review": "This paper proposes a new variant of QR-DQN with ensured monotonicity in the quantile function. Given that the quantile crossing issue is not new and the design of SPL-DQN closely follows NC-QR-DQN, the overall technical novelty is somewhat limited. Despite that SPL appears to outperform other variants of QR methods in some control tasks, the amount of improvement is not very significant compared to the benchmarking results. Therefore, I lean towards rejection for now but am willing to change my score if the authors address the concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes monotonic splines as an improvement on current approaches to parametrising quantiles in distributional RL. The idea is an obvious, natural improvement on what exists, and yields improved experimental results.
Decision: Accept (Poster)