A Cramér Distance perspective on Non-crossing Quantile Regression in Distributional Reinforcement Learning
Authors: Alix Lhéritier
Nicolas Bondoux
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Distributional Reinforcement Learning
Cramér distance
Quantile regression
Wasserstein distance
Non-crossing quantiles
Neural networks
Atari benchmark

Abstract: Distributional reinforcement learning (DRL) extends the value-based approach by estimating the full distribution over future returns instead of the mean only, providing a richer signal that leads to improved performances. Quantile-based methods like QR-DQN project arbitrary distributions into a parametric subset of staircase distributions by minimizing the 1-Wasserstein distance, however,  due to biases in the gradients, the quantile regression loss is used instead for training, guaranteeing the same minimizer and enjoying unbiased gradients. Recently, monotonicity constraints on the quantiles have been shown  to improve the performance of QR-DQN for uncertainty-based exploration strategies. The contribution of this work is in the setting of fixed quantile levels and is twofold.  First, we prove that the Cramér distance yields a projection that coincides with the 1-Wasserstein one and that, under monotonicity constraints, the squared Cramér and the quantile regression losses yield collinear gradients, shedding light on the connection between these important elements of DRL. Second, we propose a novel non-crossing neural architecture that allows a good training performance using the Cramér distance, yielding significant improvements over QR-DQN in a number of games of the standard Atari 2600 benchmark.

A Cramér Distance perspective on Non-crossing Quantile Regression in Distributional Reinforcement Learning
Anonymous Author(s)
Affiliation
Address email
Abstract
Distributional reinforcement learning (DRL) extends the value-based approach by using a deep convolutional network to approximate the full distribution over future returns instead of the mean only, providing a richer signal that leads improved performances. Quantile-hased methods like QR-DQN project arbitrary distributions onto a parametric subset of staircase distributions by minimizing the I-Wasserstein distance, however, due to biases in the gradients, the quantile regression loss is used instead for training, guaranteeing the same minimizer and enjoying unbiased gradients. Recently, monotonicity constraints on the quantiles have been shown to improve the performance QR-DQQ for uncertainty- -bssed exploration strategies. The contribution of this work is in the setting of fixed quantile levels and twofold. First, we prove that the Cramér distance yields a projection that coincides with the 1-Wasserstein constraints, the squared Cramér and the quantile regression losses yield collinear gradients, shedding light the connection between these important elements of DRL. Second, we propose a novel non non-crossing neural architecture that allows a good training performance using novel algorithm compute the Cramér distance, yielding significant improvements over QR-DQN in a number of games of the standard Atari 2600 benchmark.
3
5
7
11
and that, under monotonicity
13
15
16
17
18
Introduction
19
20 Distributional Reinforcement Learning (DRL) extends the value-based approach of DQN [1] by considering the full distribution of returns learning signal allowing to take into account all the complexity the randommess coming from the rewards, the transitions and the policy, which is hidden when considering the mean only. Even when a policy aims at maximizing the expected return, considering the full distribution provides an advantage in the presence of approximations, allowing to learn better representations and helping reduce state aliasing [I]. With this new approach comes generalization of the Bellman operator-the distributional Bellman operator-, whose contraction properties are key for guaranteeing the stability of DRL algorithms.
22
24
25
26
28 How distributions are represented and learned is also key point in DRL, since some choices can break the contraction property (see, e.g., [25] Lemma 21). Some approaches use staircase parametric
30
representations whose steps correspond to fixed quantile values like in C51 [ or fixed quantile levels like in QR-DQN [10]. Alternatively, FQN [30] fully parameterize the staircase distributions.
32 IQN [9] follows a different approach by approximating the quantile function with neural network that takes the quantile level input and must therefore be sampled during training.
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

34
DRL methods resort to different notions of distance or divergences between distributions in order to practically learn the distributions but also to analyze the effect on the contraction property of the distributional Bellman operator. In [25] Hilbert space endowed with the l2 norm on cumulative distribution functions has been shown to be a natural framework to analyze the effect of the fixed quantile value representation of C51. In 2 the squared E2 distance, called Cramér distance in that work has been proposed for Generative Adversarial Networks but also for machine learning in general due its unbiased gradients. However, attempts to use the Cramér distance loss function in DRL, yielded results that are inferior to those obtained with the heuristic Kullback-Leible divergence
35
36
37
38
39
40
41
loss used in C51, reported in [3. In [00], the Wasserstein distance has been used for defining how
43 a general distribution should be represented with fixed quantile levels and also to analyze the effect on the contraction property of the distributional Bellman operator. However, due to the biased gradients of the Wasserstein distance, the quantile regression loss is used to train the network, guaranteeing the same minimizer as the I-Wasserstein distance and enjoying unbiased gradients.
45
46
47
When estimating multiple quantiles, one faces the issue of crossing quantiles, i.e., a violation of the monotonicity of the quantile function. In DRL, crossing quantiles make the learning signal noisy, affecting disambiguation of states as shown in [31]. This issue has been addressed in the statistical literature of quantile regression (see, e.g.
48
49
19. 13 ( but also, more generally, in the machine learning literature on how to represent and learn monotonic functions (see, e.g., [12 Table ID). with different approaches like including penalties in the loss function enforcing monotonicity by design. Methods that take sampled quantile levels as input during training like (27] or 19. have been shown to alleviate the problem. In the DRL literature, [I]] enforees monotonicity with special neural network design obtaining improved results with respect to QR-DQN, in the setting of uncertainty- based exploration.
51
53
54
55
56
57 In this work, we analyze the theoretical properties of the Cramér distance for learning staircase distributions with fixed quantile levels and we propose practical algorithmic solution for DRL with the standard e-greedy exploration strategy. Section 2 we expose the necessary background and definitions. In Section 3 we present the connections between the Cramér distance, the 1- I-Wasserstein distance and the Quantile Regression loss, leading to a contraction guarantee. In Section 4 we propose a complexyyy algorithm to compute the Cramér distance. In Section we introduce novel neural network architecture enforcing non-crossing quantiles. In Section 6 we report experimental results on the Atari 2600 benchmark using the Cramér distance algorithm and the proposed architecture. Finally, in Section we give some concluding remarks. Due to space constraints, complete proofs presented in the supplementary material.
59
61
63
65
2 Background
67
68 We consider the classical model of agent-environment interactions 23 i.e., a Markoy Decision
P. 7.), with S and A being the state and action space, R S x A -> R 70 being the reward function, P(s'|s, a) S x A x S - [0. 1] being the probability of transitioning from state to state s' after taking action a and y E [0. 1) the discount factor. stochastic policy
69 Process (MDP)
71
(() : S A + [0, 1] maps state to a distribution over A.
73
2.1
Q-Learning
74 For a fixed policy म. the return (s, a) is random variable representing cumulative rewards soc the agent gains from (s, by following the policy म, e.g. Z²(ssa) a) = م R (st, dr) with ss, at) a I st). The usual goal in reinforcement learning (RL) is to find an optimal π maximizing the expectation of Z", i.e. the state-action value function
75
76
so = s, Ap = and $1+1 ~ p- (1,
77
78
Q=(x, a) = E 2 as. a). L-Learning (29 is an off- oollcy reinforcement learning algorithm that
79
directly learns the optimal action viauue function using the Bellman optimality operator
(TQ) a) = ER(x, a) + EEp max Q (a', a'EA
(1)
80 In the evaluation case, the Bellman operator T 5
29 is defined as
(T= a) = ER(x, a) +
(z', a') PM
E
(2)
In this work, we follow 25 and use the term Cramér distance for the & distance.

---

81
They are contraction mappings and their repeated application to some initial Qo converges expo- nentially to Qa or Q", respectively (6.. However, when Q represented by neural network that r,, 8') as in most deep learning studies, a gradient update is preferred since it allows for the dissipation of noise introduced the target by stochastic approximation [6] [8]. DQN (21) iteratively trains the network by minimizing the squared temporal 85
82
88
is trained on batches of sampled transitions (s, a,,,,
84
difference (TD) error 1 [r + y max, Qw (s',a') (s, a)]² over samples (s, a, s'), where w the target network, which is copy of w. synchronized with periodically. When using an e-gredy policy, the samples obtained while the agent interacts with the environment choosing actions uniformly at random with probability e and otherwise according to arg max, (8,
87
89
90
2.2
Distributional reinforcement learning
In order to extend the previous concepts to distributional reinforcement learning, the distributional Bellman operator and optimality operator are defined as
82
(T* a) _ R(x, a) + +77 (z')
(3)
(TZ)(s,a) _ ((s, + 7Z (s) (s,arg arg maxlE (s', .d)) ,s' ,s' ~p (s, ~ 1(,, ~
(-1x'),
(4)
a'EA
where Y U denotes equality of probability laws, that is the random variable Y is distributed according to the same law as U. In order to characterize the contraction properties of these operators, some notion of distance between indexed collections of distributions is necessary. The p- Wasserstein distance between distributions U and is defined as the lp metric on inverse cumulative distribution functions (inverse CDFs) [22 i.e.
dr(U,Y) 1000(( ((,15 FT (w])
-(W do
1/P
where for a random variable Y, the inverse CDF Fy! of defined by
(w) inf yy Ri Fy (b)}
93
where (y) y is the CDF of Y Y tween two indexed collections of distributions Zi and Zz is defined as dp (71. 72)
Then the maximal Wasserstein metric be-
94
95
sup dp (Z) (I, a), a)). I Lemma 3] shows that T is a contraction in dp, i.c.
dp (T Z1, T 22) S ydp (Z, Z2)
(5)
96
The case of the distributional optimality operator T more involved. In general, it is contraction I. However, based on the fact that 7 is a contraction, [ proves that, if the optimal policy is unique, then the iterates Zk+1 + T22 converge to 2" (in p- p-Wasserstein metric, Vs, a) and, under some conditions, T has unique fixed point corresponding an optimal value distribution.
37
88
99
100
2.3
Projecting distributions onto finite support
101
Previous approaches of distributional reinforcement learning project return distributions 7(r, a) onto space of distributions of finite support, by modeling it with mixture of Diracs over N support points (1(x, a), 1..N
103
Zo(x, = Σ pc a)
104 which yields staircase CDF (2) EN
E (x a)1 (a.aa) Different approaches have been followed to parameterize these distributions depending on whether p, and 0, are learned or fixed. In this work, we consider fixed and 0; learned parameter.
105
106
In order to analyze how arbitrary distributions are mapped into these finite representations, different projection operators are defined as minimizers of some distance between distribution. For instance,
108
For p 00, doo () supwe|o.a] | (w) 17 (w))-
3

---

Figure I: Midpoint minimizer. a) The curve is approximated by one Dirac (green curve) located at inverse the mid-point. The rectangles represent an approximation of lp distance. b) If we move the Dirac in one direction or the other, the blue rectangle will be replaced by larger one (in red here).
109
in 100, the I-Wasserstein projection Iw,, is used and it shown that the resulting projected Bellman 110 operator remains a contraction, i.e.,
dao (Tw; T Z1, Nw, TT S 7d.0 (Z1 72)
However, since Wasserstein distances suffer from biased gradients 1 the quantile regression
112
(QR) loss is used in practice, guaranteeing the same minimizer and enjoying unbiased gradients [10]
The QR loss for learning the parameters {0I..... 0N) of F(z) = E 1220, given target F
113
N N
(Qn(F,F)
En-s lo+, (Z - 0.)] with po (w() = u(r - 11--1ac luco)
(8)
114 where î are the midpoints of a uniform grid of N quantile levels, i.e. A 2i-1 definition makes 0g an estimate of the f--quantil. As we shall see, this correspondence will not be enforced by the Cramér projection we consider next.
2N Note that this
116
Theoretical results for the Cramér projection
117
Motivated by the practical interest of the unbiased gradients of the squared Cramér distance I (F(z) dzz) we now analyse theoretical properties related to minimizing this quantity in
2
119
120
order to define the Cramér projection and highlight the connections with the I-Wassertein projection
and the QR loss.
122
3.1
Equivalence with the 1- I-Wasserstein projection
123
We now show that, given arbitrary distribution and grid of quantile levels, there is staircase representation that minimizes the Ep. distance, which puts the quantile values at the inverse of the quantile level midpoints.
124
125
126
Theorem 1. Given P: 2 0, = 1. such that E, Pt = 1, the lp distance between F and a mixture of Heaviside step functions (2 DiIPPLL minimized with 0, (1 T2))
127
where Ti are the quantile levels T; = Ej
128
129
Proof sketch. The proof is based on the observation illustrated in Fig. See Appendix for details.
130
For simplicity, we chose 0, FFF + T--1)/2), however any permutation a in the symmetric group of size N makes = (g(()) a minimizer too.
Remark 1.
131
192
We define the Ep projection of an arbitrary CDF F with inverse F-1 onto a grid of quantile levels as
F= F==F Epi with 0, - ((7 1/22)
133 Therefore, is equivalent to the I-Wasserstein projection (see 110 Lemma 2J). This directly implies that the Cramér projected Bellman operator is also contraction.
As in 255 call Cramér distance the & distance and Cramér loss its square.

---

135
Corollary 1. The Cramér projected distributional Bellman operator is contraction in doe i.e. dra (Il,, (0.7"1,,1 Il,,, Z2) S 7doo (Z1, (2).
(10)
136
Proof. It follows directly from Lemma 3] and Lemmal
137
3.2
Collinearity of QR loss and Cramér gradients under non-crossing constraints
138
In order to put evidence the relationship betweem the gradients of the QR and Cramér loss, we first present an alternative formula for the Cramér loss.
139
Lemma 1. Given two staircase distributions F(z) = N Ei 1,20, and F(z) t/ Σ 1l 228, such that 02 <... < ON and 01 <... < 0N- Let Uij =0 0g 0g and dig luyco- The squared Cramér distance between the distributions can be expressed as 142
N
N
i-1 Σ - b j=1
[ (F(2) - -(
Σ
Lud +2
(1I)
N
-=++1
Proof sketch. We resort to a tiling operator to break the integral into pieces. Our demonstration unfold through these steps; First, we prove formally that our operator is well built: the sum of the tiling measured with the operator p equal to the Cramér distance between the two curves. Secondly, we derive Eq. (IT) by using that tiling operator. See Appendix for full details.
144
148
Corollary 2. For F(z) = 1 N
1220, and F(z) = N E 1220, we have
147
① (QR(F.FF)
1 2i
Σ
N
diz
and OE3(F,F)
1
1
2i
+
8is
(12)
00;
2
00,
N2
1=1
j=1
where 6ig- 1 Tuyeo- Therefore, their gradients are collinear, i.e.
N Ve LQR
2 Vel2.
(13)
149
Proof sketch. The results are obtained by differentiating (8) and See Appendix.
150
Remark 2. Therefore, gradient descent methods whose parameter updates are invariant to rescaling of the gradient like ADAM []6]. yield the same optimization path with both losses.
152
Remark 3. Huberization of the QR loss (see (10)) breaks the equivalence with the Cramér loss.
low-complexity algorithm for computing the Cramér distance
153
154
The formula 11] allows to compute the squared Cramér distance which we refer to Cramér loss between two staircase distributions F(2)
EEE Le26,, and F(2) 2/2 ΣN Le20, assuming the quantiles ordered, i.e., 0 <... ON and 0, <... < ŌN. That formula involves two nested sums making it of quadratic complexity in N as the quantile regression loss. Alternatively, if we consider the sorted sequence of merged quantiles 0' sort -10,)==1..N U(O:)==1..v) we have that
156
157
158
159
F(z) constant between any two consecutive quantile values of O' and the difference can be obtained by accumulating the increments from F and the decrements from F, see Fig. f an illustration Therefore, we can express the Cramér loss between two staircase distributions as follows
F(z)
160
161
2
2N-1
(F(() - F(2) Σ (0-11-10
8%)
Σ N
Σ
(14)
N
jst. 0
I j.t.
0, 28
183
In Fig. 2 we propose an algorithm that implements this formula based on sorting the merged quantiles 184 of both distributions, yielding an O(N log N) complexity. Note that this algorithm does not require
165 the input vectors 0 and 0 to be ordered.
'ee Appendix for details.
5

---

Algorithm 1: Cramér loss.
Input: 0 01,.... 0N]
0 ...... NN array
Output: 1_ 1 (F(z) F (2))
0.8
0' con 8)
i ................................ .iN argsort(0')
96
0' <- 0' .......,,
94
A, f 0'[1:] 0 -1]
A, f concat (- N 1N: *
+1
A A iN]
1/1
A f- cumsum [
+- 12
I+ A, A, A
-0.4
0.4
return sum(I)
Figure Cramér loss algorithm. The operators [1:] and [:-1] remove, respectively, the first and the last elements of the array. 1N denotes an array of N ones and * denotes elementwise multiplication.
(a) Cramér loss.
(b) Quantile regression loss.
Figure 3: Symmetry in the Cramér loss (left) in comparison to QR loss (right). The loss land- scape correspond to estimating the return distribution of a state with transitions to states 81 and 82 with probability 1/3 and 2/3, respectively, whose return distributions
Diracs located -0.5 and 0.6 respectively, with N = 3. The plots are for a fixed 0,- = 0.5.. Notice that when Og < 01 < Oz. the two losses have collinear gradients as shown Corollary
A centered non-crossing architecture enforcing ordered quantiles 186
187
5.1
Motivation
188
When using Algorithm I 0, is just location where mass of 1/N is assigned in the estimated distribution, necessarly corresponding to the f--quantll.. Therefore, Cramér distance makes the problem of crossing quantiles as usually described vanish, since equivalent distributions can be obtained with 0 = (()) with any permutation a in symmetric group of size N. Since the order of quantiles is not constrained using this algorithm, we remove an important constraint on the values that the neural network parameters can take: the domain of parameters that lead to valid distribution not reduced. could therefore expect to able to reach better results than when using QR loss (QR-DQN with same number of parameters. However another problem arises, this permutation cquivalence creates symmetries in the loss landscape as illustrated in Fig. 3 These symmetries can hinder the learning process if jumps between symmetric regions occur. In Fig. 5 we show examples of the performance of the QR-DQN network trained with the Cramér loss, denoted as CR-DQN, In particular, in the games of Breakout and Seaquest, CR-DQN shows slow learning curve.
189
170
172
174
176
178
180
For this reason, neural architectures enforcing non-crosee crossinsn quantiles are also of interest when using the Cramér loss since they drastically reduce the search space by removing the symmetries. The non-crossing architecture of NC-QR-DQV (3 is composed of two subnetworks: one network to estimate extreme quantile for 0 and the scale of the distribution, which is essentially equivalent to estimating the quantile т = and another network to estimate grid of quantiles between these
182
184
extremes using softmax. As shown by the classical theory of extreme values, extreme quantiles

---

0,1 1.N for each action
medien Pbyat for each
flip
A f112 for each action
mimmuur cumsum
(IN/2)
activation
Figure 4: Centered Non-Crossing (CNC) architecture.
186
require more samples to be properly estimated. We hypothesize that this can strongly affect the first stages of the training process where high error in the estimates for these extreme quantiles propagates to the more centered and easier to estimate quantiles. In the training curves of Fig. [ we observe indeed slower training, and no training at all the case of the game Seaquest.
187
189
190
Network description
The previous reasons motivate our novel architecture consisting of two parts: one is dedicated to median whose estimation is easier and robust the other estimates the rest the quantiles by accumulating increments/decrementt from the median instead of accumulating from the extreme quantile = like in NC-QR-DON.
192
194
195
Now we formally describe our architecture illustrated in Fig. The first part consists, as in QR- QR-DQN and NC-QR- DC-Q, of multi-layer convolutional operator series of convolutional layers each one followed by a ReLU activation) that is applied to the input state to obtain the embedded state C(s) E Rdd given as input to the two subnetworks M and D defined next. Let Fx.n, denote multi-layer fully connected operator with À hidden layers with 1 nodes each one followed by ReLU activation and final linear layer of dimension D, then the vector of quantiles ( obtained as
198
198
e
=
200
follows
M(e) = Fxm,IXIA(()) E IIII
A(e) ReLU(E (ALUUFAA,NNN-MMIIIC R(N-IIXII
A AT = [(e)[L.([N/2) E) (R(NN/A1-XXI (RNAA (e)
A (+()) [())(([N/2]_ DEN (N-112/2)x1A
((e) concat(fii(MM(e) cumsum (e))), + cumsum(A+(e)) E NxIA
202
where the operators concat, flip and cumsum operate along the dimension 1 (i.e. corresponding to the quantile index) well as the operator [-] that extracts the values for the given indices. As in
OR- DQN, the mean used to select the best action in e-greedy policy is computed N > 0,-
205
Fig.
shows the improvement achieved by our network on three Atari games.
Experiments
208
DQN_ZOO and Atari 2600 benchmark. We implemented our algorithm on top of the DQN_ZOQ 24 207
208
framework, which integrates reference implementations of RL algorithms with the gym/atari-pp py RL environment [8) DQN_ RQN_000 provides pre-computed simulation results for each of these algorithms, each of them being run on seeds and on the full set of 57 Atari 2600 games. It was necessary to copy, within atari-py package, few settings from the Atari Learning Environment (4, in order to make atari-py able to handle the game Surround, and to replace a ROM for Defender:
209
211
7

---

breakout
Proa
680
1009)
Grocrdes
PED1
462
280
389
2093
280
Figure 5: Comparison to NC-QR-DON and QR-DQN architectures. QR-DQN and CR-DQN have exactly the same number of parameters. We denote the NC-QR-DQN architecture with Cramér loss and the standard aesilon-greedy exploration as NC-CR-DQN to avoid confusion with the results reported in [3 that corresponds to the incertainty-basee exploration strategy. Note that given the results of Section 3 it is equivalent to using the QR loss. NC-CR-DO CR-DQN and CNC-CR-DON have a number of parameters within 0.1% ADAM optimizer was used with learning
10 for all.
2.02
1.52-
prortized
3.25
6.75
0.2
0.02
Figure 6: Online training performance, in terms median human-normalizen scores
213
Software-hardware setup. g4dn.2xlarge AWS machines were used to run some of the experiments.
214
By carefully selecting the pairs of games, is possible run two simultaneous experiments, taking around 22 minutes on those machines for two simultaneous iterations. Therefore, for two seeds, 57
215
216
games take around 4180 hours of computation. When possible fit in RAM, three simultaneous
runs can be performed, taking around 24 minutes per simultaneous iterations. These times were achieved by restricting each process to one cpu only (processor affinity), otherwise we experienced cpu-gpu communication inefficiencies. These inefficiencies were also experienced when running more simultancous processes on larger machines. Code and full output the experiments is available http:/:///gtttub.cco.o
218
220
ctc/ cr-dqn
Hyperparameters for CNC-CR-DON For model training,
hyperparameterss with the
223
values used in [00 (epsilon decay, ADAM's parameter, experience replay settings). Each experiment consists 200 iterations. Each iteration made of learning phase (1 million frames), followed by an evaluation phase, on 500 thousands frames. We thus use the same experiment procedure, and the same epsilon hyperparameter than the one used for the experiments provided with DQN_ZOD; also, our
224
225
226
227
neural network architecture uses the same convolution layers than the other algorithms implemented within DQN_ZOO The experiment settings being the same, our experiment performance can therefore
228
be compared to the experiment data provided with DQN aa0 for the other algorithms. Finally, each of layer n = 512 neurons, with N 201. We
230
the two heads of our neural network made of obtained the best results with the learning rate set to 5 x 10-5 See Appendix for more details.
Contenders Our algorithm is compared other pure DRI. algorithms: C51 [1. QR-DQN [TO and IQN 99. QR-DQN corresponds to QR-DQN-1, as named in [10), i.e., it is trained via the Huber quantile loss: quantile loss with non-linear smoothing approximation when close
239
parameterized by = 1. We also consider for reference DQN [21]. Double Q (28), Prioritized
235
236
Experience Replay (26) and Rainbow (15). The latter bases itself on C51 and add many orthogonal
237
improvements like prioritized experience replay. While it is interesting to show their scores beside
238
the ones of pure DRL algorithms, they cannot be used to assess the performance of our algorithm.

---

OPEE1
X00C
20080
3890
30KC
2031
100K
180
200
125
1.5
520
150
17a
200
25
375
200
Frosibiie
morbacura JAVAng
2000
Sgedte
353
0048
100
293
55
skking
113.2
-1000
2080
-2500
330
122
-3008
25
30
15
100
155
250
315
206
24
380
155
350
15
203
15
200
236
175
206
Figure 7: Training performance. Curves show average score and standard deviation over three seeds.
Online training performance Performance during training protocol: this protocol, described in
240
20). puts the emphasis on the learning quality. It consists in using normalized training scores to evaluate the algorithms. Fig. 6lsows the median over the 57 games the human- nurmannnormalized training
241
242
scores for each algorithm. The performance curve for an algorithm is obtained as follows: we first measure, for each run of the algorithm and each iteration, the median of numan-normalle
243
score; we then calculate the mean score value over the runs, apply moving average of 5 iterations in order to smooth the curve. On Fig. [ the shadow areas denote the standard deviation of the scores over the runs. It should be noted that _QN_200 ZO0 provides data for 5 different seeds, for all algorithms and games; while we only did runs with two different seeds using the CRC-CR-DQN algorithm, for each of the normalization of score given by (2]] normalized score agent
244
245
247
rrandom human - raadoom
248
games.
249
where random and human are baseline scores, given for each game. Fig. show some remarkable cases, although these must be taken with caution since the number of seeds is not high enough to
251
draw statistical conclusions. See Appendix for detailed results on the full set of 57 games.
Conclusion
252
In this work, focused learning staircase distributions uniform grid of quantile levels. We showed that learning distributions with the quantile regression loss under non-crossing constraints essentially equivalent to learning with the Cramér loss. In prior work, crossing quantiles have been studied as a problem for interpretability and state disambiguation. However, when using the Cramér loss uniform grid of quantile levels, the problem of crossing quantiles does not exist anymore since quantiles are not identified, representing locations where 1 1/N of the mass is assigned. However, this lack of order generates permutation equivalence generating symmetries in the loss landscape that hinder the learning process making important the use of non-crossing architectures. On
254
256
258
259
280
the practical side, we proposed combination of the Cramér loss and a centered neural architecture enforcing ordered quantiles, yielding significant improvements over QR-DQN in number of games, sometimes beating all the faae-of-the-as pure DRL methods.
262
263

---

References
264
[1] M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective reinforcement learning. In International Conference on Machine Learning, pages 449-458. PMLR, 2017.
266
267
[2] G. Bellemare, Danihelka, W. Dabney, Mohamed, B. Lakshminarayanan, Hoyer, and R. Munos. The cramer distance as a solution to biased wasserstein gradients. arXiv preprint arXiv:1705 10743, 2017.
268
269
[3] M. G. Bellemare, N. Le Roux, P. Castro, and S. Moitra. Distributional reinforcement learning with lincar function approximation. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2203-2211111 PMLR. 2019.
270
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253 2223,,2-9,,97 279.jun 2013.
273
274
[S] R. Bellman. Dynamic Progranming. Princeton University Press, Princeton, NJ, USA, edition, 1957.
275
[6] D. Bertsekas and J. N. Tsitsiklis, Neuro-dynamic programming. Athena Scientific, 1996.
276
[7] D. Bondell, B. J. Reich, and H. Wang. Noncrossing quantile regression curve estimation. Biometrika, 97(4):825-88) 838, 2010.
277
278
[8] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv: 1e266066600440 2016.
[9] W. Dabney, G. Ostrovski, D. Silver, and R. Munos. Implicit quantile networks for distributional reinforce- ment learning. In International conference machine learning, pages 1096-1105. PMLR, 2018.
[10] W. Dabney, M. Rowland, M. Bellemare, and R. Munos. Distributional reinforcement learning with quantile regression. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
285
284
[11] H. Dette and S. Volgushev. Non-crossing non non- parametric estimates of quantile curves. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 70(3):609-627, 2008
285
286
[12] M. Gupta, A. Cotter, J. Pfeifer, K. Voevodski, K. Canini, A. Mangylov, W. Moczydlowski, and A. Van Es- broeck. Monotonic calibrated interpolated look-up tables. The Journal of Machine Learning Research, 17(1):3790-3866 2016.
288
289
[13] Hall, R. Wolff, and Yao. Methods for estimating a conditional distribution function. Journal of the American Statistical association, 94(445)154-163, 1999.
290
291
[14] X. He. Quantile curves without crossing. The American Statistician, (((2):866112.) 1997.
292
[15] M. Hessel, Modayil, H. van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. G. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement Jeaming CoRR, abs/1710.000 1710.02998 2017.
293
[16] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICL.),, 2015.
296
297
[17] R. Koenker, Ng, and S. Portnoy. Quantile smoothing splines. Biometrika, 81(4))673 -800 1994.
298
[18] H. Kushner and G. Yin. Stochastic approximation and recursive algorithms and applications, volume 35. Springer Science Business Media, 2003.
299
300
[19] Y. Liu and Y. Wu. Stepwise multiple quantile regression estimation using non-crossing constraints.
301
Statistics and its Interface, 2(3):299-311 310, 2009.
302
[20] M. C. Machado, M. G. Bellemare, E. Talvitie. J. Veness, M. J. Hausknecht, and M. Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-566 562, 2018.
304
305
[21] V. Mnih, K. Kavukcuoglu, D. Silver. A. Rusu, J. Veness, M. Bellemare, A. Graves, M. Riedmiller. A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 5 318(7540):5295555 533, 2015.
306
307
308
[22] A. Müller. Integral probability metries and their generating classes of functions. Advances in Applied Probability, pages 429 443, 1997.
10

---

310
[23] M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & S 2014.
312
[24]
J. Quan and G. Ostrovski. DQN Zoo: Reference implementations of DQN-based agents, 2020.
313
[25] M. Rowland, M. Bellemare, W. Dabney, R. Munos, and Y. W. Teh. An analysis of categorical distributional
314
reinforcement learning. In International Conference on Artificial Intelligence and Stutistics, pages 29-37. PMLR, 2018.
318
[26] T. Schaul. J. Quan, 1. Antonoglou, and D. Silver. Prioritized experience replay. arXiv preprint arXiv:151..5555. 2015.
318 [27] N. Tagasovska and D. Lapez-Paz. Single-model uncertainties for deep learning. In Advances in Neural Information Processing Systems, pages 1417-6428 2019.
320
[28] H. van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. CoRR, 2015.
322
[29]
C. J. Watkins and P. Dayan. Q-learning. Machine learning, 8(3-4)2279-29) 292. 1992.
323
[30] D. Yang, L. Zhao, Z. Lin, T. Qin, J. Bian, and T.-Y. Liu. Fully parameterized quantile function for
324
distributional reinforcement learning, Advances in Neural Information Processing Systems, 0166993-2002 2019.
326
[31] F. Zhou, J. Wang, and X. Feng, Non-crossing quantile regression for distributional reinforcement lcarning. Advances in Neural Information Processing Systems, 33: 11115999999999 2020.
Checklist
328
1. For all authors...
330
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
331
contributions and scope? [Yes]
332
(b) Did you describe the limitations of your work? [Yes] Experimental limitations regarding
the number of seeds were discussed in Section 6
(c) Did you discuss any potential negative societal impacts of your work? [No]
335
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
336
337
2. If you are including theoretical results...
338
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
339
(b) Did you include complete proofs of all theoretical results? [Yes] in the Appendix.
340
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] But it will be published if the paper is accepted
342
343
344
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
347
348
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
349
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
351
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] Data comes from standard simulation
353
framework of Atari games.
354
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] Data comes from a standard simulation framework of Atari games.
356
11

---

358
(c) Did you discuss whether the data you are using/curating contains personally identifi- able information offensive content? [No] Data comes from a standard simulation framework of Atari games.
359
361
5. If you used crowdsourcing or conducted research with human subjects...
382
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
385
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
367
12


Reviews:
[{"review_id": "vmqUjG7ntbE", "reviewer": "Reviewer_fTr8", "summary": "The authors provide theoretical analysis of distributional RL under Cramer distance. The authors further propose a neural architecture that yields monotonic quantiles and achieved improvements over prior works.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Low. Cramer distance in dsitributional RL has been studied in prior works, the authors provide some missing theoretical results. The proposed algorithm is slightly modified from NC-QR-DQN.\n\nQuality: Medium. The theories are correct, but the experiment results are insignificant.\n\nClarity: Medium. The paper is easy to read, except for some minor issues.\n 1. l.154 Formula...\n 2. In l.92, l_p is the p-Wasserstein distance, but later used as the Cramer distance without clarification.\n 3. Figure 3. is a little hard to interpret\n\nSignificance: Low.\n\nMajor concerns:\n 1. After reading the paper, I find that while correct, the theoretical results are not likely to be able to improve dsitributional RL algorithms.\n 2. From my opinion, NC-QR-DQN should be an important baseline of the work. But the authors didn't provide further comparison other than the three cases in Figure.5, therefore it is hard to tell the actual improvements of the proposed neural architecture.\n 3. Missing score table, incomplete experiments (2 seeds) in Figure 10.\n\n\nQuestions & suggestions:\n 1. If Cramer loss and Quantile Regression loss yields collinear gradients, why not use QR loss to save the trouble of computing Cramer distance? We only need the gradients to update network parameters.\n 2. According to my understanding of the paper, the proposed neural architecture, CNC, have three major differences compared with NC-QR-DQN:\n  1). For subnetwork 1, outputs median quantile value instead of minimum quantile value. Then, substract the cumsum of the first half per-quantile value differences instead of add.\n  2). Remove the scaling factor, thereby using ReLU instead of Softmax for per-quantile value difference.\n    I would suggest an ablation study on those two modifications to further investigate which one contributes more to the improvements.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss the limitations adquately in their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "gCdXH5-7tSw", "reviewer": "Reviewer_rysK", "summary": "This paper proposed to apply the Cramér distance to replace the 1-Wasserstein distance in the class distributional reinforcement learning. To justify the validity of this update, this paper proved two key properties: 1) Cramér projected distributional Bellman operator is a contraction and, 2) QR loss and Cramér gradients are collinear under non-crossing constraints. This paper also designed a low-complexity algorithm for computing the Cramér distance and a centered non-crossing architecture to implement the proposed algorithms.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of this paper is novel and not incremental. It proposed a new framework for modeling the distance between distribution in the distributional Bellman operator. Given that most of the recent algorithms for distributional RL are based on QR, this paper is a significant plus to this field. The major issue of this paper is that the author(s) generally can not convince me that the Cramér-based models can outperform the QR-based models. The proofs generally guarantee that CR-DQN can learn reasonable distributions of Q values because it resembles QR-DQN, but none of them can help readers understand where is the improvement. This paper lists some benefits like 1) CR enables removing a constraint (line 172-174) and 2) extreme quantiles are hard to estimate in the classic QR-DQN (line 186), but none of them is fully defined or evaluated. The motivation is vague from my point of view. The empirical results do not show a clear win of CNC-CR_QDN and the results are not explained.\n\nDetail comments:\n\n1) Line 185 \"As shown by the classical theory of extreme values\", could you explain more to help readers understand it? since it is a major motivation for your network architecture. By the way, figure 4 in [31](NC-QR_DQN) has claimed NC-QR_DQN converged faster than QR_DQN, which make me believe the reduction of training efficiency (from the \"symmetries\" (line 177)) in CR is significant and adding the CNC structure can not resolve this issue. It explains why CR-DQN is generally more sample inefficient than the QR-based methods. Your results in Figure 7 and the appendix generally confirm this problem.\n\n2) The Atari 57 experiment should include both NC-QR-DQN and NC-CR-DQN as baselines since the author(s) proposed replacing the original non-crossing network with the CNC architecture. Comparing both NC-QR-DQN and NC-CR-DQN can provide a form of ablation study that helps readers understand the influence of each component (CR and CNC).\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Societal impact: The authors believe their work has no negative societal impact, I generally agree that their negative impact is limited, but I think the goal of RL is to develop autonomous agents that can replace humans in many positions, for example, drivers. The author might want to discuss the influence of autonomous agents on the job market.\n\nLimitations: The discussion of limitations should be expanded, especially about why a clear win is not observed in the Atari benchmark.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "XRro221nWUc", "reviewer": "Reviewer_mDzE", "summary": "This paper draws an equivalence between 1-Wasserstein distance and Cramer distance on staircase distributions, use Cramer distance in the place of 1-Wasserstein (and quantile regression) to avoid order constraints among the particles $\\theta_i$ and consequently, proposes a non-crossing neural net for computing the Cramer distance in DRL.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strong points**: \n-\tThe connection between 1-Wasserstein and Cramer seems novel and interesting in DRL.  Especially, quantile regression in QR-DQN imposes order into the learnable particles $\\\\{ \\theta_i \\\\}_{i=1}^N$ as each $\\theta_i$ is an estimate of the $\\hat{\\tau}_i$-quantile which implies the monotonicity in the particles, i.e., $\\theta_1 \\leq \\theta_2 \\leq ... \\leq \\theta_N$. Cramer distance on the other hand is equivalent to 1-Wasserstein in terms of projection onto the space of a mixture of Heaviside step functions and in terms of gradients. Moreover, Cramer distance does not require order among $\\theta_i $\n\n**Weak points**: \n- Many parts of the writing are unclear and need more improvement (see below)\n- I think the present paper lacks a discussion and comparison with related work that is directly related to the problem and proposed solution. \n- The motivation and purpose of the experimental section are unclear and there is no conclusion or discussion drawn from the experiment. The experimental result does not show that the proposed algorithm CNC-CR-DQN outperforms QR-DQN with statistical significance in the whole range of Atari games (Fig 6). This raises the question of whether the proposed algorithm is effective and necessary. \n- The theoretical results are quite simple and the connection between Cramer and Wasserstein is already made in [2]. \n\n\n**Comments and Questions**:  \n-\tI think the paper lacks relevant literature. Most distributional RL methods are based on quantile regression that requires ordered statistics in neural representation, creating challenges such as maintaining ordered statistics, monotonicity, and “crossing quantiles”. I think a recent work [Nguyen-Tang et al AAAI’21, “Distributional Reinforcement Learning via Moment Matching”] can completely avoid such problems in quantile-regression-based methods using moment matching; thus it is directly related to the problem and the proposed approach in the present paper. What is the difference between your approach and Nguyen-Tang's in addressing the order statistics constraint in QR-based DRL? \n-\tThe paper addresses the crossing quantile problem in DRL but it only briefly touches on crossing quantile in one sentence at line 47-48 and at line 155. I think it is inadequate. More discussion is needed and probably one dedicated paragraph for this is necessary.\n-\tLine 130-131: What exactly permutation in the symmetric group of size N? What **symmetric group** are you referring to? This part needs to be clearer. \n-\tEq. (14): What is $\\theta’$? A typo? \n-\tThe experiment section lacks a clear explanation what is the purpose of the experiment? What conclusions can be drawn from the results? Currently, this section merely just describes the experimental procedure without insights drawn from it.\n-\tI think putting markers on Fig 6 would make it easier to read. Some colors are very similar. \n-\tFig 6:  CNC-CR-DQN outperforms QR-DQN in several games but in the entire Atari set, they perform very similarly (Fig. 6). How can you conclude that the proposed method that draws on Cramer distance without crossing quantiles works better in a statistical sense? \n- 1-Cramer distance ($ \\int |F(z) - G(z)| dz $) is identical to 1-Wasserstein distance and using 1-Crammer distance also does not require order among $\\theta_i$ as 2-Cramder distance. Why don't just use 1-Cramer distance? \n- What kind of games from the Atari set the proposed algorithm performs better than QR-DQN?\n\n**Initial Recommendation**\nWhile the idea of using Cramer distance as an alternative to the quantile regression loss is interesting, I do not find sufficient novelty and/or significant empirical performance in the current form, raising the question about the necessity and effectiveness of the proposed algorithm; moreover, the writing seems in rush and lacks related work. I vote for reject this time for the current form.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not adequately addressed the limitations of their work. ", "review_text": "", "extra_scores": {"ethical_concerns": "No concern. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "XDvZxRuoGat", "reviewer": "Reviewer_P97K", "summary": "This work proposes a Cramer loss for the training process of distributional RL, and design a centered non-crossing architecture for distributional RL.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\n1.\tClarity: The paper is well structured.\n2.\tTechnical novelty: A efficient algorithm is proposed to compute the Cramer distance.\n\nWeak points:\n1.\tI am concerned about the validity of the experiment results, since the performance of QR-DQN in Figure 5 is much worse than the results in their original paper. I am not sure if the authors correctly implement this method and the conclusion seems to be unconvincing. \n2.\tThe advantages of Cramer loss over quantile (Huber) loss are not well presented. As mentioned in your paper, both losses achieve unbiased gradient and equivalence to 1-Wasserstein in minimization problem. What are the pros of Cramer loss? Computation efficiency? Or else? Corresponding ablation studies are lacked.\n3.\tUnfortunately, there is a nearly same network architecture called NDQFN (Zhou et al., 2021) with proposed centered non-crossing architecture (CNC), except that CNC focus on median while NDQFN aims at \\tau = 0. However, NDQFN seems to achieve much better performance than CNC with Cramer loss.\n4.\tThe experimental results in the present form is not strong. From the Figure 6 and Figure 10 in Appendix, it seems CNC-CR-DQN perform comparable with QR-DQN, even slightly worse.\n\nHence, I vote for rejection for current form.\n\nMentioned reference:\nZhou, F.,Zhu, Z., Kuang, Q., Zhang, L*. Non-decreasing quantile function network withefficient exploration for distributional reinforcementlearning. International Joint Conference onArtificial Intelligence (IJCAI), 2021. In press.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: As the authors raised concerns with the reviews, I have thoroughly read the paper and even discussed the situation with the senior area chair. While the reviews are all against acceptance, my individual assessment is much more positive. Not to burry the lead, but, after discussions I will be suggesting that the paper not be accepted. 

I very strongly encourage the authors to resubmit. Rather than viewing this work as a competitor with NC-QR-DQN, I see this paper as providing some very interesting theoretical connections between all QR-based methods and the Cramer distance. The reviewers are, perhaps rightly, less than confident in the impact of this work due to the empirical results in the paper. This is likely amplified by the discrepancy between the published NC-QR-DQN results (with exploration bonus) and those in this paper (with epsilon-greedy, and also using the Cramer loss?). 

However, the other point reviewers have raised, which I agree is less easily dismissed, is that if these losses are essentially equivalent, then how can this have real impact beyond what we already can be done with pure QR? However, most QR-DQN-like methods do actually use the Huber quantiles, while this method is equivalent to a true (non-crossing) QR. Perhaps there is a unique advantage to the non-crossing quantiles in the pure QR case. This seems like something that could be checked by keeping as much the same as in this paper but using the QR loss (without Huber). I think that comparing with the actual NC-QR-DQN algorithm (with the Huber quantiles and architecture, but without the exploration bonus) would be an important addition. I would also suggest improving the clarity of Figures 1 & 2 (right panel), but view this as a fairly minor issue.

Although perhaps worth discussing, I do not think that NDQFN is a necessarily empirical comparison. Obviously reviewers may disagree, and thus to buffer your chances of acceptance it may be worth attempting nonetheless. I would have very much liked to accept this paper, but at the same time I do not think it is quite strong enough to justify completely disregarding all four reviewers feedback. However, please know that your concerns with the reviews have been noted.
Decision: Reject