The Multiquadric Kernel for Moment-Matching Distributional Reinforcement Learning
Authors: Ludvig Killingberg
Helge Langseth
Venue: Accepted by TMLR ()

Abstract: Distributional reinforcement learning has gained significant attention in recent years due to its ability to handle uncertainty and variability in the returns an agent will receive for each action it takes. A key challenge in distributional reinforcement learning is finding a measure of the difference between two distributions that is well-suited for use with the distributional Bellman operator, a function that takes in a value distribution and produces a modified distribution based on the agent's current state and action. In this paper, we address this challenge by introducing the multiquadric kernel to moment-matching distributional reinforcement learning. We show that this kernel is both theoretically sound and empirically effective. Our contribution is mainly of a theoretical nature, presenting the first formally sound kernel for moment-matching distributional reinforcement learning with good practical performance. We also provide insights into why the RBF kernel has been shown to provide good practical results despite its theoretical problems. Finally, we evaluate the performance of our kernel on a number of standard benchmarks, obtaining results comparable to the state-of-the-art.

Published in Transactions on Machine Learning Research (08/2023)
The Multiquadric Kernel for Moment-Matching Distribu- tional Reinforcement Learning
Ludvig Killingberg
ludvig Battiigkillingbber@mmmmm nmm.
Department of Computer Science
Norwegian University of Science and Technology
Helge Langseth
tett ninu.no
Department of Computer Science
Norwegian University of Science and Technology CIFAR Fellow
Reviewed on OpenReview:
wew http://///eeeeeeeeee///////////r 9ea
Abstract
Distributional reinforcement learning has gained significant attention in recent years due to its abil- ity to handle uncertainty and variability in the returns an agent will receive for each action it takes. A key challenge in distributional reinforcement learning is finding a measure of the difference be- tween two distributions that is well-suited for use with the distributional Bellman operator, a function that takes in a value distribution and produces a modified distribution based on the agent's current state and action. In this paper, we address this challenge by introducing the multiquadric kernel moment-matching distributional reinforcement learning. We show that this kernel is both theoreti- cally sound and empirically effective. Our contribution is mainly of theoretical nature, presenting the first formally sound kernel for moment- matehing distributional reinforcement learning with good practical performance. We also provide insights into why the RBF kernel has been shown to provide good practical results despite its theoretical problems. Finally, we evaluate the performance of our kernel on a number standard benchmarks, obtaining results comparable to the state-of-the-arr.
Introduction
Reinforcement learning is a type of machine learning that involves training agents to take actions in an environment in order to maximize a reward signal. In traditional reinforcement learning, the agent learns value function that estimates the expected future cumulative discounted reward (value) for each state-action pair. The value function tells the agent how "good" particular state-action pair is, and the agent uses this information make decisions about which actions to take in order maximize its value. However, this approach can be problematic in some cases, such when the environment is highly stochastic when the reward signal is noisy.
Distributional reinforcement learning is a variation of reinforcement learning that addresses these challenges by learn- ing distribution of future cumulative discounted rewards (value distribution), rather than single expected value, for each state- latteacction pair. This distributional approach allows the agent to consider the uncertainty or variability in the rewards may receive, and can lead more robust and accurate decision-making
Algorithms for distributional reinforcement learning can be separated along two axis: the representation of the value distribution for given ftateacction pair and the measurement of the difference between distributions. Some algorithms utilize discrete set of samples represent the distribution (Bellemare et al. 2017 Dabney et al. [017]] Nguyen-Tang al. 2021). while others employ implicit distributions parameterized by a neural network (Dabney et al. 2018 Yang et al. 2019 Additionally, the agent must have the ability to measure the difference between the predicted value dis- tribution for a given state-action pair and the actual value distribution received, in order to update its predictions. This difference measure crucial for the agent to learn and adapt. The Kullbach-Leibler divergence is widespread in the probabilistic machine learning field but possesses some undesirable properties for use in distributional reinforcement learning. Instead, measures such the Wasserstein metric, the Cramér distance, and the maximum mean discrepancy

---

Published in Transactions on Machine Learning Research (08/2023)
have shown better results (Bellemare et al. 2017 Dabney et al. 2017 Nguyen- Tang al. 2021 Dabney et al. 2018 Yang al., 2019).
The distributional Bellman operator is key component of distributional reinforcement learning algorithms. It is used to update the value distribution for a given state-action pair based on the observed rewards and transitions the environment. The distributional Bellman operator contraction under certain conditions, which means that it guarantees that the distance between the updated and original value distribution will always be non-increasing, converging to the true value distribution in the limit (with convergence properties similar to those of the usual Bellman operator used, e.g., in Q-learning).
Our main contribution in this paper is theoretical analysis and extension of the work by Nguyen- Tang et al. 2021 on on moment matching for distributional RL. Additionally, our results may also be of theoretical interest for under- standing the properties of the distributional Bellman operator.
2 Background
This section introduces the necessary background in both classical reinforcement learning, and distributional rein- forcement learning, and describe some current state-of-the-art distributional reinforcement learning methods.
2.1 Reinforcement Learning
We consider the standard reinforcement learning (RL) setting, where interaction between agent and its environment is modeled Markov Decision Process (S, A, R, 7) (Puterman 1994). where S C R and A denote the state and action space respectively. The stochastic reward function R S x A R is a function that maps the state and action of an agent to a reward value with some element of randomness or uncertainty. P(. ar, a) denotes the transition probabilities starting in state æ S and doing a E A, and y E [0,1) is the discount factor. A policy (-|1z) maps states to distributions over actions.
The Bellman equation is often used the foundation for algorithms that solve MDPs, such Q-learning. In general, the equation describes necessary condition for optimal solution in dynamic programming optimization. In the context of reinforcement learning. it is often used describe the optimal state-action value function Q", According to the Bellman equation, described by ( the accumulated discounted reward, or value, of a state-action pair is equal to the sum the immediate reward from the current action and the discounted expected reward from future actions:
= [R(2 + max Q" (a',aa)) a'EA
A common approach to solving the Bellman equation uses the Bellman operator. The Bellman operator, 7", maps state-action values to their next state-action values. The Bellman operator for policy क is defined as follows:
(=Q( [R(x, a)] + "B 1Q(x', a=))
Alternatively, we can define the Bellman operator for an optimal policy, also known as the Bellman optimality operator as
max Q( at)
T*Q(x, a) == [R(x, a]] Fear
(3)
a'EA
The significance of the Bellman operators is that they are contraction mappings wrt the supremum norm. In other terms, for some & (0, 1), we have
| -TQI-T"Q2lll < 1|Q - Q21loe
(4)
According the Banach fixed-point theorem
Banach (922).. that means that have the following properties:
2

---

Published in Transactions on Machine Learning Research (08/2023)
lim (T=) = 11000
(5)
lim (T*)*Q= k-+00
Q
(6)
Deep Q-Network (DQN) (Mnih et al. 2013) approximates Q using neural network, which we will denote Qo. Mnih et al. (013) proceed to approximate Q" using gradient descent to minimize the distance
L = d (7*Q0(0;, ar), Qo(xr, az))
More specifically, they let d be the lg distance, in which case this also called the squared temporal difference.
2.2 Distributional Reinforcement Learning
Distributional reinforcement learning focuses on learning the distribution of cumulative discounted rewards, rather than just the expected value. In distributional reinforcement learning, the algorithm typically uses a function to ap- proximate the distribution over returns for each state-action pair. This function is updated over time based on the rewards received and the predicted distribution rewards for each state-action pair. By doing this, the algorithm can learn to anticipate the range of possible cumulative discounted rewards and make decisions that maximize the expected value.
For an agent following policy Z", or value distribution (Bellemare et al. 2017). is the sum of discounted rewards along the agent's 's trajectory:
2 (x a)
Σ yt a(w,,))),
starting in state Zo summarized as finding the optimal policy, that is the sequence of actions that an agent should take, in order to maximize the reward signal Q* (æ, a) := E., 12 (x, a)] of a given state-action pair. In other words, we look for the strategy that the agent should follow to achieve the best possible outcome expectation.
with do de ((112)). and
((11x;11,0-1)) The objective of RL can be
The distributional Bellman operator mathematical operator used in distributional reinforcement learning to esti- mate the probability distribution of future rewards for each state-action pair in an environment. This operator is based the Bellman equation. The distributional Bellman operator extends the Bellman equation to estimate the probability distribution of future rewards, rather than just the expected value. This allows the operator to capture the uncertainty and variability of the rewards in the environment, and to make predictions about the range of possible outcomes. Given a random variable z denoting the value distribution, the operator is defined as
T a) R(x (x',
(9)
where A B denotes equality in distribution, ie. random variable and B have the same distribution, and a' n, ad P(x' ax, a').
The distributional version of Equation is
(10) but in this case, because we are operating on distributions and not values, we can no longer use the l metric. There are many distance measures for distributions to choose between, all of them are suitable for this use case. It is
L(0) - d Zo( ax), Za(a,, az))
important for distributional Q-learnig that the distributional Bellman operator is contraction in d. What this means is that for some (0, 1), we have that

---

Published in Transactions on Machine Learning Research (08/2023)
d d (TT < kd (d1, 72)
(11)
We can then use d, or some estimator of d, as loss function, and our algorithm will converge.
Bellemare al. (2017) show that the distributional Bellman operator for fixed policy (7*) is contraction in the p- p-Wasserstein metric and Cramér distance, but not in KI--divergence, total variation, or Kolmogorov distance. Nguyen- et al. (2021) further show that 7 is a contraction in maximum mean discrepancy for carefully selected kernels.
2.2.1 Learning Categorical Distributions
One popular approach to distributional reinforcement learning is to approximate the distribution of values, Z, by discretized or categorical variable. C51 was introduced by |Bellemare et al. (2017) as an extension of DQN. Like DQN, C51 uses neural network to approximate the action-value function, but instead of outputting a single value for each action, it outputs a distribution over the possible returns. This distribution is represented using a set of discrete "atoms", which allows the agent to represent a wide range of possible returns. 51 discrete atoms per action were used in their Atari 2600 experiments, hence the name C51.
In C51, the distance between two distributions is calculated using the Kullback-Leible (KL) divergence. One issue with using the KL divergence as measure of the difference between two distributions in distributional reinforcement learning is that the distributional Bellman operator is not a contraction in KL divergence (Bellemare et al., 2017). Because the Bellman operator is not a contraction in KL-divergence, there is no fixed point, hence it cannot be used for TD learning.
For this reason, KL divergence is generally not considered good measure of the difference between two distribu- tions in distributional reinforcement learning. Instead, other methods typically use measures such the Wasserstein distance the Cramér distance, which do satisfy the contraction property Bellemare et al. (2017).
2.2.2 Quantile Regression
Quantile Regression DQN (QR-DQN) is variant of DQN introduced by Dabney et al. (017)). tribution, QR-DQN uses quantile regression Q-learning, which is based on quantile regression, a statistical technique for estimating the conditional quantile function of random variable.
).
To learn the value dis-
One advantage of QR-DQN is that it uses the Wasserstein distance, also known the Earth Mover's distance, as measure of the difference between two distributions. The Wasserstein distance is well-known metric in the field of probability theory and statistics, and has several desirable properties that make it good choice for distributional reinforcement learning:
The Wasserstein distance is a true metric, meaning that it satisfies the triangle inequality and symmetric. This makes a more consistent and reliable measure of the difference between two distributions than the Kullback-Leibler (KL) divergence, which does not satisfy these properties.
The Wasserstein distance a smooth, continuous function, which makes well-suited for use with gradient- gradient-based optimization leam the value function.
based optimization methods. This is important in reinforcement learning, where algorithms often rely on
The distributional Bellman operator a contraction in Wasserstein distance.
2.2.3 Implicit Quantile Regression
Implicit quantile regression DQN (IQR-DQN) (Dabney et al. 2018) is variant of the QR-DQN (Dabney et al. (017) algorithm. Unlike QR-DQN, which uses an explicit quantile regression model to estimate the distribution of returns, IQR-DQN uses an implicit quantile model, which is a neural network that directly maps states and actions to quantiles of the return distribution.
Work on IQR-DQN (Danney et al. 2018 Yang et al. 2019) has focused on developing efficient and effective algo- rithms for learning the implicit quantile model, as well demonstrating the performance of IQR-DQN variety

---

Published in Transactions on Machine Learning Research (08/2023)
of reinforcement learning tasks. Results have shown that IQR-DQN can learn more accurate value distributions than other distributional reinforcement learning methods and that it can achieve better performance.
2.2.4 Moment-matching
Nguyen-Tany et al. (202T) proposed to minimize the maximum mean discrepancy (MMD), rather than the Wasserstein distance. The squared maximum mean discrepancy between two distributions p and q relative to kernel & defined as
MMD k)) E [A(X, + (Y,YYY))) 2E ((X,)))
(12)
where X, are independent random variables with distribution and Y. Y" are independent random variables with distribution Given two sets of samples { ~ and {y: ( ~ q. Gretton et al. (2012) introduced the following unbiased estimator of MMD
MMD ((:)). = m(m 1)
Σ k( 23) + k(y Us) 2M( yz)-
Nguyen- Tang et al. (2021) use biased estimator MMD, of MMD, but justify the choice by noting that the estimator in practice shows lower variance than its unbiased counterpart. This biased estimator, also introduced by Gretton et al. (201)), is defined as
2 MMD ((aa), (b:);))
m Σ k(a,, my)) + k(y;, Uj) 2k y3)-
Kernels One set of interesting kernels are those on the form k( y) - y112. Nguyen-Tare Tang et al. 2002 call this class of kernels unrectified kernels. MMD with the unrectified kernel and = 2 is distributions equal in expectation (Székely & Rizzo 2013). This means that will not the convergence in distribution, but rather in expectation, essentially reducing it to classical Q-learning. Nguyen- Teuyen-Tang et al. (202T) look at the practical performance of this kernel, and finds it performs unfavorably compared to some other kernels.
long as the two
Szćkely& & Rizzo 2013) showed that with kernels the form (x y) = =112 for 3 E (0, 2), then MMD 1/; k) is zero if and only if ju and are equal in distribution. Nguyen- Dguyen-TTang al. (202T) further show that the distributional Bellman operator is contraction in these kernels. Practical performance was evaluated for 3 1, but the results were not promising.
The RBF kernel is a common choice in Gaussian processes and other applications. It is defined as
ky (z. y) = exp (-h² | y
(13)
where is free parameter. Nguyen- Tang et al. (0021) showed by counterexample that the distributional Bellman operator not a contraction under MMD with the RBF kernel, but also found that the RBF kernel performs favorably compared the unrectified kernels when using a mixture of RBF kernels with different length-scale values, defined
k(x y) = Σ exp (-1-11| -
(14)
In this paper, we will investigate the performance of the multiquadric kernel (MQ) (Hady 1971) for moment-matching distributional reinforcement learning. Given a free parameter h, the multiquadric kernel defined as
kn (x, 3) - V 121l 211
(15)
5

---

Published in Transactions on Machine Learning Research (08/2023)
Radial Kernel
(5(1)
3
Multiquadrie
-V++122 1:112
RBF
exp(-11(2)
Unrectified Unrectified
-16
Table I: Property 1 for various radial kernels.
Note that the kernel often used as shorthand for positive-definite kernels. We will make this restriction and instead, refer kernel any symmetric real-valued function of two variables. The MQ, unlike the RBF, is not positive definite but is in class called conditionally positive definite. In Section 3 we look at the properties of this kernel.
Pseudo-samples MMDQN (Nguyen-Tang et al. 2021 use the QR-DQN network structure to approximate Z. They define the N outputs per action to be pseudo-samples from Ze. and then minimize MMD between the pseudo- samples from Za and Tz.. This means that true samples from the value distribution are never drawn, and that the approximation is completely deterministic. There are no underlying restrictions with the algorithm that would suggest that a sampling- saaseing-bbaedd approach would not also work.
Open question Nguyen-Tann et al. 22021) pose question on what the sufficient conditions on the kernels are for the Bellman operator to contraction in MMD. They further question whether scale sensitivity is necessary condition. In Section we show by counterexample that scale sensitivity is not a necessary condition for contraction in MMD.
3 Theoretical Work
In this paper, we propose to use the multiquadric kernel for moment-matching distributional reinforcement learning. Figure F provides an illustrative comparison between the multiquadric kernel and the kernels discussed in section 2.2.4 Our choice of the multiquadric kemel, ky, is motivated by several desirable properties it possesses:
1. (MD(;;;A))) kn)) is metric on probability distributions,
The Bellman operator a contraction in MMD with kn.
3. MMDD(.;:k)) is smooth,
(x, y 12 Cp < oo for some constant C₁ dependent on h. Um
sup=ER"
Table shows property 1 4 for a range of radial kernels, i.c. kernels on the form k(æ, y) +0|x 8112). We observe that only the multiquadric kernel possesses all the aforementioned properties. As far as know, the MQ kernel is the only kernel that satisfies all properties 1 4 and is the first non-scale-sensitivv kernel with the contraction property. Proofs for property and 2 will follow. Proofs for property 3 and 4 are available in the appendix as Lemma A. and Lemma A.2 respectively.
First, we show that under the multiquadric kernel, maximum mean discrepancy (MMD) is a metric in the family of valid distributions. In other words, satisfies the triangle inequality, is symmetrie, non-negative, and is zero and only if the distributions are equal. This is important in the context of distributional reinforcement learning, as it ensures that the distance measure used to compare the predicted and true distributions of returns is consistent and reliable.
Second, we demonstrate that the distributional Bellman operator is a contraction in MMD with the multiquadric kernel. This property is important in reinforcement learning, as it ensures that the agent's 's estimate of the distribution of returns will necessarily get closer to the true distribution as takes more actions and gathers more experiences. This can help the agent to more accurately learn the distribution of returns and make good decisions based on that distribution.
Furthermore, the multiquadric kernel has smoothness properties, resulting smooth maximum mean discrepancy when using the multiquadric kernel. In many practical scenarios, it is commonly observed that smoothness property

---

Published in Transactions on Machine Learning Research (08/2023)
k( k(s.y)
XQQ
xx yl:
Figure 1: The multiquadric (MQ) kernel compared to the kernels discussed in section 2.2.4 as function of the Fuclidean distance between two points. h = was chosen for both MQ and RBF
in the loss function tends lead to gradients that exhibit greater degree of stability and better performance (Gokcesu & Gokcesu 202T Patterson al. 2023). In the context of reinforcement learning, this can ultimately lead faster convergence and better performance of the model. This is one of the motivations behind the use of the l metric and the Huber loss over & metric deep learning.
Finally, the magnitude of the gradient of the kernel is upper bounded by constant. If the magnitude of the gradient is not upper-bounded, the learning algorithm may oscillate or diverge, rather than converge to a stable and accurate so- lution. By upper bounding the magnitude of the gradient, it is possible to ensure that the learning algorithm converges to a stable and accurate solution in reasonable amount of time. This is one of the reasons the Huber loss has become so popular reinforcement learning, and why we often see gradient clipping being performed when the & metric is used as loss function.
Our proof that the distributional Bellman operator is a contraction in MMD with the multiquadric kernel requires few definitions and existing lemmas. We begin by defining completely monotonic functions and conditionally (strictly) positive definite kernels.
Definition 3.1. f(x) is a completely monotonic function (Schilling et al. 2012 Definition 1.3) if it is infinitely differentiable and
(()) for and 0, 2......
Definition 3.2. A vellevalued kernel & is called conditionally positive definite if it is symmetric, i.e. k(x,y)
=
k(y, æ),, and satisfies
m
Σ cackla,, 2
(16)
for any R with
m
Σ c =
It is called conditionally strictly positive definite if the greater-than or equal sign in Equation can can be replaced by a greater-than sign.
The following three lemmas provide a basis for our main results in Theorem 3.1 and Theorem 3.2
Lemma 3.1 (Szćkely & Rizzo 2013 Proposition 3). MMD(: is a metric on P(X)) and only if is a condi-
tionally strictly positive definite kernel.
Lemma 3.2 Micchelli 1986 Theorem 2.1). Let gt)) == f(vi). Then f is conditionally positive definite if and only
if gg' (t) is completely monotonic.
Corollary 8.20). f is conditionally strictly positive definite if and only if it is condi- tionally positive definite and not polynomial of order 2 or less.
Lemma 3.3 (Wendland 20044

---

Published in Transactions on Machine Learning Research (08/2023)
Theorem 3.1 formally states property of the multiquadric kernel. Theorem 3.1. MMD(s-; is metric on P(2') for the multiquadric kernel (x, y) - y
Proof. By utilizing Lemma 3.1 we only need to show that the multiquadric kernel is conditionally strictly positive definite.
We rewrite (a, y) as f(h2|aæ 2) We can directly show that for g f(Vi) -vI+t with 0, then (t) - 2014t is completely monotonic. By induction, we see that its n-th derivative
d dt 2 2++
(T 1)-n 21' -
has an oscillating sign on (0, oc), due to the sign of the gamma function for strictly negative values. By Lemma 3.2 we conclude that (() is conditionally positive definite. By Lemma 3.3 (() is conditionally strictly positive definite. 7
Lemma 3.4 is the basis for our proof that the distributional Bellman operator is a contraction in MMD with the multiquadric kernel (Theorem 3.2 Lemma 3.4. [[I; a, == va Vb + 12 is positive definite for b.
To establish the positive definiteness of [(1), we begin by taking the n-th derivative of f(v[) and demonstrating that it is completely monotonic. Utilizing Lemma 3.4 we can then conclude that f(t) is positive definite. comprehensive proof can be found in the appendix.
Theorem 3.2. The distributional Bellman operator is contraction in MMD with the multiquadric kernel.
The detailed proof of Theorem 3.2 is given in the appendix. The main step is to show that it is sufficient to prove that the kernel,
V 122 1
k(x, y) =
+ 1/21
722
is conditionally positive definite. result of Lemma the kernel is positive definite; hence, is also conditionally positive definite.
4 Limitations of the Radial Basis Function for MMD
In this section, our aim is to investigate the sensitivity of MMDQN Nguyen-Tang et al. 2021) performance on Atari problems to RBF kernel parameters. While MMDQN has achieved remarkable performance across all 57 games, questions remain regarding how sensitive this performance is to changes of the kernel. The use of single mixture kemel for all 57 games, may give the impression that the method is relatively insensitive to the kernel, however, we will show that this is not the case. We will also identify settings where the optimal RBF parameters are more difficult to find. It worth noting that the authors do not claim that these parameters will generalize to different environments: quite the contrary, they state that the kernel's role performance crucial. They also state that identifying kernel that performs well in all games is challenging. In this context, explore the impact of the RBF bandwidth parameter MMDQN performance in simple toy settings.
It is interesting to see whether there exist settings in which the RBF kemel does possess the contraction property. Using a similar proof to that of Theorem 3.2 we can show that for two distributions v, y with bounded support, the distributional Bellman operator a contraction in MMD with the RBF kernel under certain bounds for h. Corollary 3.1 defines the bounds. The proof is given in the appendix.
Corollary 4.1. Let k( 8) exp(-h?|| y112). Then the distributional Bellman operator is contraction in MMD v3 k) for distributions μr, v with support Sp,, S, if for some E (0, 2)
log(>?~a)
sup
||
2112
# S
(72
1)6²
8

---

Published in Transactions on Machine Learning Research (08/2023)
The implication of the corollary is that when the approximation of the value distribution has bounded support, RBF's h t-parameter can be chosen such that convergence is guaranteed. Conversely, if the support is unbounded, there is no guarantee of convergence. With finite rewards and E [o, 1) the true value distribution will always have a bounded support. This means that a model with bounded support does not necessarily limit the model's 's ability to approximate the true value distribution. Choosing small enough to satisfy this condition, however, makes the kernel very flat near 0, hence reducing the kernel's power to differentiate between samples close together.
We noticed that for all the Atari games that MMDQN (Nguyen-Tanm al. 2021) released videos of the Z-distribution was uni-modal. Multi-modal modeling of the Z-distribution is important when making risk-adjusted actions, where it may be worth choosing an action with slightly lower expected return if it means not risking a really low re- turn. Although our experiments do not take risk into account when performing actions, we hypothesize a uni-modal approximation could still be a limiting factor for performance in the Atari benchmarks. Therefore, we designed a procedure to evaluate MMD's ability to fit multi-modal distributions with the RBF kernel. The procedure is described in Algorithm
Algorithm I Procedure for evaluating the ability of MMD with kernel k to approximate a distribution.
Require: k, P, (x)
AD-statistics & Array([ : :7]
for 1: do
M1 '~P
= fori=l,...,
(AuhIN t- MMD, (fa)a,, ku)
{xJIN + Optimiser((==1na (AN)
AD-statistics(f] + Aateeroon-Dalinge P) end for return AD-statisties
The procedure starts with some pscudo-samples {=}1:: and a target distribution P. Over T gradient updates, the distribution represented by the pseudo samples should move closer to P. The Anderson-Darling test (Anderson & Darling 1952) tests whether given sample comes from a distribution. We use this test to evaluate the effectiveness of the kernels fit the distribution. The test scores samples scale of 0 to 1, where low score indicates that it is unlikely that the samples come from the target distribution.
For the Anderson- -arrlig test experiment, we chose a Gaussian mixture model with two components, W(-2,0.2) and (2, with weights 0.2 and 0.8 respectively. For comparison, we tested the multiquadric kernel (MQ) with h = 1, h = 10, and h = 100. For the RBF we used the same kernel that Nguyen- Tang et al. (202I) used in their Atari experiments, viz. Equation 14 Search for more effective RBF yielded no positive results. Even though such kernel may exist, our results show that it is not trivial to find such kernel, and suggest that performance result. The results of the experiment is shown in Figure A learning rate of 0.1 was used for MQ and a learning rate of 0.01 was used for RBF. MQ was scaled by h to avoid the magnitude of the gradients growing significantly with h.
suffer
5 Experiments
To evaluate the performance of the multiquadric kernel, we conducted experiments on 8 Atari games from the Arcade Learning Environment (ALE). Our goal was to see whether our algorithm was able to achieve results similar to those obtained by MMDQN with the RBF kernel, a state-of-the-art distributional reinforcement learning algorithm, while also demonstrating the benefits of using kernel with theoretical guarantees.
To show that we are not handpicking games that the multiquadric kernel performs better on, we use the 6 Atari games Nguyen-Tang et al. (2021) used for tuning (Breakout, Assault, Asterix, MsPacman, Qbert, and BeamRider). We have selected two additional games, one where MMDQN performed significantly better than QR-DQN (Tutankham), and one where performed significantly worse (Spacelnvaders). We are interested in whether the multiquadric kernel, with its contraction property, will outperform the RBF kemel environments where the RBF kernel struggles. By inspecting the original results presented in the MMDQN paper Nguyen-Tang Tang al. (202)), we see that most of the
9

---

Published in Transactions on Machine Learning Research (08/2023)
Target Distribution
Anderson-Darling Test
1.00
0.75
AD statistic
RBF mixture
p(x)
1.0
MQ, h=
0.50
MQ, h- h-10
MQ. h=100
0.25
MmmN
0.0
0.00
3
1000
X
Gradient updates
Figure Figure showing target distribution and p-value for the Anderson-Daite test with the null hypothesis that the samples drawn from P against the alternative hypothesis that the samples not drawn from P.
environments which MMDQN significantly outperforms QR-DQN are part of the 6 games used for tuning. This suggests that appropriate kernel parameters are important for performance.
To ensure that do not involuntarily skew the experiments in favor of the multiquadric kemnel, used the orig- inal implementation of MMDQN by Nguyen- Tang al. (2221) and all except the kernel parameters remained the same. Equation 14 defines the parameters used for the RBF mixture kernel. We investigate two parameter val- ues for the multiquadric kernel, / = and h = 10. The results in Figure 3 show similar performance between the multiquadric and RBF kernel, despite the hyperparameter optimization that has been done for the RBF kernel parameters over these games. This could explain the slightly worse performance Asterix and Assault. Inter- estingly, the RBF kernel performed significantly worse on Spacelnvaders. Although no definitive conclusions can be made from this, it does support argument that the performance is more sensitive RBF kernel parameters. Raw result data for MQ is available at http:://gibbbbcoom/mduv//MMMMMM
The numbers for RBF and QR- DQN are taken from Nguyen- Tang al. (2021), and are available along with the implementation of MMDQN at hhapggghuuubooooooanaannnggggeee mmdre
For more evidence that MQ performs on par with RBF, we have pre- sented human-norme normalizee agregates in Figure 5 As suggested by Agarwal et al. (202T), we have used the more robust interquartile mean (IQM)
RBF(h-1]
aggregate, rather than the mean median. Al- though these results suggest that MQ performs significantly better, because we are only evaluating the methods on 8 environments, the results for RBF are heavily punished by its poor performance on SpaceInvaders, despite the use of the more robust IQM statistic. In combination with results presented Figure S we believe that these results show that MQ performs at least on par with RBF.
RBF(h=5]
RBF(h-10)
RBF(mix]
MQth=Si
MQ/h-10)
MQXmixt
To evaluate the robustness of the MQ kernel compared to the RBF kernel for its application in MMDQN, we focused Qbert, one of the Atari games from Figure 3 where mixture of RBF and MQ showed approximately equal performance. Figure presents the highest normalzed scores achieved with each kernel. The results unanbiguously demonstrate that non-mixture RBF lags be- hind MQ. providing solid support for the assertion that RBF's per- formance is significantly more sensitive to hyperparameter tuning.
1.00
1.25
1.50
1.75
2.00
Human Normalized Scores
Figure 4:
Comparison of highest human- normalized scores for MMDQN on Qbert using different kernels (3 Seeds).
10

---

Published in Transactions on Machine Learning Research (08/2023)
Average Scote
McPacmmnn
Qbert
Spacelusacioss
Totsunllhiae
Averuge Soore
0.5
200
Namber Prnmnos Millions)
Number of Pismes (in Millicon)
Number Fromees Millions)
Numsbee
Frames (in Millicass)
Figure 3: Training curves for QR-DQN and MMDQN with the MQ and RBF on 8 Atari 2600 games. Curves for MMDQN are averaged over 3 seeds and smoothed over a sliding window of iterations. QR-DQN is averaged over seeds. 95% confidence intervals are shown. Reference values for QR-DQN and RBF are from Nguyen- Tanyen-Tanggaa et al.) (2021).
Method MQQa-11 QR-DQg
IQM Human Cortallieed Seares
R83
MQ
MQ(==10)
QR-DQN
10.0
12.5
15.0
Number Franos (in Millions)
IQM Hmman Normalized Scores
Figure 5: Left IQM human-normalize scores function of number frames measured. To make the graph more
readable, a running mean of size 10 for each seed was used. Shaded regions show pointwise 95% percentile stratified bootstrap Cls. Right IQM human-normaliize scores with 95% percentile stratified bootstrap CIs of best evaluation for each seed. Reference values for QR-DQN and RBF are from Nguyen- Miguye-TTang et al. (221).
11

---

Published in Transactions on Machine Learning Research (08/2023)
6 Discussion and Conclusions
We have explored the use of the multiquadric kernel for distributional reinforcement learning in this paper. We have shown that the multiquadric kernel has several key theoretical properties that have been studied in this context:
1. The distributional Bellman operator is contraction in MMD with the multiquadric kernel.
MMD is a metric under the multiquadric kernel, which means that the MMD between two random variables only be zero when they equal in distribution.
There are several reasons why a theoretically sound kernel might be preferred over a kernel that performs similarly but lacks theoretical guarantees:
The contraction property provides rigorous foundation for the algorithm and can give us confidence in the performance and behavior of the algorithm under different conditions. This is especially important in reinforcement learning, where the agent is learning to interact with and make decisions complex and potentially unknown environment.
The properties discussed in fection help to ensure that the algorithm is well behaved and robust and can help to avoid pathological or degenerate cases that might arise with kernels that lack the metric or contraction property.
The discussed properties, particularly the metric and contraction properties, can facilitate the development of new techniques and variations on the algorithm by providing a clear set of assumptions and guarantees that can be built upon or modified.
MMD with MQ kernel's improved ability to model bimodal distribution can be critical for risk-sensitive policies where low returns are particularly detrimental, such as in investment management.
Overall, while it is certainly important to consider the practical performance of a kernel, a theoretically sound kernel can offer a number of additional benefits that can make it compelling choice for moment-matchinn distributional reinforcement learning. Additionally, our experimental results show that the multiquadric kernel is stable with regard to parameter changes and that a mixture kernel is not necessary to achieve good results.
Furthermore, through Corollary 4.1 we have provided insight into the practical performance of the RBF kernel. The relationship between convergence, the parameter h, and the support of value distributions can provide insight into tuning h for each task. Alternatively, it could provide the basis for an algorithm that learns and changes h during training. Such an algorithm was proposed as future work by Nguyen- Taayen-Tang et al. (202T). It is possible that the RBF kernel may encounter difficulties in such a scenario due to the absence of a convergence guarantee and that the multiquadric kernel may be a more suitable alternative.
Finally, our results demonstrate the potential of the multiquadric kernel for distributional reinforcement learning. Its contraction and metric property make it promising choice for wide range of reinforcement learning tasks. Further research is needed fully explore the capabilities of the multiquadric kernel and to compare its performance other kernels in more diverse and challenging environments.
References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34, 2021.
Dmitri Alekseevsky, Andreas Kriegl, Mark Losik, and Peter W. Michor. Choosing roots of polynomials smoothly, 1998.
T. W. Anderson and D. A. Darling. Asymptotic Theory of Certain "Goodness of Fit" Criteria Based on Stochastic Processes. The Annals of Mathematical Statisties, 23(2):193 212, 1952. doi: 10. 177729437. URL
https://dof org 121 4/wmm//////////////////////l/ /
12

---

Published in Transactions on Machine Learning Research (08/2023)
Stefan Banach. Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales. Funda- menta Mathematicae, 3(1): 113-11,,,,,,, 1922. URL eeddmldmll... /rtp///uooooo/////////3/2
Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 449-458. PMLR, 06-11 Aug 2017.
URL https://proceeddin
press/
270/bellemazel7a..
Will Dabney, Mark Rowland, Marc Bellemare, and Remi Munos. Distributional reinforcement learning with quantile regression. Proceedings of the AAAI Conference on Artificial Intelligence, 32, 10 2017. doi: 10.1609/anai v32il. 11791.
Will Dabney, Georg Ostrovski, David Silver, and Remi Munos. Implicit quantile networks for distributional reinforce- ment learning. Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1096-1105, PMLR, 10-15 Jul 2018.
mmr. press/ v///// 178C/dabbeyll 18m.mmll
Kaan Gokcesu and Hakan Gokcesu, Generalized huber loss for robust learning and its efficient minimization for robust statistics. 2021.
Arthur Gretton, Karsten M. Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel
two-sample test. Journal of Machine Learning Research, 13(25):723-773, 2012. URL http:://mmmr: orgg
html
Rolland L Hardy. Multiquadric equations of topography and other irregular surfaces. Journal of geophysical research, 1605)1190511115, 1971.
Charles A. Micchelli. Interpolation of scattered data: Distance matrices and conditionally positive definite functions.
Constructive Approximation, 3(1::11-22. 1986. doi: O0.100//FF1883414 URL https://dbiiorr//1
1007/BF01 011///000089941444
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013. URL https: wwtpsiiv..rg/bbs/13322
5602
Thanh Nguyen- Tang, Sunil Gupta, and Svetha Venkatesh. Distributional reinforcement learning via moment matching,
Proceedings of the AAAI Conference on Artificial Intelligence, 35(10)99144-9152, May 2021. doi: 10. 1609/aai.. 135110.17104,
aaai. inee//// phpppaaT//apiiieeeeaaeee
Andrew Patterson, Victor Liao, and Martha White. Robust losses for learning value functions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45()):6157-6177, 2023. doi: 10. 10.10///AAMM.2222221100
Martin L. Puterman. Markov decision processes: Discrete stochastic dynamic programming.
Wiley Series in
Probability and Statistics, 1994.
René I. Schilling, Renming Song, and Zoran Vondracek. Bernstein Functions: Theory and Applications, volume 37 of
De Gruyter Studies in Mathematics. Walter de Gruyter GmbH Co.KG, Berlin/Boston, 2. aufl. edition, 2012. ISBN 3110252295.
Gábor Szěkely and Maria L. Rizzo. Energy statistics: A class of statistics based on distances. Journal of Statistical Planning and Inference, 143(8); 2013. ISSN 0378-3758. doi: http:///oo..ogg//1..101........ 018. http:///www..ataaae.......aaaaaa
Holger Wendland. Scattered Data Approximation. Cambridge Monographs on Applied and Computational Mathe- maties. Cambridge University Press, 2004. doi: 10.1017/CB09780511117777
Derek Yang, Li Zhao, Zichuan Lin, Tao Qin, Jiang Bian, and Tie- Yan Liu. Fully parameterized quantile function for distributional reinforcement learning. In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), December 2019. URL https: wttp::/wwwwwww.ooootttttt wtte:ttt.ooomeeetaeeeeeeeee
ful ly
for
13

---

Published in Transactions on Machine Learning Research (08/2023)
A Proofs
Lemma A.1. Let ky be the multiquadric kemnl,, then MM is smooth.
Proof. The multiquadric is infinitely smooth, i.e. is infinitely differentiable. We can see this by looking at its deriva- tives. First, for n = we have
d - =
dx
and for >> 1
dm
V1 12x = h² (2) + トク
dx
where Pr is a polynomial of degree n. Since MMD kb)) is finite sum of infinitely smooth functions, it is also infinitely smooth.
According to Alekseevsky et al. (1998), non-negative infinitely smooth functions always have smooth square root. (M kk)) is non-negative (Gretton et 2012) and shown above, infinitely smooth, hence MMDs(1, kn) must be smooth.
Lemma A.2. Let kn be the MQ, and MMD,
(a (fx:) (yihi : A B R, where A and sets of real values 0ien E Rd and yin E IR respectively. Then MMD, is Lipschitz continuous wrt A and B.
Proof.
The first derivative of the MQ kernel is bounded,
0
V 1 + h2 12| y11
-1²= y) (1+11(x--))) - s 11-hlall2 Vdh.
dæ
MMD, ((x;), (y:); k.) is finite sum of functions with bounded gradient, its gradient is necessarily also bounded.
Because
Lemma A.3. a) == V a vb+ fst is positive definite for a > b.
Proof. Let g(t == f(VB)= va+t vb+ We use that a function f() is positive definite if and only if g()) is
completely monotonie. By induction, it follows that
d"
Vctt= (c+ (3
dm
where (+)n is the Pochhammar symbol. Hence,
dm
Va - vo+t
dta
(a 4)222/ (3
3
- - +bb+t)
2 )
)
2
(3
(aa (a (b+ (b+ 1)/
The first term is positive for n = and negative otherwise. The second term is positive for = and has sign 1)m+1 otherwise. It is therefore evident that g(t) is completely monotonic and thus f() is positive definite (Lemma 3.2
Definition A.I. ( is the distribution of random variable f(X), with X μ.. This is known as the pushforward measure of y by S.
Theorem A.I. The distributional Bellman operator is contraction in MMD with the multiquadric kemel.
14

---

Published in Transactions on Machine Learning Research (08/2023)
Proof. Let & be the multiquadric kernel, i.e.
k< v) 1b(x v) -V1 h2
Nguyen-Tang al. (2021) prove that for the distributional Bellman operator be contraction it sufficient show that for all y E (0,
MMD? (frr)) (Fr)) MMD? (M, v; k),
where (fr.) denotes the pushforward measure of by (frr)# Nguyen-Tane et al. 2021 Lemma 4) states that this holds with equality for all scale sensitive and shift-invariant kernels k. i.e. whenever k yo, = 7 k(r, v) for some a > We will show that this condition holds for the multiquadric kernel. First, we will define
1 72 111121112 (x) = -V
with
(x v) 25 y).
Note that (112) (785(22)
/ k(z, (()
MMD? (Ura)mms (H
+ / / k(w, w') ((d))))))))
2 / f / k(w, (fr, (v(w))
/ / k(r nz,r
+ k(r + yeur + "rr')uu(dd (dd)))dddoo
2 /
k(r + y2,
/
007x 72
s(Tw (dd
2 /
4(72 yw (w)
/ / n 2' )(dx(')) + /
nā - w (()))(d)))((d
2 1B, w))ulz))u(ww)
y MMD2 (1,1;
Showing that MMD (n, < MMD v; k), or equivalently MMD (H, v; k)) MM 2 0 there- fore sufficient.
15

---

Published in Transactions on Machine Learning Research (08/2023)
( / v= =))((z))(dz2) + / [/ ( (d)((du))((du')
MMD"(W, MMD3 (M, (M
/ / </ ())(())(((ww) - ( (z))==()((dz((((((
/
fy w') ('w(((dd))(((cco 2 / 8, w))((z))((ww)
= / v(z - z') - 2iy(2 z') 2))pdz))(dz))
+ / / V/w w') "5 w' (ww':wwww(dw))))))) (d')(dho)u(d''
/ 612 w) - -,,z w))ulz)u((dw))
= MMD (H, kny)
Since MMD? is non-negative for conditionally positive definite kernels, it is sufficient to show that (=)) v/. (z)
is conditionally positive definite for y E (0, 1). Lemma 3 shows that
((z) - (2) =
722 + 1 V 1211=118
is positive definite for y (0, 1), and therefore conditionally positive definite, which concludes the proof.
Corollary A.1. Let k(ar, y) exp(-h² y (2))- Then the distributional Bellman operator is contraction in MMD2 (μ, ur k) for distributions μr, v with support Sp S, such that for some < 2
1og(>2-9)
sup E.S
U112
(17)
(72 1)A2
Proof. Utilizing the proof for Theorem
3.1 we note that all we have to is show that
> 3) ((73,,8))
is conditionally positive definite when Equation 17 holds.
By Lemma 3.2 this means that
d r" exp(-1i2) exp(-127²2) h²² exp(-1²2) h22²2 exp(-x22 2)
dz
is conditionally positive definite. Let begin by showing that the first derivative is negative.
h2 exp( ex
- exp(-hi22) -- exp(-(??=))))
- -12 > log(n2 1122²=
- 11722 - 1/ 2 log(?"-99 2-a)
- 1²(2" -1)= 2 log (72 22aa)
1og()2-a) (n2 1)h²
(This is the requirement in Equation 17
16

---

Published in Transactions on Machine Learning Research (08/2023)
Now show that for n 2 1
(-1)" d dz" ? exp(=1?=) exp(-11372)) 0
whenever
dz 7a exp(-1²2) - cxp( cxp(-hh3²2) 20
y E (0,1) we have
d (-1)" dz" d " exp(-h²2) exp(-107²>)
h² 2m exp(-h²²2 2 2n exp(-hi?²2)
= h 2 (7 h2 exp (-h²z) h2n2"0 2n exp ( (-h²?²2))
2 2 (7 exp (-h²z) - 12 exp (-11?²2))
Given that t nh2 cxp (-h²2) 1192))-22- h2n² exp (-11)²2) 2 0 this shows that the (-1) do 7 exp(-hi h222)²²2))) -xp(( -xxp(-12722) is also greater than equal zero.
17


Reviews:
[{"review_id": "pR1oNLp42U", "reviewer": "Reviewer_KxkT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n* The background section is well written and covers the distributional RL landscape well\n* The proposed kernel is an addition to the literature that might be of interest to researchers in the distributional RL community.\n* The procedure to evaluate MMD with the RBF kernel to showcase some of its weaknesses crystallizes the requirement for an improved kernel.\n* Figure 2 is a good qualitative comparison of the different kernels\n* The theoretical analysis of the new kernel gets at important properties desirable of any measure for the distributional Bellman error.\n* The analysis to rationalize the empirical success of the RBF kernel is also a worthwhile addition to the manuscript.\n* The experiments seem to indicate the the multiquadratic kernel is not worse than the RBF kernel for use in the MMDQN algorithm.\n\n## Weaknesses\n* There are certain changes that would make the paper stronger which I list out below\n* Apart from the toy example in Section 3 and the results in Figure 2, there is no experiment showcasing the stability of the MQ kernel compared to the RBF kernel. The `SpaceInvaders` result could be construed as such, but improved performance could be attributed to various differences and not just improved convergence.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on the moment matching approach to minimizing the distributional Bellman error. The aim of the paper is to propose a theoretically sound measure of difference between two distributions that can be used for this approach. To do so it proposes the multiquadratic kernel, and analyzes its properties when applied to distributional reinforcement learning.\n\nThis theoretical analysis shows that the maximum mean discrepancy (MMD) for the multiquadratic kernel is a metric in the space of distributions, and that the distributional Bellman operator is a contraction in MMD with this multiquadratic kernel.\n\nAdditionally they also show some conditions under which the distributional Bellman operator is also a contraction in MMD for the RBF kernel.\n\nThe experiments modify MMDQN with this multiquadratic kernel and show that it performs just as well or better compared to the RBF kernel.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The requested changes below are in decreasing order of importance. I am also willing to waive the requirement for any of these based on compelling reasoning against them.\n\n* In Section 3, some analysis of the presented result, a description of the Anderson-Darling test, and what a reader should expect and take away from the section would improve its effectiveness\n* In experiments, a comparison with some of the other effective distributional RL techniques such as QR-DQN would give a more holistic view of the proposed kernel beyond just the MMDQN algorithm.\n* To showcase the stability of the MQ kernel, perhaps the result of a sweep over `h` for the MQ kernel and for the radius over the RBF kernel could be done to show the relative stability of the MQ kernel?\n* Section 3 and Figure 2 discusses the MQ kernel before it is introduced. Rework this section and the next. Perhaps introduce the MQ kernel earlier if the analysis in Section 3 is reliant on comparison with the MQ kernel.\n* In Section 4, the paper states: \"First, under the multiquadric kernel, maximum mean discrepancy (MMD) is a metric in the family of valid distributions.\" Clarify that this statement is proved in the paper. As it stands it seems like a statement without any source or proof behind it.\n* In paragraph 3, on page 7, the paper states: \"On the other hand, if the loss function is smooth, then the gradients will be relatively stable, which allows the model to make more consistent and reliable updates to its parameters during training.\" Is there a citation to back this claim?\n* The list of desirable properties for the MQ kernel on this page (page 7) is also out of order compared to their description in text above it. Reorder them to be consistent, maybe?\n* In Definition 4.1, could $\\psi$ be described, and the connection of the equation to the function $f$ also be clarified?\n* In Definition 4.2, could the significance of the * in $c_j^*$ be clarified?\n* Since the results are a comparison over only 3 seeds, perhaps metrics such as the interquartile mean across all the runs or some of the other suggestions put forth in [1] could be used?\n\nReferences:\n[1] Deep Reinforcement Learning at the Edge of the Statistical Precipice https://arxiv.org/abs/2108.13264", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hbZjRh7lON", "reviewer": "Reviewer_KzRQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ A novel result that has not been addressed yet.\n\n+ Half survey on the topic of kernels for MMDs.\n\n+ I liked that the authors showed the practical results were not compromised by their proposed metric. \n\n- I'm not sure how many researchers will take interest in the result, that are very limited in scope. \n\n- The significance and novelty, while exist, are very modest - previous kernels comply to almost all 4 properties, and as the authors suggested themselves smoothing the l1 near 0 is very natural thing to do that was already done in multiple other cases.\n\n- Some clarity issues (see requested changes). While the writing was overall clear, it wasn't organized enough.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors analyse the properties of multiquadratic kernel for moment matching distributional RL. They show it holds 4 desired properties:\nsmoothness, bounded gradient, metric properties and contraction. They also show that empirically it works just as fine as the standard alternative.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Small clarity issues (none would affect the recommendation for acceptance) but I think would help the paper:\n\n1. “Our contribution is mainly of a theoretical nature, presenting the first formally sound kernel for moment-matching distributional reinforcement learning with good practical performance. “\n\nThis sentence is two sentences in one and thus confusing: 1. The contribution is the first theoretical result 2. good practical performance (this already exists in the previous sentence).\n\n2. You relate to MQ before it is defined in the end of Section 3 and in Figure 1. \n\n3. Introduction and related works very long making the paper survey-like. \n\n4. In Section 4 the order stuff are written in is confusing. You assume MQ is a contraction, but then you prove it, same for metric properties. The table is after the main points. The easiest fix for me would be to reference the table and theorems as soon as you talk about things related to them.\n\n5. In concisely formulated, Ch is not defined.\n\n6. In Definition 4.1, psi is not defined.\n\n7. Figure 3 – missing axis names (y=episodic reward, x=steps ?)\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Yb19XOGHNm", "reviewer": "Reviewer_376H", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- The experiments demonstrate strong empirical results, and are relatively statistically significant (8 Atari games x 3 seeds each).\n- They make a novel observation that using the MMD with an RBF kernel was not able to accurately fit multimodal distributions.\n\n\n**Weaknesses**:\n- I have serious doubts regarding Corollary 4.1. The inequality seems impossible to hold: the LHS is always positive as it is a norm, while the RHS is negative (the numerator is positive, $h^2$ is positive, and $\\gamma^2-1$ is negative since $\\gamma \\in (0,1)$, so the whole term is negative). As the corollary states that the contraction property holds whenever we have that LHS $\\leq$ RHS, this statement appears to trivially never hold.\n\n- The fact that properties (1)-(4) (using the naming of the paper) holds for the multiquadratic kernel is one of the main contributions of the paper. The authors did not prove properties (1) nor (2) however, and claim “Properties 1 and 2 can be verified by inspection”. These properties do not appear trivial to me however, and in fact I am doubtful they both hold. To this end I request the authors to include a proof of these properties in the paper (in the appendix is fine).\n\n- Definition 4.2 as presented is not the definition of positive-definite kernel used in the literature (e.g. Gretton). It seems the authors define a positive definite kernel to be the negative of a distance of negative type as introduced in (https://arxiv.org/pdf/1106.5758.pdf). Of course the authors can define things as they wish, but I believe that by using the commonplace term “positive definite kernel” with a different definition misleads the reader and should be changed. Note that the multiquadratic kernel k_h they introduce is not a proper positive definite kernel in the traditional sense (the one used in gretton…) - a simple counterexample is that k(1,1) < 0, while this must be positive for any p.d. Kernel.\n\n**Minor comments/typos**:\n\n- In Definition 4.2, the terms are sums of $cc^*$ – I assume this should be $c^* c$ as $c c^*$ is a matrix.\n- After introducing the optimal Bellman operator and demonstrating that $(T^*)^kQ\\to Q^*$ as $k\\to\\infty$ for any $Q$, the authors state \"This means that if we adopt π in a “GLIE manner”, then $Q^π = Q^*$ in the limit\". I don't believe this discussion is relevant here, as the convergence of applications of the Bellman operator does not have anything to do with exploration as presented, and this discussion appears out of place to me.\n- There are a number of places in the text which would benefit from a careful re-reading, there are a number of minor typos and grammar mistakes.\n- You refer to the multiquadratic kernel in Section 3 before defining it in Section 4.\n- The equations MMD_b and MMD_u referenced were both introduced by [Gretton et al. (2012)](https://www.jmlr.org/papers/volume13/gretton12a/gretton12a.pdf), I would recommend citing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors build upon Nguyen-Tang et al. (2021)’s work on distributional RL using the MMD. They combine Nguyen-Tang et al. (2021)’s algorithm MMDQN with a new kernel, the multiquadratic kernel, and demonstrate that it expresses various theoretical guarantees. They evaluate it empirically against MMDQN with the RBF kernel used in (Nguyen-Tang et al., 2021), and obtain reasonably strong results. They complement this with theoretical support for the multiquadratic kernel, by demonstrating that the MMD with this kernel is smooth, has bounded gradients, and the distributional Bellman operator is a contraction under the MMD with this kernel.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to see my weakness listed above to be addressed, as I consider them of high importance.", "broader_impact_concerns": "The paper does not contain a \"Broader Impact Statement\". Despite this the contribution is mainly fundamental and I think this is OK.\n\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers the MMD approach to distributional reinforcement learning. It proves that Multi-Quadratic kernel has certain desirable properties (being a metric in the space of probabilities, Bellman operator being contraction w.r.t. the kernel, smoothness, and boundedness of derivative). Other commonly used kernels, including RBF, do not satisfy all these properties.

We have two Accepts and one Leaning Accept among reviewers. They find the revisions satisfactory. So I recommend acceptance as is.

Decision: Accept (poster)