An analysis of distributional reinforcement learning with Gaussian mixtures
Authors: Mathis Antonetti
Henrique Donancio
Florence Forbes
Venue: Accepted by TMLR ()

Abstract: Distributional Reinforcement Learning (DRL) aims at optimizing a risk measure of the return by representing its distribution. However, finding a representation of this distribution is challenging as it requires  a tractable estimation of the risk measure, a tractable loss, and a representation with enough approximation power. Although  Gaussian mixtures (GM) are powerful statistical models to solve these challenges, only very few papers have investigated  this approach and most use the L$_2$  space norm as a tractable metric between GM. In this paper, we provide new theoretical results on previously unstudied metrics. We show that the L$_2$  metric is not suitable and propose alternative metrics, a mixture-specific optimal transport (MW) distance and a maximum mean discrepancy distance.  Focusing on temporal difference (TD) learning, we prove a convergence result for a related dynamic programming algorithm for the MW metric. Leveraging natural multivariate GM representations, we also highlight the potential of MW in multi-objective RL. Our approach is illustrated on some environments of the Atari Learning Environment benchmark  and shows promising empirical results.

Published in Transactions on Machine Learning Research (12/2025)
An analysis of distributional reinforcement learning with Gaussian mixtures
Mathis Antonetti
mathis.antonnetti@hrri...
Unio. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble, France.
Henrique Donâncio
henrriuue.oonancioohhrr..
Unio. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble, France.
Florence Forbes
Horemce..orbes@iinn
Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble, France.
Reviewed on OpenReview: https: // oww.neveee..........omo net/ form///rmmm id= b4VgI1RTv8
Abstract
Distributional Reinforcement Learning (DRL) seeks to optimize risk-sensitive objectives by modeling the full return distribution rather than only its expectation. key challenge to choose a return distribution representation that allows (i) efficient estimation of risk measures, (ii) tractable optimization, and (iii) sufficient expressiveness. Gaussian mixtures (GM) provide flexible and powerful representation for this purpose, yet they remain un- derexplored in DRL, with most existing methods relying on the T2 norm as tractable metric between GM. In this work, we conduct a theoretical and empirical study of alterna- tive metrics for GM-based DRL. We show that the L2 norm not suitable and introduce two principled alternatives: a mixture-specific optimal transport distance (MW) and a max- imum mcan discrepancy (MMD) distance. For the MW metric, we establish convergence guarantees for dynamic programming algorithm related to temporal--diffrencee (TD) learn- ing. Leveraging multivariate GM representations, we also highlight the potential of MW in multi-objective RL. Experimental results on selected Atari Learning Environment tasks illustrate the practical benefits of the proposed metrics, showing promising performance.
1 Introduction
Deep Reinforcement Learning (RL) has shown outstanding results in robotics (Li ct al. 2018 and control Yang al. 2018 Kaufmann et al. 2023 by estimating and optimizing the expectation of the total reward (or return) given to an agent. Distributional RL (DRL) (Bellemare et al. 2023) generalizes the approach by estimating the whole distribution of the return, leading to better results than the non-distributional approach (Dabney et al.] 2018a)). DRL has many other benefits. It fits better than the traditional approach in stochastic settiigg (Martin et al. 2020 and it allows for new exploration strategies taking into account alcatoric uncertainty (Mavrin et al., 2019 DRL can also be used to design risk-aware RL agents using distortion risk measures that could lead to better results depending on the environment. Indeed, risk-averse policies lead to longer play times and better returns, in life-dependent games (Dabney et al. 2018a).
To derive practical DRL algorithms, two ingredients are required. First, tractable and informative represen- tations of returns as probability distributions, which are infinite dimensional objects. Second, an efficient quantitative way to compare them through tractable metrics, to define a relevant and useful loss, based on so-called distributional Bellman operator and typically similar temporal difference (TD) lcarning loss in RL. The metric choice is also crucial since each metric offers different theoretical convergence guarantees, e.g. to characterize the contraction property of the distributional Bellman operator and its projection.

---

Published in Transactions on Machine Learning Research (12/2025)
Choosing an appropriate representation of the return distribution remains challenging as it has to meet somewhat opposite requirements. The representation has to be rich enough to capture the return distribution complexity and simple enough to allow tractable implementations at reasonable computing costs. Current the-art DRL methods need large computational resources and/or are mainly based on convenient choices with no performance guaranlee in lerms of their ability to represent actual return distributions. For instance, some methods use discrete distribution representations (C51 in Bellemare et al. (2017a), QR- DQN in Dabuey al. (2018b)) leading to low approximation rales and consequently large numbers of parameters, which increase the computational cost. Another theoretical downside is that those algorithms learn statistics of the distribution and not the distribution itself. It follows that they need to satisfy the approximate Bellman-ccosedneen property as most, such as C51, do not satisfy the exact Bellman--losednes property (Rowland et al., 2019). Another set of methods use continuous representations (IQN by Dabney et al. (2018a), FQF by Yang et al. (2019)) that do not lead to a proper distribution in practice as they approximate the quantile function with non-monotonic surrogates, which questions their use with a risk- aware agent (Théate al., 2023). More generally, most approaches use nonparametrie empirical measure representations opposed to parametric ones corresponding to family of parametric distributions.
In this paper, we investigate paramctric representations and more specifically Gaussian mixtures (GM). Introduced in the DRL framework by Choi et al. [2019), Gaussian mixtures present interesting features. They often lead to closed-form formulas, more likely to yield tractable loss and risk measure ostimations. They are proper distributions, available in any dimension and with good approximation power, see e.g. [gguyen et al. 2023) for a recent reference. When comes to computing metrics, there exists a number of closed- form expressions or efficient estimators specifically tuned to handle comparison between GM. Surprisingly, the large varicty of tractable divergencos between Gaussian mixtures has not been fully exploited yet in a DRL context. In this work, we consider three such metrics, the Jensen-Tsallis (JT), the maximum mean discrepancy (MMD) and a mixture-specific optimal transport distance, introduced by Delon & Desolneux (2000) and named the Sasturr-aassssttein (MW) distance. We compare them in terms of their theoretical and practical performance. More specifically, our main contributions can be summarized as follows:
We prove that, although the JT metric has been used in DRL with GM (Choi et al. 2019), does not guarantee that the distributional Bellman operator is contraction mapping, leading to poor results for certain environments and justifying the search for alternatives. We propose extended JT formulation with more flcxibility.
We introduce MW in DRL new possible metric, which is more tractable than the classical Wasserstein distance when comparing mixtures. We prove the contraction of the projected distribu- tional Bellman operator with MW and the convergence of a dynamic programming algorithm related to TD learning. We then provide a generalization to multi-objective RL.
We study various MMD kernels in the GM setting and give new insights on their performance in DRL. We observe that the so-called unrectified kernel performs as well as other common choices such as the mixture of Gaussian kernels. This is consistent with the good theoretical properties of the former but contrasts with previous results by Nguyen-Tang et al. 2020
Overall, we address a critical problem in DRL, which is the search for effective metrics that maintain theoretical tractability and improve performance. We introduce a new bridge between classical mixture models and modern reinforcement learning, by revisiting GM through the lens of new probability metrics. We provide extensive theoretical justification (Table D Theorems I to 10) for these new metrics and a practical implementation framework. These metrics have not been studied in prior work in GM-based DRL (Choi el al.) 2019 Nam et al. 2021 Zhang el al. 2024). To our knowledge, we present the most broad study of probability metrics under which it is feasible to learn GM representations of return distributions. In terms of impact, our work shows that although the most natural, the JT metric should not be used. Also, we propose the first alternative to the categorical approach, which comes with proved convergence and straightforward extensions multi-objective DRL.
2

---

Published in Transactions on Machine Learning Research (12/2025)
2 Distributional RL
In the standard RL setting, a Markov decision process (R, models the interaction between an agent and an environment. X' and A denote the state and action spaces, R(z, a) is a reward random variable depending on a given state x E X' and action a E A with distribution p(x, a.),, P( a) is a transition kernel to a new state from state a after taking action a, and y E (0, 1) is a discount factor. In RL, we search for a policy <(-la) that maps a state x to a distribution over actions in A. To emphasize this, we use upper case to denote random variables and lower case for their realizations. For an agent taking actions given by policy π, the return is the random variable denoted by (2, a) Ei= R((,, Ar) where Xa- x, Ao = a, Xt ~ P(- | It-1,0t-1), A, T(- mx). The Q-value function is defined by the expected return (z) = (z a)] and can be characterized by the Bellman equation, (²(xzaa = E Ep(z,a) [R(z a)] 1TBP(()0))) (X' ,')], following from the random transition from (I, a) to realizations of A') given by X' ~ P((1 x, a), A' ~ T(. x). The objective is then to find the optimal policy x* which maximizes the expected return, with Q" (s,a) 2 Q" (a;, a) for all (x, a) and и. Q" satisfies the optimal Bellman equation and can be found as the unique fixed point of the Bellman optimality operator T defined as a) = (2( a) with TQ( a)= o)),, [R(x, +.ppp,)))) [ Q(X'', d')]. To this end, TD lcarning consists of minimizing the squared temporal difference (TD) error, which is an estimation using the observed states, actions and rewards of the squared difference between a parameterized Qo and its update via the operator TQa. In deep Q-eaaniig, this is usually performed by considering an e-greedy policy and a neural network to parameterize Qe.
In distributional RL (Bellemare et al. 2023), the idea is to consider the whole random variable Zz(x, a) rather than just its scalar expectation Q" a). An analogous distributional Bellman equation can be derived between random variables. Denoting by of (z, a) the distribution of 2 (2., a), 71 * 72 the convolution between two distributions Tn and 72, and (T,)* the pushforward operator through function T,(2) = 72, we where the expectation corresponds to a mixture distribution over next states. This equation defines the so-called distributional Bellman operator denoted by T and so that (z" = 7f (1, a). See Proposition 2.17 and Figure 2.6 of Bellemare et al. (2023)
can write, (x( = p(z,, a) # Bp(l),)),=((T) (T,)) "7"
for an illustration. Note that 7(2, (resp. TT a)) is the probability density function of Z(z, 0) (resp.
T (z,a)), an equivalent random variable formulation, with d
meaning equality in distribution, is
T 7 a) _ R(x, a) + 7
where X' ~ P(. x, x, ~ 7(. | z). This is referred to as the aundom-variable Bellman equation by Bellemare et al.] (2023) (Proposition 2.16). The hope is then to find the return distribution as a fixed point of the distributional Bellman operator. The TD learning principle can be extended differences between distributions leading to the minimization of quantities of the form D(T (x, a), Fa(z, a)) where D is a discrepancy or quasi-metric between distributions, equivalently denoted using random variables by D(T (IT"Zi(z, Z1 a), Zz(z, a)). We will refer to D as a probability metric. Unfortunately, searching for a solution in the whole space of probability distributions is impossible. To use this approach, two main ingredients are required. We first need representations of distributions, rich cnough to capture the return complexity and second, a choice of D for which a tractable fixed point algorithm can be implemented. This requires tractable evaluations of D(T (z, a), Zz(z, a)) and T being a contraction mapping with respect to D the supremum extension of D defined as D(m, 72)) = sup (1 a), 72(,, a)), where 7h and îa denote two (a.a)EA'xA
collections of distributions, also named return functions by Bellemare et al. 2023 (Definition 4.21)
2.1 Representations of the random return
For DRL, the goal is to define a good approximation (E, of the probability metric space to which Z(x, a) belongs. Several ways to uniquely characterize a distribution are well known, such as the probability densily function (PDF), the cumulative distribution function (CDF), the quantile function or the inverse CDF (QF), the characteristic function (Φ), etc. These representations are recalled in Appendix A In the DRL literature, most choices are based on empirical or particles representations of the above, sometimes referred to as non parametric in statistics. In contrast, in this work, denoting by F latent functional space giving the neural
3

---

Published in Transactions on Machine Learning Research (12/2025)
networks architectures, we propose to investigate a parametric choice using the PDF representation and F = M = UKeNMR where Mr is the set of univariate K-component Gaussian mixtures (GM) whose PDE are of the form,
K (12) = Σ (=(
where PR: of) or simply N (Hz of denotes the Gaussian PDF with pa's 's (resp. o the component means (resp. variances) and mE 's the components weights with my [0, 1] and Σ Th 1.
2.2 Probability metrics
In DRL, probability metrics are used to provide comparison of two random return distributions, generally in order to apply a TD learning principle. To compare distributions, in principle, all divergences are suitable candidates, but only a few have been used in DRL. The main used ones are the Kullback-Leibler (KL) divergence, the Cramer distance (also named energy distance) and the Wasserstein distance. The choice of metric may be particularly important as each metric offers different thcoretical and practical properties. al., (017b)), we have two first desirable properties to look for. First, we need
In DRL (Bellemare provide a contraction property for the distributional Bellman operator, i.e. to have possible return functions 771 and f2, D(T T2) S aD(m,, This property is important as it implies the existence of fixed point that can be reached by applying repeatedly the Bellman operator. To obtain such a property, D is often proved to be ideal, which sufficient condition assuming that D is p-convex (Theorem 4.25 of Bellemare et al. (2023 recalled in Appendix D.1]. Being ideal means satisfying the two (SI) (sun-invariant) and (S) (scale-sensitive) properties below. For X, random variables, (0, 1),
such that for all
(SI) (1 = - D(A+ $ D(XY)
(S) 3c 0, D(AX, S X))(X,Y)
In addition, since minimizing D(T mh 772) is usually done using stochastic gradient-based algorithms, we also need make sure that the numerically found optimum is the good one. For instance, it is proved by Bellemare et al. 2017b) that the Wasserstein distance has biased sample gradients which leads to convergence towards wrong optima in practice according to the authors. Although it is possible to deal with biased stochastic gradient algorithms, see e.g. Rhee & Glynn 2015 Demidovich et al. (2023), second important property of the metric is thus that it satisfies the following (USG) (Unbiased Sample Gradient) property (Bellemare) jet al. 2017b). Let 7e be a probability distribution parameterized by 0 and {Xm]me[M]] a collection of M i..dd. random variables distributed X 7/x Define îM M EM 8x,, the empirical distribution of the (XmIne[M), the (USG) property is satisfied if for all M E E,
(USG) Ex Vo DoD(f = VoD(nx.
As already mentioned, the Wasserstein distance does not satisfy the (USG) property.
In addition to theoretical properties, an important feature the tractability of D to allow practical imple- mentations. In this work, we are in particular interested in tractable D when comparing Gaussian mixtures. The following Table I summarizes, for a number of distances D, whether the four main characteristics im- portant for practical DRL are satisfied. Details are given in the next sections. From the listed features, three metrie-dependent properties can be derived: (1) non-expansive projection, (2) a contracting Bellman operator, (3) a (USG) metric. While (1,2) are of theoretical nature, (3) is an algorithmic issue, that can be more easily addressed. As for QRDQN (Rowland et al.) 2024). a solution to find an (USG) metric with the same projection. Among metrics tractable for GM, the Cramer distance (Section 6 and the mixture-speeific Wasserstein distance MW, (Section 5) are thus the most promising ones as they satisfy
(1,2) and experimental results in Section g show their practical effectiveness.
4

---

Published in Transactions on Machine Learning Research (12/2025)
Table Metrics properties. V (resp.x) means satisfied (resp. unsatisfied), means unknown. The KL, Cramer, MMD, Wasserstein metrics are studied by Bellemare al. (2017b). The Jenson-Tsallis (JT) is proved to satisfy (USG) by Choi et al. (2019). MW2 is Wasserstein distance introduced by [Delon & Desolneux 2020 but not investigated before in DRL. Properties showed in this paper are in red.
Metric properties
Metric
SI
USG
GM Tractability
KL MMDDE. /
/C
MMD bEDE
X
MMD tap
Wasserstein
x
2W2
/
JT
JT
2.3 Analysis of GM dynamic programming
In practice, when choosing GM representations de Mr, also have to handle the fact that the result of the distributional Bellman operator on this class does not in general remain in the class. Practical implementations thus require a projection back on Mx. When choosing probability metric, it is then also important to check whether this projection is non expansive, so that the combination of the projection and the distributional Bellman operator is still a contraction with respect to the chosen metric. We further discuss and specify this aspect in the case of TD learning.
In principle, a contraction property of T deduced from (S) and (SI) is sufficient guarantee the convergence of TD learning. However, practical TD learning algorithm cannot represent the full return function T"n, which could be any return function. Hence, we need to take into account the approximation made at each iteration by the stochastic gradient descent. In this paper, we are only interested in the dynamic programming part of TD learning summarized in Algorithm [ with a projection II that is typically defined as a solution of min Dnn,r)) for any E M. This solution is not necessarily unique, so the projection needs to be parametrized with some parameter w². An example is provided in Scction 5 and Appendix D.8 The full operator is then defined as (11" (x, a) 11 (7 a)) for all return function 7j and (x, a) X A. Using this definition, we can formulate our following main result below. As summarized in Table T only the MADD/Crmer (C), the Wasserstein and Mixture Wasserstein (MW2) satisfy the (SI) and (S) conditions, which limits the hope to get convergence results for the others. Among the former ones, only the Cramer distance is (USG) but it is mentioned in Wiltzer et al. (2024) p.6 that the associated projection is not a non-expansion. In contrast, we show in what follows that the MW, metric is not in general (USG) (Theorem but admits a non-expansive projection IKW,, (Theorem 6). follows Theorem that states the projected Bellman operator contraction. All proofs are provided in the Appendix.
Algorithm 1: Mixture Dynamic Programming
Require: Mixture parameters estimates (f(x)) (f.(x)) : x E X)), projection IIp
for E X' do
Let (x, R, X') define the random transition under policy æ with T(x)
Update ((z) to the mixture parameters corresponding to the distribution of II + mmx()))
Return:
Theorem 1. Assume that p(z, a. E M for every state-action pair (z, a) (where p(z, is the law of the reward R(x, a)), then 11 T TM is contraction mapping with respect to MW2.
Mw.
Thus applying Banach's fixed point theorem, Algorithm 1 converges towards a unique fixed point of k T if R(x, a) is a Gaussian mixture. Unfortunately, as for quantile temporal difference (QTD) learning (Rowland
5

---

Published in Transactions on Machine Learning Research (12/2025)
et al. 2024h, the sample-estimate of Mw, T" is biased (see Appendix D.9 wardly deduce the convergence of our TD algorithm with stochastic gradient descent (SGD). However, we could hope to prove such a convergence by using techniques from Rowland et al. (2024) for QTD, with another metric that satisfies (USG) and for which the SGD converges to the same optima as MW2.
that cannot straightfor-
3 Related work
Most methods (QR-DQN, IQN, FQF) use a quantile- represenlation. The advantage of such an ap- proach that it is straightforward to derive the Monte-Carlo calculation of risk-aware policies with a dis- tortion risk measure (Dabney al. 2018ap using the functions largely studied in economics. Another advantage is that the change of variables, by the distributional Bellman operator is easier to manipulate. Among quantile- guantile-based solutions, state-of- the-art performance is reached by the FQF method that builds a neural network supervising the choice of the sampled quantile fractions. However, it scems to fix an imaginary problem since does not arise from the original RL problem but from the way we represent it and more particularly from the Monte-Carlo estimation of the ID integral that defines the loss. In addition, the monotonicity of the represented quantile function not guaranteed and the loss used (Huber variant of the Wasserstein loss) leads biased gradient estimations. A more detailed discussion is given in Appendix B
the quantile function F-1, (72+,.(T) (T) induced
Therefore, it is interosting to search for alternative representations. In their paper, Choi et al. (0019) take an architecture inspired from Gaussian mixtures where the output of their neural network 7/ (named neural Gaussian mixture) is: no(x, a) Σ TK, (a((z,a)Nirr a)N(Hx, a), (z, a)), k=1
K
where (2, a), (x, o a) have the same architecture in DQN MMnih et al. 2015). To compute = (x) (> d2z, where f/x, resp. y,; the PDF
the temporal difference, the authors use of variable X, resp. Y. This metric also used by Malekzadeh al. (2023) in a different setting. However, it was shown recently by Zhang al. (024)) that the DRL statc-of-the-art could be outperformed using Gaussian neural mixtures with the Euclidean distance between the mixtures parameters instead. In this work, investigate two other metrics detailed in the next sections.
4 Tansen-Tsalli divergence
The Jensen- Tsallis divergence is a generalization of the Jensen-Shannon divergence Tsallis 1988 In their work on DRL, Choi et al. (019 refer to this divergence but they use only its simplest form JT which is the norm the L2 space. To better understand the issue with this metric, we consider a weighted version of it.
Definition 1. Let R R+ measurable function. Let N, X and Y two random variables, whose PDFs, πI and ny, are assumed to have their power p integrable with respect to the w(z)dz measure. JT is defined as,
(X,) - - (r)
and also denoted by ||x - by I1,
When = = 2, we recover the Jensen-Tsallis metric used in Choi et al. (2019). This metric has the advantage to provide closed-form formula for Gaussian mixtures. Let Z1 ~ Eks Tik Iik and Zg Ike m2k f2k two random variables distributed as Gaussian mixtures, with git denoting a Gaussian PDF,
JT, 22) = Σ TuuTreluus grehs + - T (g26. 92th - Σ TH k.f.
A.P.
where (, denotes the weighted Lg scalar product, which for two univariate Gaussian PDFs 91 N(Un, and g2 N(u2, is closed-form, (2) N(uns of oz) [a(w(G)), where G

---

Published in Transactions on Machine Learning Research (12/2025)
N (maitwari
- Unfortunately, in its tractable form, the Jensen- Tsallis is not ideal. More specif-
ofto%
ically, the following result holds.
Theorem 2. 222 is not ideal. Furthermore, if R + R+ is a measurable function such that w(x)
0 I = 0, we have, for A.X.Y random variables, with A independent on X and Y, and such that [A) < 00,
(T + X+Y & 2 X, ())
(1)
w ((x)) JT 2 (X,
JT (AX, S sup zE Xw(r)
(2)
In particular, it can be deduced from 2 that J' does not satisfy (S). The distributional Bellman operator T is then not always a contraction mapping with respect to the associated metric. Moreover, it does not seem easy to correct the problem by taking another w. For instance, 2T,2,2 satisfies (S) but does not always satisly (ST) according to the previous result. See details Appendix D.3
The following result shows that if the reward is not noisy cnough, then the distributional Bellman operator T cannot be a contraction mapping with JT1,2.
Theorem 3. Let y E (0, 1) and X,Y two non-identically distributed random variables. There exists max > 0 such that for any random variable A independent of X,Y, we have
V[A] S a maz 2 + X, A + M
where V[A] denotes the variance of A.
Hence, if a deterministic (or sufficiently low-noise) setting and all states are accessible, T not a contraction mapping with respect to JT,2- A simple illustration of the result can be given in a simplified Gaussian setting. Consider a policy æ that maps any stale x to a single action ag and a transition that leaves the state invariant, i.e. x(-|ar) (.)(.) and P(-Zz, ao) = dz(-). Consider then reward with a Gaussian distribution p(z, on) = NN (.) For two Gaussian distributions m (1r ao) = .V(11, and n2 an) N(22, o²), it follows that [=, (2) = =N(M,, 7202 + +0)) for any 7 and i = 1, 2. For Gaussian mixtures and single Gaussian distributions, the J2 metric is available in closed-form, so that 2TT.2(TT TT)))=
+o (1 -1-a0 exp (- -2(m1-22)2 )) and 2T,,2(1, (7)) V7VOz [1 exp (- (m=-)a)²) )) It is then easy 4(7202
40%
to see that if 72 < 1 and o2 7²02 + ohh then (T T T2))) > JTi,2(m, (2))))1))))).11)) And this occurs as soon 7 E 1) and oR is taken low enough. Conversely, when OR increases 2Ti,2 " 7/1 n²))) decreases and can go below JTi (2). A graphical illustration is given in Figure [ with ag = 0, 7 = 0.5 and on = 0.002 or OR 0.2. For low values of on, the effect of the T is to decrease variances and thus increase the separation between the two Gaussians and consequently their JT distance (Figure (a)) In contrast for higher oR (Figure D (b)), T mh and TT are more intertwined and the distance decreases. To address this contraction issue with the JT1,2 metric, we propose studying other metrics that seem more promising.
5 Mixture-Wasserstel distance
Delon & Desolneux (2000) have introduced distance specifically designed for Gaussian mixtures based the Wasserstein distance. In optimal transport context, by restricting the possible coupling measures (i.e., the optimal transport plan) to a Gaussian mixture, they propose a discrete formulation for this distance. This makes it tractable while in general using the standard Wasserstein distance between mixtures is problematic. Delon & Desolncux| (2020) refer to the proposed new distance as MW2, for Mixture Wasserstein. The MW₂ definition makes use of the tractability of the Wasserstein distance between two Gaussians for a quadratic cost. The standard quadratic cost Wasserstein distance between two univariate Gaussian PDFs 91 = of), 92 = NN(H2 0 is,
W W 92) (w1 - 22)² + (0 ((1--00) 2.
(3)
Section 4 Delon & Desolneux (2020) shows that the MW2 distance between two mixtures can be computed by solving a discrete transport problem.

---

Published in Transactions on Machine Learning Research (12/2025)
(a) Low-noise (a 0.002)
(b) Higher noise (aR 0.2)
Figure I: JT1,2 contraction low (a) and higher (b) noise. The purple curve shows the (1 - (apppp///44) / term in the 1 expression and illustrates how the distance increases with the separation (y).
Definition 2. Let 7n = Eks Tie Iik and by 722 = Ex= T2k O2k two Gaussian miatures. Then, the mizfure Wasserstein distance MW2 is defined as,
min M 12) ()Tl()))T)))))
Wre We(gin Σ k.e
(4)
where ni and iT are the discrete distributions on the simpler defined by the respective weights of the mirtures and [1(11 πT) the set of discrete joint distributions w (wke [0, 1], e [Ki], [K2]), whose marginals nı and n2
Finding the minimizer w² of boils down to solving simple discrete optimal transport problem, where the entries of the K, x K2 dimensional cost matrix are the W2(0I..920 92x) quantities. As implicitly suggested above, MW2 is indeed a distance on the space of Gaussian mixtures; see Delon & Desolneux 2020) In particular, for two Gaussian mixtures ! and 1/2. MW2 satisfies the equality property according to which MW2(n, 7727 = 0 implies that 7 = 77. Expression (4) is interesting as when using (3), it can be favorably compared to the Euclidean distance used in Zhang al. (2024). The latter compares parameters of two Gaussian mixtures, which need to have the same number of components in prescribed order for the comparison to make sense. The MW2 distance instcad is generalization, that be computed between any mixtures without requiring manual alignment, thanks to the transport map. In addition, when considering mixtures tn and m2 of components 4x in distributions family L, we can define the following generalization,
MWP (m) min wEll(m).m))
Σ WHl DP l2)) R.
When D is the Euclidean distance and = 2, such a generalization is discussed in Section 4.6 of Delon & Desolneux (2020). For instance, mixtures of elliptical distributions satisfy the required properties, in particular when considering the easier case of univariate mixtures. In what follows, we will thus assume that MWP D.C is a metric. Details on results that are still valid if MWP. is a quasi-metric are given in Appendix D.6 We can show the following Lemma.
Lemma 1. Assume that D is ideal and C is stable by scaling and summing, then MW D.C is ideal.
It follows from Lemma 1 and Theorem 4.25 in Bellemare et al. 22233 that MWP can make T a raction while maintaining tractability. More specifically, we prove the tollowing result.
Theorem 4. Assume that D is ideal, C is stable by scaling and summing and p(x, a) E L for every state- action pair (x, a) (where p(z, a) is the law of the reward R(I,a)), then T is a contraction mapping with respect to MWP D.C
MW D.C variants and in particular MW3 provide then new interesting alternative metrics for neural mixtures. Unfortunately, the MW3 distance not satisfying in general the (USG) property and does not have unbiased
8

---

Published in Transactions on Machine Learning Research (12/2025)
sample gradients. Indeed, note that 2W2 and W2 coincide on Dirac mixtures and W2 does not have unbiased sample gradients (see Bellemare et. al. (2017b)) so that it is impossible for MW2 to satisfy (USG). The following theorem gives the exact formulation of the bias.
Theorem 5. Let fIM 11 E 5xm. be the empirical distribution of M ii.d (Xm]me[M) from vx = =Ik= (Mx,N R,,0) parameterized by then, EE (xxNNieex, Define 7o
E Ex V -
2 0
Σ # wttpe/osooxxxox..o toxoR0)) 2 RiC
Σ UR B
Σ whm (X) {
where the E [T xx) is the optimal coupling defining MW Me) and wim that of M MW2 no).
The right- side above is in general non zero. If all variances ne are constant, i.e. oh = o, then Vo EE,, ok, = oV, (E xxxxxxx = 0. This condition would be acceptable in practice as the resulting GM remain flexible models but not enough to cancel the right-hand side. However, (USG) is trivially satisfied if no has only component (K 1).
Nevertheless, MW3 remains interesting metric. Theorem below shows the existence of non-expansive projection. Combined with Theorem 4 it guaranties that the corresponding projected Bellman operator is a contraction mapping with respect to 2W2 as announced Theorem As discussed in Section 2.3 this is an important feature to analyse the behavior of a TD algorithm.
Theorem 6. Let K N. There exists a sel W of functions w" : [0, 1] verifying for all L N, [L], Ek= WGx(() (1) = ne, where 1) = =Ei TeN(He, o?), such that the MW2 projections are characterized by w² E W and for all 17 E M, the projection is defined by
TW Σ (x
with (1) E, wix (n)), (u(n)) E. Moreover, Tir,, is non-espansion with respect to MW2 for all w² E W
(aiie and (x = Je.
whenn) Hnla)
Ra(n)
As an example, if n Nipl, 0²) and n NiN (1)),00) T2N 0²) then 1W 71 N(TiM + T2p2, o²). MW2 r/) = x (M1 - n/)² + T2(122 - -2²)² and MW MW, 7,7') (T1/1 + T2/2 - n²)2. It follows from the Cauchy-Schwarz incquality that MW2 ( n, n/) MW (1) n'). Figure 2 gives graphical illustration,
where the non-expansion property is more clearly visualized in 2D, below in the case || μ'll || μLll.
M(n E)
I1 - -111l
(Ti/4) + T2p2)
IeW 7
11 1'll
M(2,, Σ)
Figure 2: Illustration of the non-expansion Riw, (Theorem 6 for K and L 2.
6 Maximum Mean Discrepancy
In their work, Nguyen- Tang al. (2020) use MMD for Q-learning in DRL with particles and have demon- strated competitive performance. Different. MMD exist depending choice of kernel.

---

Published in Transactions on Machine Learning Research (12/2025)
Let ke (R + R kernel. Let P.Q. be two distributions and X, X. (resp. Y.Y) two independent variables following P (resp. Q), with X also independent of Y. The squared MMD (Mazimum Meon Discrepency) between P and Q is defined as
Definition 3.
MMD P. Q) -X X)] B| F) 2X]
If k is a reproducing kernel, the metric MMD, is equal to 1up- - lqql l where K. is the RKHS associated to k and pe (resp. µq] is the mean on this RKHS of P (resp. Q). In this work, we only consider translation- invariant kernels and more particularly, the Laplacian kernel lap y) = toll----ll,, the Gaussian kernel Rn (2r v) = e-0.5%
and the energy kernel (on 9) = -- v112-
The reason why Gaussian kernels perform better than energy kernels, called unrectified kernels by Nguyen- Tang et al. 2020), is still unclear in the literature. Indeod, it is shown by Killingberg & Langseth (2023) that the so-called multiquadric kernel seems better than the Gaussian kernel on a theoretical point of view and less sensitive to hyperparameters. In this work, we propose to shed another light on these kernels comparison by using them with our GM representations. We show in Theorem 10 in Appendix D.11 that the above kernels all lead to tractable formulas for GM.
Regarding the use of MMD with Gaussian mixtures, Nam et al. 2021 have shown the good performance of the Cramer distance, which is equivalent to the energy distance, as recalled in Theorem 11 in Appendix D.122 algorithm and no real assessment has been made for TD algorithms,
However, they use a although they have been considered in /Zhang (023). The MMD metric always satisfies (USG) (Bellemare) et al. 2017b) so all the good properties of J'T w. are recovered but also makes 7 a contraction mapping for the energy kernel. In Nguyen- NNguyen-Tang et al. 2020)), the authors report that the energy kernel (called unrectifiod kernel in their paper) does not seem to give promising results. In contrast, they show that a mixture of Gaussian kernels give better results in practice although the MMD with Gaussian kernel suffers from the same problem as J' They explain this with a moment-matching-iike property of MMDR,..art that should also be valid in our case with Gaussian mixtures. However, we show in our experiments that the energy kernel leads to better results, suggesting that the results obtained in Nguyen-Tane Tang et al. (0020) might. be due to their choice of representations using particles.
7 Generalization to multidimensional rewards
Another good feature of our proposed GM setting is its natural generalization to higher dimensions. Although a full empirical analysis of multi-objective RL approaches would go beyond the scope of this paper, this section provides some thcoretical discussion and first steps towards multidimensional rewards. Generalizing DRL algorithms to the multi-objective case requires to consider multidinensional rewards; see Appendix C.3 for details. This has been considered before but only for independent dimensions (Zhou et al. 2021) or with particles (Wiltzer et al. 2024). However, accounting for dependence between objectives allows to keep track of more information and the use of multivariate particles (or quantiles) soems unsatisfying as quantiles do not generalize satisfyingly in dimension greater than one. In contrast, multivariate Gaussian mixtures provide a natural way to define a TD algorithm for multidimensional rewards. We can easily generalize our previous results in the multidimenssonal case except for the projection associated to MW2. The MW2 as defined by Delon & Desolneux (2020), corresponds to MW2 D.c introduced in Section 3 with D = W. the 2-Wasserstein distance. This distance between two multivariate Gaussian distributions ions in dimension d, 91 = Na(pI, )n) and g2 = Na(u2, Σn) is, W = ||| 122l² + Tr(E; + Σg - 2( /"2:://;;;////11 and there is no obvious way to show that the associated projection is a non-expansion. As an alternative, we propose to make use of our MW generalization to define a new multivariate metric MW2. F. where Md is the set of d-variate Gaussian mixtures and, F²(g1 = || - H2ll2 + Tr((E)/2 /)/3))). For two multivariate GM in dimension d, m CR=1 TINNa(HIRR End) and 772 -kke T2P TTaNa Na(H2RR Ezs,, we define,
Ki
2W FMMaln 12)
min Σ R.
Whe (IIeRR 2 + Tr((11 -
we)l(m).5a)
10

---

Published in Transactions on Machine Learning Research (12/2025)
F.M" is ideal using Lemma 1 and to derive the same kind of non-expansive It is possible to check that MW projection as for MW2 (details are in Appendix DD10 Thus, we obtain the following result. generalizing Theorem 1 to the multivariate case.
Theorem 7. Assume that p(x, a) Mn for every state-action pair (I, (where plz, a) is the low of the reward R(x, a)), then 11 MW F.pad
T" is contraction mapping with respect to MWF. P.Ma.
8 Experiments
Considering a standard TD learning framework, we first compare our proposed algorithm with Gaussian mixtures and different metrics, by running our agent on a selected subset of Atari games referred to as Atari-5. Atari-5 is a subset of 5 representative Atari games exhibited in the study Aitcchios et al. (2023) based on which global normalized score can be computed using weights. These games are by default deterministic. We use the same standard architecture as in DQN (Mnih et al., 2015). We then also rllustrate the different metrics behavior the previous Atari games, with a modified MDP by adding sticky actions with probability 0.25, bringing stochasticity. Details on the hyperparameters used in these experiments are provided in the Appendix. The compared metrics are MW2, J'T,, and two MMDs. As in Nguyen- Tang et al. 22000, consider the kernel built as the following sum of Gaussian kernel kunx ebr( v) Ei=1 / v) and the energy kernel. The results for the Gaussian kernel are not shown as it can be seen as simpler version of kmix rhr and we observed that the Laplacian kernel behaved similarly to the Gaussian one. Results showing average normalized scores and their standard deviations (over 5 runs), using the weights given in Aitchison et al. (202)). , are reported in Table 2 MMD MMDkmttrrrt MW2, perform better than the JT1,2 metric. Interestingly, no environment. shows a real advantage of MMDKuusetr over MMDk which contrasts with the results of Nguyen-Tanng et al. (2020). In particular, in the deterministic setting, the mean normalized score is better for MMDx... The results also illustrate the superiority of our proposed metrics compared to JT2,,, whenever the environment is not stochastic enough, suggested by Theorem 3
Table 2: Metrics comparison on Atari-5. Average final normalized scores and standard deviations (over 5 runs) using the weights given in Aitchison et al. (2023) (best in bold characters).
Environment
JT
MW2
MMD Ken
MMD Kmbxebt
Stochastic
37.24 4 6.02
43.38 + 4.72
46.84 + 4.31
57.66 + 4.47 54.07 + 8.54
Deterministic
34.06 + 6.89
39.90 14.77
57.82 + 18.67
Figure 8 shows in the deterministic (first row) and stochastic environment (sccond row), moving average returns over 30 millions of training frames for each game and the resulting normalized score. For games Qbert and Name This Game, the gap between the JT and other metrics learning curves clearly reduced the stochastic case. This is also observed in the normalized results (last column), comforting our theoretical statement that the noisier the environment, the more contractive T becomes with respect to JT1,2-
We then perform an ablation study, varying the number of mixture components K E {1,3, 5, 16) in the deterministic case with the Atari-3 subset of 3 games proposed in Aitchison et al. (2023) Results for metric MW2 are reported in Appendix Table Surprisingly, a too high K = 16 seems to degrade results. However, Appendix Figure 5, e.g. for the Battle Zone game, suggests that the K = 16 learning curve might catch up with the other curves with more training frames. The similarly performance for K E (1,3,55 5) also probably due to a too small number of training frames. In contrast, in different environments, such that of the Asteríz game, the K = 5 results are significantly better than the K = 1 case with more training frames (200 millions). Experiments made on a set of 3 other Atari games (sce Figure G in Appendix) corroborate this analysis when the number of training frames is increased to 175 millions.
Finally, we observe that the metrics we propose do not introduce much additional computation cost except for MW2 (see Tables 5 and G in Appendix). In the MW, metric case, most of the computation overhead is due to the optimal transport (EMD) solver used internally. This computation overhead may be reduced by reducing K or optimixing the implementation using recent developments in applied optimal transport.
11

---

Published in Transactions on Machine Learning Research (12/2025)
Our code is implemented in Python and is available at https://giitlb. http:://gttiaa...ia..f//r//
9 Conclusion
We showed that. mixtures were interesting models to represent distributions in DRL. Their tractability and expressiveness allow to consider various metrics and result in implementations requiring less parameters than other algorithms. In a standard TD learning sctting, we illustrated that the simple J2 metric may not be suitable and proposed alternative Wasserstein-like and MMD metrics, with better theoretical and empirical properties. In particular, the proposed Mixture-Waat Wasserstein metric showed both tractability and promising performance. To fully justify its use in stochastic gradient-based methods, it would be useful to study whether it is possible to handle its biased sample gradients. We also proposed an extended formulation JT of the Jensen-Tsallis distance with an additional weight term w, which provides more flexibility and could be further exploited. Additionally, as briefly discussed in Section 7 the MW setting provides promising approach to multi-objective DRL but a more complete study was out of the scope of this paper. Appendix Section E we also presented another advantage of mixtures, which their particular adaptability to an alternative way to solve the distributional Bellman fixed point using stochastic approximation principle. This alternative opens the way to the design of new efficient DRL procedures whose investigation is left for future work.
Acknowledgments
The authors were granted access to the HPC resources of IDRIS under the allocation 2024-A0171015651 made by GENCI.
IT,,2
MW?
MMD
Sontle Zone
Double Cunk
Name This Game
Mhoerix
Ubert
Acari.5
30000
4000
3000
30000
-2
50.00
300C
batume
-14
40.9
20000
-13
22-
frames
Tuining Framss
Taining flamess
Training fromes
Tainig frames
maining frames
JTs.2
MWV
MM Decuee
MMA
Eattle Zone
Douele Dunk
Name Thile Dame
Phoerix
Obert
20000
30.55
-73
$3000
20003
-002
15:99
4003
30039
-125
-152
3000
10:9
20
10C38
-125
2000
500
-22
1003
frimes
Imiinng
Comparison of (orangg),, JT (blue,, MMD (red) and MMD Mnieery (green) metrics for Atari-5 games in deterministic (first row) and stochastic (second row) environment. Moving average return for each game and normalized score, over 50 episodes, with respect to the number training frames. Curves are averaged over 5 runs with shaded areas representing standard deviations.
Figure 3:
12

---

Published in Transactions on Machine Learning Research (12/2025)
References
Matthew Aitchison, Penny Sweetser, and Marcus Hutter. Atari-5: Distilling the arcade learning environ- ment down to five games. In Andreas Kranse, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 421-438. PMIR, 23-29 Jul 2023.
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 449-458. PMLR, 06 11 Aug 7a.
Marc G. Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi Munos. The Cramer Distance as a Solution to Biased Wasserstein Gradients, 2017b. URL
httpg//bs11100...11 10743
Marc G. Bellemare, Will Dabney, and Mark Rowland. Distributional Reinforcement Learning. MIT Press, Cambridge, 2023.
Fabio Bellini and Elena Di Bernardino. Risk management with expectiles. The European Journal of Finance, 23(6):487-506, 2017.
Vladimir Bogachev. Measure theory, volume I of Theorem 3.9.4. Springer, 2007.
Xin-Qiang Cai, Pushi Zhang, Li Zhao, Jiang Bian, Masashi Sugiyama, and Ashley Llorens. Distributional ppreto-optimal multi-objective reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, pp. 15593-15613, 2023.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gclada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: A Research Framework for Deep Reinforcement Learning, 2018. URL http://arxiv.org 1010/881206610
Arthur Charpentier. An introduction to multivariate and dynamic risk measures, 2018. URL https: www.seecceehaa-1131......
Zaiwei Chen, Siva Theja Magulur, Sanjay Shakkottai, and Karthikeyan Shanmugam. Finite-Sample Anal- ysis of Contractive Stochastic Approximation Using Smooth Convex Envelopes. In Advances in Neural Information Processing Systems, volume 33, pp. 8223 0234,,23,--- 2020.
Yunho Choi, Kyungjae Lee, and Songhwai Oh. Distributional Deep Reinforcement Learning with a Mixture of Gaussians. In The 2019 International Conference Robotics and Automation (ICRA), 2019.
Will Dabney, Georg Ostrovski, David Silver, and Remi Munos. Implicit quantile networks for distribu- tional reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 1096-106100 1105. PMLR, 10-15 Jul 2018a.
Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018b.
Julie Delon and Agnès Desolneux. Wasserstein-type distance in the space of Gaussian Mixture Models. SIAM Journal on Imaging Sciences, 13(2):936-970, 2020.
Yury Demidovich, Grigory Malinovsky, Igor Sokolov, and Peter Richtárik. A Guide Through the Zoo of Biased SGD. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
Remi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurelie Boisbunon, Stanislas Cham- bon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Leo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, Ievgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. POT: Python Optimal Transport. Journal of Machine Learning Research, 22(78):1-8, 2021.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Conor F. Hayes, Roxana Radulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Math- Teu Reymond, Timothy Verstracten, Luisa M. Zintgraf, Richard Dazeley, Fredrik Heintz, Enda Howley, Athirai A. Irisappanne, Patrick Mannion, Ann Nowé, Gabriel de Oliveira Ramos, Marcello Restelli, Pe- ter Vamplow, and Dicderik M. Roijers. A practical guide to multi-objective reinforcement learning and planning. CoRR, 0212/0003005668, 2021. URL hnttps://arrtt www.aawwwww.rgab/////////////
Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620:982 987, 2023.
Ludvig Killingberg and Helge Langseth. The Multiquadric Kernel for Moment-Malching Distributional Reinforcement Learning. Transactions on Machine Learning Research, pp. 1-17, 2023.
Zhijun Li, Ting Zhao, Fei Chen, Yingbai Hu, Chun- Yi Su, and Toshio Fukuda. Reinforcement learning of manipulation and grasping using dynamical movement primitives for a humanoidlike mobile manipulator. IEEE/ASME' Transactions on Mechatromics, 23(1):121 131, 2018.
Elliott H Lieb. Convex trace functions and the Wigner- Yanase- Dyson conjecture. Advances in Mathematics, 11(3):267-288, 1973.
Parvin Malekzadeh, Ming Hou, and Konstantinos N. Plataniotis. A unified uncortainty-awwree exploration: Combining epistemic and aleatory uncertainty. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2023.
John Martin, Michal Lyskawinski, Xiaohu Li, and Brendan Englot. Stochastically dominant distributional reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning. PMLR, 2020.
Borislav Mavrin, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional reinforcement learning for efficient exploration. In Proceedings of the 36th International Conference Machine Learning, volume 97, pp. 4424-4434. PMLR, 09-15 Jun 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Toannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):522 533, 2015.
Daniel W Nam, Younghoon Kim, and Chan Y Park. GMAC: A Distributional Perspective on Actor-Critic Framework. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 7927 7936. PMLR, 18 24 Jul 2021.
T.T. Nguyen, F. Chamroukhi, H.D. Nguyen, and G. J. McLachlan. Approximation of probability density functions via location-scale finite mixtures in Lebesgue spaces. Communications in Statistics Theory and Methods, 52(14):5044 5059955599 2023.
Thanh Nguyen- Tang, Sunil Gupta, and Svetha Venkatesh. Distributional Reinforcement Learning via Mo- ment Matching. In Proceedings of the AAAT Conference on Artificial Intelligence. AAAI, 2020.
Chang-han Rhee and Peter W Glynn. Unbiased estimation with square root convergence for SDE models. Operations Research, 63(5):1026-1044 2015.
Diederik M. Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48(1):67-113, 2013.
Mark Rowland, Robert Dadashi, Saurabh Kumar, Remi Munos, Mare G. Bellemare, and Will Dabney. Statistics and samples in distributional reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 5528 5536. PMLR, 2019.
Mark Rowland, Rémi Munos, Mohammad Gheshlaghi Azar, Yunhao Tang, George Ostrovski, Anna Haru- tyunyan, Karl Tuyls, Mark G. Bellemare, and Will Dabney. An analysis of quantile semporal-differon learning. Journal of Machine Learning Research, 25, 2024.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Gabor Szekely. E-statistics: The energy of statistical samples, Oct. 2002. Technical Report 2-16.
Thibaut Théate, Antoine Wehenkel, Adrien Bolland, Gilles Louppe, and Damien Ernst. Distributional reinforcement learning with unconstrained monotonic neural networks. Newrocompulingg 534(C))119 219, 2023.
C. Tsallis. Possible gencralization of Boltzmann- Gibbs statistics. Journal of Statistical Physics, 52:479-487 1988.
Harley Wiltzer, Jesse Farebrother, Arthur Gretton, and Mark Rowland. Foundations of multivariate dis- tributional reinforcement learning. In Advonces in Neural Information Processing Systems, volume 37, 2024.
Derek Yang, Li Zhao, Zichuan Lin, Tao Qin, Jiang Bian, and Yan Liu. Fully parameterized quantile function for distributional reinforcement learning. In Advances in Neurol Information Processing Systems, volume 32, 2019.
Yaodong Yang, Jianye Hao, Mingyang Sun, Zan Wang, Changjie Fan, and Goran Strbac. Recurrent Deep Multingent Q-Learning for Autonomous Brokers in Smart Grid. In Proceedings of the T Twenty-Seventh International Joint Conference Artificial Intelligence, IJCAI-18, volume 23, pp. 569 575, 7 2018.
Ruichong Zhang. Cramer Type Distances for Learning Gaussian Mixture Models by Gradient Descent, 2023.
URL https://aaaai /ttp:////xxvvvvvo org/ 235/ 2307 06753
Weijian Zhang, Jianshu Wang, and Yang Yu. Distributional Reinforcement Learning with Sample-set Bell- man Update. In IEEE, International Conference on Robotics and Automation (ICRA), 2024.
Fan Zhou, Chenfan Lu, Xiaocheng Tang, Fan Zhang, Zhiwei Qin, Jieping Ye, and Hongtu Zhu. Multi- objective distributional reinforcement learning for large-scale order dispatching. In Proceedings of the IEEE International Conference on Data Mining (ICDM), pp. 1541 1546. IEEE, 2021.
A Representations of the random return
Random returns can be characterized in several ways. The most common representations are using the probability density function (PDF), the cumulative distribution function (CDF), the quantile function the inverse CDF (QF) the characteristic function (Ø). Specifically, denoting by F latent functional space giving the neural networks architectures, the most used representations are the following.
CDF representation: E (Z FFFF,F2=11 In that case, we have Ezz+r(2) Fz (-=))) and
B[Z(,,a)] (2)d+ / (1- (z)))
QF representation: E (2 Ef = f}. In that case, we have F-1 (2) = =>FZ' (FZz(sa) (2) +r and
E| EIZ(r, a)) [ (t)dt.
PDF representation: E (Z 3f e F,1z = f}. In that case, we have (2))((2) = 1/ --r and
E [[)(x,a) = 2))
Characteristic function representation:
E = (f F, f)- We have (t eir and denoting by 3 the imaginary part, EiZ(z, S (o.)
15

---

Published in Transactions on Machine Learning Research (12/2025)
B Details on related work
Most methods (QR-DQN, IQN, FQF) use a guased representation. The advantage of such an ap- proach that it is straightforward to derive the Monte-Carlo calculation of risk-aware policies with a dis- tortion risk measure 3 using the functions largely studied in economics as follows
Q(x,a) S.' [ (2 (3(T))
B.1 Discrete representations
The principle of discrete algorithms is to take mixture of atoms 72 = =E wrdgg. Then there are two common points of view. Either we set the quantiles 0g and we classify the weights using the KL divergence as this is done in C51, or we set the weights wy (taking usually wk - xx and we regress the quantiles (quantile regression) as this done QR-DQN using the loss
K
2)
= Σ k=1
Blp (2 - Bx)],
where (u) = -1-luu and ΣE W/.
Later, Rowland et al. 2019) apply the expectile regression in the same manner, by taking p+(u) = (r luco) u instead, leading to better practical results for their ER-DQN algorithm. This is not surprising, as the goal to estimate the mean return and expectiles are generalizations of the mean, while quantiles are generalizations of the median.
Alternatively, Nguyen-Taaie Tanen-TTangee et al. 2020) estimator
the MMD metric, instead of the previous losses, with the biased
(xMD(X,Y) N 1
XX,,,))) Σ K(Y,Y, 1
Σ k(,,, 1
M
NM
where = (X.)eE() and Y = (Yi)e[[M are sample sets generated from the distributions of Zo and Ž. They use this estimator because it leads to less variance in practice, compensating the bias. However, they do not consider the possibility of exact computation in the case of parametric representations.
B.2 Continuous representations
Dabney et al. (2018a) use a fully-continuous representation of the distribution using neural networks which leads to a better approximation but they use quantile samples (Monte-Cate to estimate the Huber loss. Thus, this method adds unnccessary noise in the loss estimation depending on the choice of quantile fractions. This is why the stale-of-Che-art is now the FQF method that builds neural network supervising the choice of the sampled quantile fractions.
C Advantages of parametric representations
We further specify, in the next sub-sections, some advantages of using parametric representations instead of the more common approaches.
C.1 Computational tractability
Gaussian mixtures (GM) provide good tradc-off between expressiveness and computational cost. Sctting typerparamelers to the values in the original papers, we can assess, for various methods, the number of parameters as a function of the number of states [A]. For our approach, denoted below GM-DQN, the number of mixture components is set to K = Conformity satisfied when the representation leads valid distribution. The comparison reported in the following Table, which shows that GM provide the smallest. number of parameters and preserve conformity.
16

---

Published in Transactions on Machine Learning Research (12/2025)
Method comparison
Method
Number of parameters
Conformity
GM-DQN
15 x [
C51
51 x |
QR-DQN
200 [A]
IQN
576 [A]
FQF
608 AA]
C.2 Tractability of risk measures
In DRL, generally use risk-aware policics and variants to recover the agent policy from the Q-network using something like
(a x) = LaeK(() [[(x)]
K(x)
(
d'EA
where R. is a risk measure.
Let a E 1) and X a random variable, the expectile ea is uniquely defined as the solution of (see Bellini & Bernardino (2017)
aEE[(X - ca(x))=]= ((X) - ex
It is well-known that ey (X) E[X] and a + ea (X) is increasing. It leads to an interesting greedy policy with R ea. This policy risk-averse if a < and risk-seeking otherwise. Similarly, we can take the quantile version R 9a.
In our case with Gaussian mixtures, we can casily compute the expectiles and the quantiles using the Newlon-Raphson method.
Theorem 8. Let X EK (NN( of), we have (a = 0 and Qa = 0 where
K
K
Ea(x) := (1 - (0x)
Σ
Th (oWN (z: oR) + (x - Mk) FN (x) + Σ
xb(I - mkk),
1=1
Qa(x) Σ TK (uFiiinn (x)
Moreover, we have
(x) = (1-2a)Qa(a) + 20(11 a))
Another possible risk measure R(X)= = Σ ( +nxx) for X X~ Ek Tg (1), All these proposed risk measures are popular statistics and fully tractable using Gaussian mixtures.
As illustration, Figure shows the effect of more (or less) risk-seeking policy on performance, still measured and shown the average return. When considering the Assault ALE game, compared to the standard risk-ncutral policy based on maximizing expectation, better returns are obtained with more risk-averse policy (0.05 expectile), while worse returns are obtained with a more risk-seeking setting (0.95 expectile). This suggests that for this game, controlling actions in a more pessimistic manner is a better stratogy. This should also be typically the case in life-dependent games. The choice of the expectile level impacts performance by influencing the exploration strategy but the effect of exploration also depends on the task, which may tolerate not very risking options.
17

---

Published in Transactions on Machine Learning Research (12/2025)
Mm standard
800
MMD expectile 0.95
MMD expectile 0.05
700
600
500
400
300
200
1e7
Figure 4: Gaussian mixture representations with the MMD energy distance. Effect of the risk-aware expectile policy on the ALE Assault game: Test return averaged over 500k testing frames for 47M training frames.
C.3 Multi-Objective Applications
Designing a reward function in RL can be challenging, as it must encapsulate and balance potentially conflicting objectives. In the classical RL setting, the agent observes single scalar reward after interacting with the environment. This scalar reward implicitly reflects the weighted importance of different goals, guiding the agent to learn policy æ that maximizes it. However, once the policy is learned, it becomes difficult to disentangle and prioritize the individual contributions of each objective. This limitation occurs because the reward function does not explicitly represent. the separate impact of each objective on the outcome.
Multi-objective RL (Roijers al. 2013 Hayes et al. 2021 addresses this limitation by explicitly representing and ordering each objective in vector of cumulative returns, allowing the agent to learn single policy that optimizes all objectives. Thus, action selection, guided by Pareto dominance, can be formalized follows:
a" arg max P (vi, Va' (Z) 2 (z,z) aEd(x)
This approach enables the agent to select actions based on prioritized sub-goals without the need for re- training the policy each time one objective must be emphasized over the others. Recently, some works have explored paramctric representations of returns (see e.g. Cai ct al. (2023): Wiltzer et al. (2024)), opening up the possibility of representing multidimensional distributions suitable for multi-objective settings. GM are good candidates. Indeed, multivariate risk measures are well-known (Charpentier 2018) and we have seen two types of probability metrics (Jensen-Tsallis and Mixture- Wasscrstein) that are tractable with multivari-
ate GM. This metrics could leverage the powerful ability of DRL represent aleatoric uncertointy, allowing different straegiess to handle it in the multi-objective RL setting.
18

---

Published in Transactions on Machine Learning Research (12/2025)
D Proofs of main results
D.1 Sufficient conditions on D
Bellemare et al. (2017b) and [Bllemaree et al. (2023) introduced the following useful properties of D. D is said to be ideal D satisfies the following to conditions,
(SI) if is independent on X and Y + D(X,Y) (S) considering (0, 1). 3c VX,Y, D(AX, x XD)(X,)).
If in addition D is p-convex, meaning (see Definition 4.24 in Bellemare et al.
2023
Definition 4. Given > 1, the probability metric D is p-coneer if for any E (0,1) and distributions n. 12, 1/1 1/2 E P(R), we have
Da (1-a) + an22 s aD*(m, + (1 a))
then Theorem 4.25 of Bellemare et al. 2023 states that (ST) and (S) imply that, for any two return functions Th and 172,
D(T 71, T 72) S D(m,,
Note that the assumptions in Theorem 4.25 of Bellemare al. 2023 the fact that D is regular, which is equivalent to satisfying (ST) (see Definition 4.23 therein); and that D is c-homogencous which is (S) but where the inequality is replaced by an equality. However, it is easy to see from the proof of Theorem 4.25, that the inequality given by the (S) condition is enough.
D.2 Proof of Theorem
Proof. The proof is straightforward combining Theorems and i that show respectively that T is con- traction and that the projection is non- non-expansive. So that for two return function 7i, and 722
MW2 T n,, Wirw? 7 12)) S MW2( m. TT S 7M 72).
D.3 Proof and specifications on Theorem
Proof. We assume [[w( oo, although inequality [ can still makes sense if the right-hand site of ( is infinite. With na the PDF of A and A independent
X and Y, the PDF of X A (resp. Y + A) is the convolution nA 77x (resp. 174 ). This latter condition is also important as it prevents to use inequality 1 with X set to X + 4, Y set to Y + A and A to A, which would lead to the opposite incquality and then to an equality in 1 We thus obtain, (A + X, A I|( TY) * m Hence using the Young inequality (Bgaahhvv 2007), get (1] follows,
(T + X, A
| my
112 11 IIall S (),
Then, using (xx(x)= - (A- -2), obtain
-== (m)
(5)
from which 2 follows. It is straightforward to see that JT does not satisfy (S) according to Indeed,
applying 3 with w 1, leads to (AX, AY) ) = -- (X,, Y). So that if (S) was satisfied for JT
this would mean x 2 for some e > 0 and À 2 1, which conflicts with the assumption (0, 1).
Furthermore, when considering w(x) x2, inequality in the paper leads
JT,2 JTx²2(AX,XY)
AJT
19

---

Published in Transactions on Machine Learning Research (12/2025)
For E (0,1), 1), it follows that condition (S) is satisfied with 1. In contrast,
implics that
JT X, 2T22 Y)
If [E[AA > 1, this may prevent (SI) to be satislied since the Young inequality is optimal.
2 For instance, if A is constant, A = a, with > max (0. -(,((1)) it comes
/ (xxx(x - a) (m - dx) / (mm())))x)))m((())))))))))))))))
JT X
which leads to
D.4 Proof of Theorem 3
Let us first note that the result is easy check if V[A] 0. In that case A is a constant random variable and for any scalar a, we can show that since na+ (x) (²x) then for 7 E (0,
Jazz(a a =
= > - (X)).
More generally using the mean/ variance decomposition A E[A] + V[A]u where U is a standardized random variable with [EUU] = and V[U] 1, it follows that
= JTi2 (V[4 + 7X, U(A +78).
where the right-hand side is independent on the mean of A.
Then, if v[A] + O, the distribution PMip//o + 6,, where 8n indicates the Dirac mass in 0. Using that the convolution operator is bilinear continuous operator (owing to the weak Young inequality) and that dg p p d0 p. deduce that when V[A] + 0 then
JT 2 + + VAA + + = -INAAAj)A * (14x - -1,x)
11(17x - =
More formally, let us decompose the difference
= - -T
+ -
The second difference is J'i,,(7X,,1) 2X, Y) ( 1),TTr,(XXX)) which is strictly positive if E (0, 1) and 2T, Y) + 0 or equivalently 1xx # ml. For the first term, using continuity,
Ve 0,3k' 0, that [[A] | 7x, -7aa7,))) se.
Choosing < ( - 1)JT7,2(XXX) and maar = e' ends the proof.
D.5 Proof of Lemma I
Let A be independent of X and Y and all these random variables distributed according to mixture of k k where represents in turn X,Y Y or A and
distributions in C, denoted respectively by π = Eh= o E E C. For Z X and Z =,, by distributivity of convolution and summing stability of C, Z + A also follows a mixture of elements in L given by,
2 K
k K
Σ
Σ
Z
(1%
e,a
A ki
12+A = 72 * 7
*
t=1
i=l
20

---

Published in Transactions on Machine Learning Research (12/2025)
Let w (waa & [K* E [KY] be discrete distribution. For E [I x)), the marginals of w are x> and x.. For such a w, define the discrete distribution ü 10= (wrni k E [K ]. E [KT], E [K*]) where,
Whii = गे ii=
= 0 otherwise
The set of such distributions is denoted by
IT' {w, s.t w E IT X n² ,²)).
For an element in II', the marginals are respectively n X+A and x + since,
kY KA
KY
Wkt no ò(i=)} =
Whi
"k
X X+4
Wki
1=1
and similarly,
K KA
xx
Wkl x5 A ](==)} zj
m. Y Y+A
Wilt
mij
It follows that II' II( +AA).
Let us first prove the (SI) property. By definition of MW D.
D.C.
MW (X A, + A) MWP
D.C A
min Σ
Whasy (P ex+A) 19
wEIl(x X+A x k.li
Σ m
min
Whl
Using the (SI) property of D, we have for MW D.C: that is,
(Y+ S D(eg, , from which we deduce the (SI) property
ki
MW 1x + A,Y + A) <
min
DP (( = MW (X),))
wel
k.1
For the (S) property, for all X > O, we have m)2(x) = E T l/xx with E2 = PAa since is stable by scaling. Thus, using the (S) property of D, we obtain
MWP, (AX,AA)
min
Σ
Whs DP(EA ,ea
kdl
min wEll(a (x)
Σ
Wki N D2 = X2 MWP,(X,,)
This achieves the proof that MW D2C is ideal.
D.6 Proof of Theorem
As mentioned in Section D.1 Theorem 4.25 of Bellemare et al. (2023) can be used to get the desired contraction result. Note that although Theorem 4.25 of Bellemare et al. 2023 is stated for probability metrics, it is easy to check that the proof does not use any particular property of probability metrics so it is still valid for quasi-metrics or discrepancies.
Thus, as Lemma above implics the (S) and (SI) conditions, it remains to show that MW D.C. is 1-convex. This follows from The convexity of the discrete Wasserstein distance, which we prove below for completeness.
21

---

Published in Transactions on Machine Learning Research (12/2025)
Let a E (0, 1). Let for i= 1,2, ' = - Fuca and v/f Ei=1 T te be four mixtures distributions with elements in C. By construction, distributions of the form T (1 a) ry. are also mixtures of elements in L. More specifically,
K
K Σ
a th + m/
Σ
a TeR O + +(1 - a) the 01 =
tul lux.
Morcover, defining
-= {(a I5K + (1 a) 1Kk with w E II( and w' E 1(11,12))
have ñi [I( f2). It follows,
K1+K; K2+K;
MW D. 7n + (1 a)) 1/ a 12 + -1- a)) 1/22= = min #EIl(f),
Σ
wkl waa DCis, E2))
KitKi Ko+K; Σ
I min weth
Σ
Wer B2i)
min wEll(mT
Oowk,1 D(ikk; B21)
k=ll=1
K KS
ΣΣ a w/ i 22)
min
w'Ell(T
k=1/=1
S <oMW (o0n,
MW (
D.7 Proof of Theorem 5
The goal is to investigate the reliability of performing stochastic gradient descent with respect to 0 over MW2 loss. In practice, we rather equivalently consider the square of the metric, MW3 to avoid fractional exponents. We thus consider the estimate VoMW? (fM mo) the gradient VgMW3(nx no) and show that is in general biased. Write X ) me[a]]- [Xm]me[a]] Recall that [[ x²) denotes the set of discrete joint distributions w (Wke, & E [K] [K']), whose marginals are (TI,...,, TT) and n' === ( Tk.).
11(T, 7') ww (wke)k.e,sth s.t. Σ Whe = T, = Σ Wke Tt = Σ wket
k.E
Using Definition of the squared MW2 metric, we denote by w (X) (wim [K], m [M]) the optimal coupling defining MW3(ffr m18) and similarly by w (wās k [K], E [Kx]) the optimal coupling defining MW3(1x, no).
We can then write,
MW2(fM = Σ When (X) ((Xm -M,,0) +z2
(6)
Σ Troluho oh.o) + E
Xx2 /
Σ
wkm 88)mmllts
(7)
k,m
(8)
Then,
E VoMMW((mumno)) Vo
Emoolune + oko0
2V.
Σ E Elwinn (X)Xmm) me.
(9)
k.m
Vo Σ TH. + 0h00
2V
Σ M. E Wkm (8) (8)
(10)
22

---

Published in Transactions on Machine Learning Research (12/2025)
While the target distance is
MW (πXx m10)
Ennte The (WHE, mx,0)++ + ox.0)2) k,
(11)
from which we can derive that
VeM ne)= Vo Σ Th ( +oho) 2V8 Σ @helpexxR o2.x
(12)
Tt follows that the difference E VoM (fM,
VoMW3 (1xx no) is equal to
2V 8
Σ k,C
WhelMe + + - 2Vo Σ HK Σ wpm (X) Xm)
(13)
which in general non zero. Standard stochastic gradient optimization does not come with standard guarantees when applied MW3 loss. However, several solutions using biased gradients have been investigated and may be possible without too restrictive assumptions; see the recent review of Demidovich
et al. 2023
D.8 Proof of Theorem 6
Let us consider the projection on Mr the space of GM with fixed number K of components. The proof consists mainly is exhibiting the subset W Recall that M denotes the set of all GM. The goal is to show that when considering the MW3 metric, the projection on Mr is non expansive. This projection is defined for any GM distribution 7 E .M as arg m MW3 (17, 77). As the univariate MW3 metric is defined with the Euclidean distance as cost function, we can first make use of standard results to explicit this projection.
Consider for some arbitrary L N, 1 Ei TeN of) and n/' Σk (, o(2) two univariate mixtures in My and Mи respectively. Denote π (TI...., TL), n' (T) Tk) and the set of couplings between ı and x' by
11TTxx' = {w- =wan)e s.t. Σ
Wir = 1,
Σ k
Weky T = Σ werh.
t,k
Recall that, for any random variables X and Y, the characterization of R[x Y] as the unique T.2 projection of X leads to
E[(X -Y B[(X E[X Y])
(14)
For w = (wexeex E [1T, x'), we then consider two variables X and so that (X,Y) ~ âLK where ÍLKK = Σe, Wer (enw))) For all (r (ynta, w E II(T,T'), inequality writes,
Σ E.k
Wwel yk)²
Σ t.k
Wilk
Et
Σ
Wirk
(15)
T
Wet with equality if and only if y = Er Then using that MW3 n/) is defined the minimum over w E 11( T²) of
Fy for all k.
Aw,n.
Σ
wa (lue 24, (a: -02)2)
e
23

---

Published in Transactions on Machine Learning Research (12/2025)
we replace and y in (15) with successively the means and then the standard deviations of the two mixtures components to get,
wee ((--
2
2
n/) 2 Σ
WER 1
Me
+
WER
De
k
e'
It follows that,
(mee
2
)
2
Mw2(1
min
Σ
wek
Σ
wek de
wEll(T,TT)
T
t,k
and then that,
2
min MW3(m. 2 min d'EMR
min (El)(
Σ E,k
(ne
Σ e'
Work
+
Werk
TH
T
Then denoting by w² (1) the coupling where the minimum above is reached, we define is Ee
Fx(m)
pe:
Tx( Ee or, (n) Ee wer (1) and consider the mixture (w) E Mr, Tw
(1)
( (z(ा)
Σx (() (()))))))) Thus we obtain the projection operator WWwg Rho that satisfies
(w "w
min MW2 g'EMR
In addition, (m) is a mixture whose expectation the same that of 7) iee. E so that E|II" ((x, a)] = E[Z(x, a)] if the PDF of Z(z, a) (resp. IT Z(x, is v (resp. II"" n) so the projection is mean-preserving.
Similarly, it easy to show the non-expansion. For two mixtures n and r/' and w e 1l(,, T'), we consider a random vector (Xo, X1, Yo, Y1) ~ Eig. (i..k,e such that
Vi
Σ
üij.ht (n):
k.
wk; e
Σ
Wi.j.h.f = wke(!))
f.j
Vi,k,
Σ
Wi.jkk. = Wik
j.8
with Xp (resp. Yo) independent of Y1 (resp. X1). Using Cauchy-Schward incquality, we obtain
E[X, Yo ii) (n/, vf)a < (X,, Yn)² (X1, Yi) (n/) v/)l
Moreover, since we have independence, we get
E E[X Yo (X1, Vi) (1/,, = Exx
X1 = 2 -E Yi víl.
so the following inequality holds,
wiy(n) (n)
wTr(n') Te())²)
Vj.4 e,
Σ
Σ
Uk
5 Σ
P((Xo,
(X1, Y2) = - 2x
Hence we obtain
2
w wiy(1))
w
EP((X,)) = (2') v/6))
Σ
Σ
(π)
24
#
wish wx - y
(16)
3.
i.k
24

---

Published in Transactions on Machine Learning Research (12/2025)
for all y, w II (TT, x²). Since (1) (222, E (((((n),((()))))) in 16 by the means and standard deviations of the two mixtures components to get
replace the rz's' and ys's'
MW2 (TT,,, Tow, S Mw2(n,1)
thus krw, is a non-expansion with respect to 2W2-
D.9 Unbiasedness property
If the sample estimates of the projected distributional Bellman operator are unbiased, then it easy conclude the convergence of the associated TD algorithm. This is how the convergence of the categorical TD algorithm is typically shown (see Bellemare et al. 2023 However, this is not possible in our case.
Indeed, deline for all return function 7 and for all E
7 E a()aa)<(a x).
aEA
p²(x) Σ a(zxa)
aEA
(= x)) = Σ P(: ((.12,a)x(0 ()).
aEA
The standard property of unbiasedness defined in p. 166 of Bellemare et al. (023)), which would amount to check the following equality, with our projection Hxw, defined previonsly,
[ ((r + +Id) ((+
Vx E X.
1))rrd'= = Haw, Tz (x)),
is not satisfied, as is also not the case for quantile temporal difference (see example 6.3 of Bellemare et al. h023)).
In fact, we show instead that
Je Tiwws(rr ((d) (1))) w))rdd' = T "
(17)
Thus we indeed have a bias since the result " nnt differs from 11TH T"
Proof of [17 To compute 11 ((r + (ld) (x')), we use that w ((r + yld) defined as
2
(105) ))
(yee (mee + +))= +1(0 Σ
w' ((d++
min Σ
WEE
Tipe Tue
Σ wek
wiz Σ.
Work
WEL E.k
Wist
It follows that w² w(r + yId) can be chosen the same as (n) and thus that IT ((r + 7Id) (1/))
(r (r+4Io ++I+ II" (17 (x'),, which leads to (17 as desirod.
D.10
Generalization to the multidimeensionll case
D.10.1
General metric properties
We define the F2 probability metric between two multivariate Gaussian distributions in dimension d, g1 Na(H1, Et) and g2 =Na(p2, Σn) as
F2 (2) = || - 2 + Tr ((x) -
x²)")
25

---

Published in Transactions on Machine Learning Research (12/2025)
This semi-metric is upper-bound of the 2- Wasserstein distance making it appealing alternative with projection easier to manipulate. Indeed, we have
Tr( - - - + + 2ΣドE3) Tr(Ei - E2
using the classical inequality Tr((AA"))) 2 Tr(A) with = - Moreover, it is ideal according to the following thcorem g
apply Thcorem - to get that T is contraction mapping with respect to FR,Me if the reward is a multivariate Gaussian mixture for each state-action pair.
F2.
Theorem 9. is ideal.
Proof. It is clear that F satisfies (S). In addition, for all i E [1,2] and o' = -Na(1', E'), we have g' * * Na(u + nl, E. + 2") so we get
(g 91.9 92) F (91. 2T' ( + ト -E E' 111222299 +))))
(18)
Moreover, The function o: (M,N, K) + T N KM:) ) is jointly concave in (M,N) E S. (see Lieb 1973 Corollary 1.1) where S+ is the space of positive semi-definite matrices so we have
1 (6E,, } (I,, E2 2 ) "E (EI,, I2,1,
i.e.
( Tr (E)+ (E2+ -Tr ( (22 E ())))))()+ Eン}²)
Tr Tr ( = Tr 3' ズ
Using this inequality and 18 we obtain that F satisfies (SI).
D.10.2 Projection
Similarly to the d case, using that MW2 (. 1/)) defined as the minimum over w E [[(,, of
A(w, (.,,).))) = Σ
EEE - (E))
Wikk
1
can replace æ and y in 15 with successively the means coefficients and then the covariance matrices coefficients of the two mixtures components to get,
2
)
2
A(w, n/ 2
Σ Wekk
Σ
WHR
Me'ss
+
D
We'k 1/2 TE
e,k
i,j
Following the same methodology in Theorem
6 define w" (17) minimiser of the previous expression, (x) Ee wpx(()) and we get the following projection
(n) = Ee win
(n)
1/2
(n) = Er (ey (1)
1 pe,
2
H and 1 Σ #x (f)))) Ex()) MK.
As previously, replace the 2z's' and ys's' in 16 by the means coefficients and covariance matrix coefficients of the two mixtures components to get
MW2
(T MW an T T
11 S MW2 (1.
F.Ma
thus I ra
is a non-expansion with respect to MW F.MA.
26

---

Published in Transactions on Machine Learning Research (12/2025)
D.11 Expressions of MMD distances for GM
In the MMD case, Theorem 10 provides closed-form expressions for number of kernel choices, when the compared distributions mixtures Gaussians.
Thcorem 10. Let X E
Ek= (P ofw) and Y E mak Nu 2 two independent random variables distribuled GM. We have the following closed- foseal-form expressions:
Iap (X,
Σ R,f
TIATTG (Nofk,, )ork,e)
hy bbe beb (X, = Σ
711
(1 -
TIkTTe
Th. Vi
R.B.
(iR. (ōkee)
Eliien (X,Y) =
with F(x, v) = e - ==x111)(101)
I (-
where G (2r, v) F(r,y) + F()x,y)
11
V(x) x22FFx(0.)))) - 1) + (2; and oh.2 = =s + oze.
Px. = Pik - p2t.
Proof. For any kernel k,
K1 K₂
E(A(X, =
T E[k( Xx. Ye)],
k=1
where Ng (() Ye Y(~N(p2,0f))) We only need compute expectations for two Gaussian variables.
Laplacian kernel. Since X₂ - -Y, ~ N( - 122, of + +ole), , for all 7o 2 0, we need to compute 10e-11 where is univariate Gaussian distributed. For ~ N(u,0*)
toc
102 nz (x)dd
e 2
1900
1z 700
e
1990
z| (r))dc
200 #t
2 2
( - -
=
e
470
FN(0.1)
0
From which, deduce that
Ele 100
c0/ml (z())d
+00
c 70 (1z(z) + nz( (x)(d
e 70 (1z(x) + 17 z (z)
Hence,
Ele -ol Nx -Vl] =6
2,02
-mpue PN((0.1)
juke
cāāe
Tze
(n troper Fx(0.1)
(Fike To
+ e
The
27

---

Published in Transactions on Machine Learning Research (12/2025)
Gaussian kernel.
We also have
Nx 12
=V22T11 / Na 10,7))
Ele
2
juke, oke)
= =V2TM>(MM 10,7 + kke)
71 Vnf + oke
2079 12
Energy kernel. Using that
B = (1- - Flz dx [ (Fz(x) + F-z(z)) de
(
(19)
and considering the standardization with U (Z -p)/o NO,,1), 1), comes
B Z = ( Fx(x)) dr + | F-u(z) de
pulo
Integration by parts leads to
Fo(x) dx = uu/==u/ I N(I di N(;;0,,
-uFu(u) +,(,0,
Fu(x) dx = -uFu (-u) + 0,
= --u + 0,;
and E [|] (u(2Fu(u) 1) 2N, 1)) ov),, where V(u) is stated in the theorem.
D.12 Equivalence between the Cramer and energy distance
Zhang (2023) implicitly provides proof that the Cramer distance is proportional MMD that different from the already well-known geometric proof cited by Nam et al. (2021) To highlight. this, let us consider the following lemma.
Lemma 2. Let Y,Y be two independent random variables. We have the following formula,
Vr E / - FF+Y(=))d [ (x)x
(20)
where Fx is the CDF of X.
Proof. We adapt here the proof used the particular case of Gaussian mixtures in Zhang 2023 we have
Indeed,
82
θ θ (1 Fy (x r)) Fx (z) dx Or dr
f
= (x
0/2
R
(x)x))xx(x+ + r)da= mr(x))x + r)dæ
dr
(z) (n-r
Hence there exist constants C1, C2 independent of r such that.
Cer + f 1" (n-Y (t)d.
28

---

Published in Transactions on Machine Learning Research (12/2025)
By taking the limit - -oo, it comes C1 C2 = 0 and thus
J (1 -x+
= S (n-y * (t)dttd = - Fx Fx-y(x)dz.
Using this lemma, we can prove that the Cramer distance C is proportional to MMD k... is well-known but we provide a proof different from the classical proof of Szekely (2002)
if = 1. The result
Theorem 11. Let P. Q two distributions, we have C² Q) MMMD Kmn (P,Q).
Proof. The Cramer distance is defined by (²(P.Q) (r(()) (o(x)/2dr. Then,
1 [Fp()) Fo(x)
Fp(x)( (z) + /
Fp(x))do
/ Fr(x)( FF(p))dd FQ(x)(1 - Fo(x))dzz
JR
For some independent variables (X,X,Y, with X,X ~ P and Y,Y Q, Lemma 2 leads to
1 (1)(()-FQ(()) 1 Fx-r(x)de [rd Fx-x(x)de S Fx-x(1)de L (
Using formula 19 and that Fx-8= = Fx-x and Py-p - by symmetry, it follows,
(r(()) = = yJ E X - Eljy Yy MMD N... (P,Q).
E Stochastic Approximation solution
Although most DRL solutions are based on a TD learning principle, the approximation of returns by Gaussian mixtures opens the way other approaches to access the targeled fixed point. Stochastic approximation (SA) is a popular approach for solving fixed-point equations where the information corrupted by noise. Such is the case in RL and this technique has theoretical advantages over TD-eearning (Chen al. 2020
More recently, it has shown tremendous results in DRL using Gaussian mixtures (Zhang al. 2024
The principle is simpler than TD-learning. The online network is updated using point-wise stochastic approximation,
I a) = (1 (1, a) + 8(r + TT) T
with a transition sample (x, a, r. x', d'). As the right-hand term a mixture of two distributions, it is natural use mixtures as representations, as mixture of mixtures is a mixture. However, it impractical to use directly the resulting mixture since the number of components increases exponentially. The solution of Zhang et al. (2024) uses an EM estimation to approximate the resulting mixture by fixed number of components. This is crucial in their approach which then optimizes loss that can only be computed between mixtures with the same component numbers. One advantage of the metrics we propose is that this approximation is not necessary as they can handle mixtures arbitrary sizes. It follows the possibility to design new simpler algorithms whose global principle illustrated in Algorithm 2 (sg is the stopgrad operation). In practice, this would require refinements out of the scope of this paper.
F Experimental setup and hyperparameters
All experiments were run on our local cluster. The main experiments on Atari-5 were rum on NVIDIA L40S GPUs. The ablation study Atari-3 carried out NVIDIA A40 GPUs. Additional experiments
29

---

Published in Transactions on Machine Learning Research (12/2025)
Algorithm 2: SA training for 1 episode
while environment is not terminated do
Sample at ~ Ti -g-good (- xx,)
Get 2+41 ~ P(- Tt, ae) and r1 ~ R(2,,a)) ai)
Thew + (1 (7 (Tte at))
Thew Thew + Bsg( + +T) #7 (t+1))
Minimize (n (2t, az), Nuew) over network parameters
1-141
shown in Figure were run on NVIDIA V100 GPUs. We mostly used the Dopamine (Castro et al. 2018 hyperparameters recommandations (Table 3 but we had to use gradient clipping to stabilize the training. To guarantee fair comparison, we used the same hyperparameters for all methods for each environment and we selected the gradient clipping by selecting for each environment the highest possible giving stable training for all methods. This leads in our case to gradelip = 100 for Asterix and Bowling, and gradclip = 1 for Qbert. As regards the GM representations, the number of components was set to K = in the Atari-5 experiments, while K was varying in (1,3,5, 16) in the ablation study. To compute metrics, the MW2 was implemented using the POT package Flamary et al. 2021 and for MMD computations, we used the same setting as Nguyen-Tang al. (2020).
Table 3: Hyperparameters used in experiments
Hyperparameter
Setting
Sticky actions probability
0.25 (stochastic case)
Discount factor (7)
0.99
Frames stacked
4
Mini-batch size (B)
32
Replay memory size
105
Online network update rate (A)
4 steps
Target network update rate (T)
10 steps
Initial exploration (c)
Exploration decay rate Exploration decay period
10-2
2.5 x 10 steps
Test exploration
10
Environment steps per itcration
2.5 x 10 steps
Starting step
5 x 10a steps
Adam hyperparameters
31 decay
0.9
B2 decay
0.999
Eps
10-?
B
Learning rate
10 10-5
G Additional experiments
Table g and Figure 5 provide results for an ablation study, varying the number of mixture components K E f1,3, 5, 16) the deterministic case with the Atari-3 subset of games proposed Aitchison et al.) (2023). Figure 6 provides additional illustrations on 3 other Atari games, in a deterministic setting, plus one of the games in stochastic setting similar to that of Section
30

---

Published in Transactions on Machine Learning Research (12/2025)
Table 4: Ablation over the number of components K, in the deterministic case with metric MW2. Normalized scores on Atari-3 using the weights given in Aitchison et al. (2023), for varying K.
Environment.
3 K
K 16
Deterministic
43.80 3.63
44.40 + 13.20 48.37 + 13.36 31.82 + 19.98
K-1
K-3
K-5
K-
16
Battle Zone
Name This Game
Phoenix
Azari-3
25000
0000
000
40
Recums
15000
3000
4000
10000
2000
1000
3.5
1.0
1.5
1.0
1.5
1.0 1.5
2.0
1.0
1.5
2.5
Training frames
Training
Training
Training frames
Figure Ablation over the number components K, using the Atari-3 games the deterministic case with metric MW2. Moving average return, over 500k frames, with respect to the number of training frames. Curves averaged over 5 runs with shaded areas representing standard deviations.
300
200
(a) Asterix
(b) Bowling
(c) Qbert
(d) Qbert stochastic
Figure 6: Comparison of MW (green) JT ( MMD (blue) and MMD Kmazray (red) metrics for Atari games in deterministic (a.b,c) and stochastic (d) environments. Moving average return, over 500k frames, with respect to the number of training frames. Curves are averaged over 5 runs with shaded areas representing standard deviations.
31

---

Published in Transactions on Machine Learning Research (12/2025)
H Computation times
Tables and 0 present respectively computation times for the Atari-ō experiments and the ablation study on K for the Atari-3 games. Computation times tend to increase with K.
Table 5: Average computation overheads and standard deviations (in hours) on Atari-5 for the different compared metries in deterministic and stochastic environments. The shortest times are indicated in bold characters.
Environment
JT
MW2
MMD
MMD
Stochastic
46.12 + 0.27
166.45 0.42
46.38 + 0.16
45.71 + 0.06
Deterministic
44.45 +: 0.57
166.25
0.65
46.29 1 0.47
43.68 1 0.33
Table 6: Average computation overheads and standard deviations (in hours) on Atari-3 when using the MW2 metric in deterministic environment and varying the number of mixture components K. The shortest times are indicated in bold characters.
Environment
K=1
K =
K
Deterministic
16.02 + 0.18
62.05 0.31
135.46 0.53
32


Reviews:
[{"review_id": "cVXCgooygV", "reviewer": "Reviewer_ynDy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- Strong theoretical contribution: Clear identification of the failure mode of JT and rigorous proofs of contraction for MW with GM projection.\n\n- Unified analysis of multiple metric classes (JT, Wasserstein-type, MMD) under the GM representation, including closed-form MMD formulas and new connections to the Cramér distance.\n\n- Practical tractability: The GM framework enables efficient risk measure computation (e.g., expectiles, quantiles) and parameter efficiency relative to other DRL methods.\n\n- Empirical results are consistent with the theory, especially the dependency of JT performance on environmental stochasticity.\n\n**Weaknesses**\n\n- Limited experimental scope: Only three Atari games (plus one stochastic variant) are used; this narrow set limits the empirical generality of the conclusions.\n\n- Hyperparameter confounding: Atari experiments involve many factors beyond the metric choice, making it harder to attribute performance differences solely to the metric’s theoretical properties.\n\n- Lack of small-scale controlled experiments: No minimal finite MDP or grid-world example is provided to directly illustrate the contraction/non-contraction behaviors under different metrics.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a theoretical and empirical analysis of metrics for distributional reinforcement learning (DRL) when return distributions are represented as Gaussian mixtures (GM). It identifies limitations of the commonly used Jensen-Tsallis distance, proving that it may fail to satisfy contraction properties in low-noise environments. In contrast, the paper introduces and analyzes the Mixture–Wasserstein metric, showing that it inherits ideality from its component-level distance, admits non-expansive projection operators in the GM space, and thus ensures contraction of the projected Bellman operator under mild assumptions. The authors also revisit maximum mean discrepancy (MMD) metrics, providing closed-form solutions for GM inputs and showing theoretical connections to the Cramér distance. Experimental results on a small set of Atari games (deterministic and stochastic variants) empirically support the theoretical findings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add a small-scale, fully controlled MDP experiment.\n\nThis would strengthen the paper but is not critical for acceptance.\nA finite-state MDP with tunable reward noise could vividly illustrate Theorems 2–4 and 6, showing the precise conditions under which ​JT fails and MW2 succeeds.\n\n2. Expand experimental diversity. This also would improve empirical robustness.\n\nInclude additional Atari games or alternative benchmarks (e.g., simple continuous-control tasks) to demonstrate metric robustness across domains.\n\n3. Add conceptual diagrams or figures to illustrate theoretical concepts. It would significantly improve readability and accessibility.\n\nConceptual diagrams would make the theoretical results more accessible to readers unfamiliar with the underlying metrics, bridging the gap between formal proofs and intuition.\n\nExamples for Figure Content:\n\n - A schematic showing two narrow Gaussians after Bellman update to illustrate JT non-contraction in low-noise settings.\n - An optimal transport mapping diagram for MW between Gaussian components, highlighting why it preserves contraction with GM projection.\n - A visual of MMD (energy kernel) comparing CDF shapes to convey its sensitivity to distributional differences.", "broader_impact_concerns": "No significant ethical concerns. The work is theoretical/methodological, with potential benefits for stability and interpretability in DRL. No obvious new risks are introduced.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HeV4ZbAxSt", "reviewer": "Reviewer_yYNN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper advances theoritical understanding of the distributional reinforcement learning, while there have been few influential papers in this field, there are lots of unstudied aspects, this paper addresses some of those.\n- The paper provides a detailed theoritical comparison of various metrics.\n\n**Weaknesses**\n-  The experimental evaluation is minimal, which limits the evidence of the claims/ observations made in that section.\n-  The section on generalization to multidimensional rewards also is minimal.\n- There are many instances where sentences lack technical precision and are occasionally vague, particularly in the abstract. In general, the presentation needs substantial improvement. In several places, it is challenging to distinguish between existing results and the paper's contributions, primarily due to the presentation. Please see the \"Requested Changes\" section for more feedback.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper advances the theoretical analysis of distributional reinforcement learning, especially on the use of Gaussian mixtures. The theoretical results in the paper provide a comparative understanding of various metrics. The paper also provides limited empirical evaluations and few results on generalization to multi-dimensional rewards.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- While most of the theoritical results are supported by evidence (proofs in this case). The empirical study lacks evidence of claim / observation made due to limited nature of the study. \n\n **Empirical Evaluations**\n\n- It is strongly recommended to provide an exhaustive empirical study that would add significantly to the contribution of this work. Particularly, a more exhaustive study is warranted to make contrasting comments such as \"It is also interesting to note that no environment shows a real advantage of MMDkmixrbf over MMDken, which contrasts with the results of Nguyen-Tang et al. (2020).\" \n- There is no reason to exclude results. Please include them in the appendix, along with the corresponding observations. \"The results for the Gaussian kernel are not shown, as it can be seen as a simpler version of kmix rbf, and we observed that the Laplacian kernel behaved similarly to the Gaussian one.\"\n\n**Presentation**\n-  Abstract: It is recommended to rewrite the abstract in order to make it more precise and clear. The following are some parts of the abstract that are imprecise: \"on previously unstudied metrics\" and \" prove a convergence result for a related dynamic programming algorithm.\"   \n- Similarly, there are many instances of vague and technically imprecise sentences in the paper. The following are a few examples. \n  -   \"The distributional Bellman operator is *likely* not to be a contraction mapping.\"\n  - \"the most significant reference\"\n  - \"have obtained outstanding results with\"\n-  Please cite the corresponding prior works here. \"These metrics have not been studied in prior work in GM-based DRL.\"\n\n- Assumptions should be clearly highlighted and the consequence (which results are still valid or not) should be discussed in more detail. :\"In what follows, we will thus assume that $MW^p_{D,L}$ is a metric although some results are still valid if  $MW^p_{D,L}$ is a quasi-metric\".\n\n- Inconsistent use of upper/lower case for related notations, e.g., from $(x, a)$ to $(X^′, A^′)$.\n\n**Typographical Errors**\n- non-monotonous  $\\to$ non-monotonic (page 2), caracterized $\\to$ characterized (page 8), caracterization $\\to$ characterization (page 22),", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DwvQKhgiij", "reviewer": "Reviewer_dzzu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Strong theoretical foundations: Contraction proofs and projection analysis are rigorous and rely on clear assumptions.\n- Novel metric ($MW_2$): Tailors optimal‑transport ideas to GM DRL; the discrete cost makes it tractable while retaining favourable geometry.\n- Closed‑form MMD gradients: Avoids Monte‑Carlo estimation and biased gradients that plague Wasserstein‑based DRL, broadening the toolkit for practitioners.\n-  Experiments align with theory: deterministic vs. stochastic ablation corroborates analytic claims.\n-  Clear exposition: metric‑property table and theorem numbering aid readability.\n\nWeaknesses\n-  Limited experimental scope: only three Atari titles plus one stochastic variant; no continuous‑control or multi‑objective tasks. No ablations or sensitivity analysis on the number of mixture components ($K$), metric hyperparameters, or projection variants.\n-  Computational overhead unreported: $MW_2$ solves a $K_1 \\times K_2$ optimal‑transport problem every gradient step. The paper does not report wall‑clock time or scaling with $K$. A comparison with the Euclidean‑parameter loss of Zhang et al. (2024) would strengthen the practical case.\n- Biased‑gradient issue remains open: Bias in sample gradients for MW₂ remains unresolved, complicating its use in standard SGD-based DRL training.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies parametric distributional reinforcement learning (DRL) when value distributions are represented as Gaussian mixtures (GMs). Its key technical and empirical contributions are:\n- Critical analysis of the Jensen‑Tsallis ($JT_{1,2}$) metric – proves it lacks scale‑sensitivity, so the projected Bellman operator need not contract, explaining instabilities that practitioners have observed.\n- Introduction of the Mixture‑Wasserstein metric $MW_2$ – a discrete optimal‑transport distance between GM components with closed‑form Gaussian costs. The authors prove that $MW_2$ is ideal (sum‑invariant & scale‑sensitive) and that the GM‑projection is non‑expansive; hence the projected Bellman operator is a $\\gamma$-contraction.\n- Analytic MMD (Maximum‑Mean‑Discrepancy) formulas for GMs – closed‑form gradients for Laplacian, Gaussian and energy kernels remove Monte‑Carlo noise from kernel‑based DRL losses.\n- Proof‑of‑concept experiments on three deterministic Atari games and a stochastic Qbert variant show that $MW_2$ and GM‑MMD outperform $JT_{1,2}$, with the gap shrinking as environment stochasticity grows—matching theory.\n- Extensions sketched for multivariate rewards (an F‑distance variant) and for stochastic‑approximation TD.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "To improve the clarity and rigor of the work, I suggest the following:\n1. Expand experiments.\n- Include more diverse environments from Atari games. \n- Perform hyper‑parameter sweeps per metric to rule out artefact‑specific tuning advantages.\n- Include multi‑objective RL benchmarks (e.g., MO-Gymnasium) to demonstrate Section 7.\n2. Report compute cost\n- Provide wall‑clock time per environment‑step as a function of mixture size $K$ and compare with an $L_2$ PDF loss baseline.\n- Discuss any use of entropy‑regularised OT (Sinkhorn) and its effect on contraction guarantees.\n3. Analyse optimisation bias\n- Plot learning curves for $MW_2$ with biased gradients vs. an unbiased control (e.g., analytic MMD) across varying batch sizes.\n- Relate observed behaviour to recent biased‑SGD theory cited in Section 2.3.\n4. Code release\n- Although the Appendix mentions a zip archive, a public repo with training scripts would aid reproducibility.", "broader_impact_concerns": "There are no immediate ethical or societal risks apparent from this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)