Studying Exploration in RL: An Optimal Transport Analysis of Occupancy Measure Trajectories
Authors: Reabetswe M. Nkhumise
Debabrota Basu
Tony J. Prescott
Aditya Gilra
Venue: Accepted by TMLR ()

Abstract: The rising successes of RL are propelled by combining smart algorithmic strategies and deep architectures to optimize the distribution of returns and visitations over the state-action space. A quantitative framework to compare the learning processes of these eclectic RL algorithms is currently absent but desired in practice. We address this gap by representing the learning process of an RL algorithm as a sequence of policies generated during training, and then studying the policy trajectory induced in the manifold of state-action occupancy measures. Using an optimal transport-based metric, we measure the length of the paths induced by the policy sequence yielded by an RL algorithm between an initial policy and a final optimal policy. Hence, we first define the Effort of Sequential Learning (ESL). ESL quantifies the relative distance that an RL algorithm travels compared to the shortest path from the initial to the optimal policy. Furthermore, we connect the dynamics of policies in the occupancy measure space and regret (another metric to understand the suboptimality of an RL algorithm), by defining the Optimal Movement Ratio (OMR). OMR assesses the fraction of movements in the occupancy measure space that effectively reduce an analogue of regret. Finally, we derive approximation guarantees to estimate ESL and OMR with a finite number of samples and without access to an optimal policy. Through empirical analyses across various environments and algorithms, we demonstrate that ESL and OMR provide insights into the exploration processes of RL algorithms and the hardness of different tasks in discrete and continuous MDPs.

Published in Transactions on Machine Learning Research (05/2025)
Studying Exploration in RL: An Optimal Transport Analysis of Occupancy Measure Trajectories
Reabetswe M. Nkhumise
@ uk
Deportment of Computer Science, The University of Sheffield, UK
Debabrota Basu
debabrota. basu@tinii..aaaj..ooooooo
Equipe Scool, Unio. Lille, Inria, CNRS, France
Tony J. Prescott
t.j. presccot@@sheffilld..oo uk
Department of Computer Science, The University of Sheffield, UK
Aditya Gilra
nl
Centrum Wiskunde Informatica, Netherlands
Department of Computer Science, The University of Shefficld, UK
Reviewed on OpenReview: https: opwnreviee........ net/ per/forum? id=pdC092NN8V
Abstract
The rising successes of RL. are propelled by combining smart algorithmic strategies and deep architectures to optimize the distribution of returns and visitations over the stato-action space. quantitative framework to compare the learning processes of these eclectic RL algorithms is currently absent but desired in practice. We address this gap by representing the learning process of an RL algorithm as sequence of policies generated during training, and then studying the policy trajectory induced in the manifold of state-action occupancy measures. Using an optimal transport-based metric, we measure the length of the paths induced by the policy sequence yielded by an RL algorithm between an initial policy and final optimal policy. Hence, we first define the Effort of Sequential Leorning (ESL). ESL quantifies the relative distance that RI. algorithm travels compared to the shortest path from the initial to the optimal policy. Furthermore, the occupancy measure space and regret (another metric to understand the suboptimality of an RL algorithm), by defining the Optimal Movement Ratio (OMR). OMR assesses the fraction of movements in the occupancy measure space that effectively reduce an analogue of regret. Finally, we derive approximation guarantees to estimate ESL and OMR with finite number of samples and without access an optimal policy. Through empirical analyses across various environments and algorithms, we demonstrate that ESL and OMR provide insights into the exploration processes of RL algorithms and the hardness of different tasks in discrete and continuous MDPs.
connect the dynamics of policies in
1 Introduction
In recent years, significant advancements in Reinforcement Learning (RL) have been achieved in developing
exploration techniques that improve learning (Bellemare et al.) 2016 Burda et al. 2019 Eysenbach et al,
2019) along with new learning methods (Lazaridis et al.] 2020; Müller et al 2021 Li 2023). With growing computational resources, these techniques have led to various successful applications of RI, such as playing
games up to human proficiency (Silver et al. 2017 Jaderberg et al. 2019). controlling robots (lbar et al.
2021 Kaufmann et al. 2023), tuning databases and computer systems (Wang et al. 2021 Basu et al. [2019). etc. However, there remains lack of consensus over approaches that can quantitatively compare these exploratory processes across RT algorithms and tasks (Seijen et al. 2020 Amin al. 2021 Ladosz et al.

---

Published in Transactions on Machine Learning Research (05/2025)
This is attributed to some methods being algorith (Tang et al. 2017 p. while others provide theoretical guarantees for very specific settings [Lattimore & Szepesvári 2020 Agarwal et al. 2022 Thus, comparing the exploratory processes of these eclectic algorithms across the multi-directionll space of RL algorithm design emerges as a natural question. Howover, the present literature lacks a metric to compare them except. regret, which is often hard lo estimate (Ramos et al. 2017 2018).
2022
This paper aims address this gap based on two key observations. Pirst, we observe from the linear programming formulation of RL that solving the value maximisation problem is equivalent to finding an
optimal occupancy measure (Syed et al. 2008 Neu & Burke 2020 Kalagarla et al. 2021). Occupancy measure is the distribution of state-action pair visits induced by a policy (Altman 1999 Laroche & des Combes 2023). Under mild assumptions, a policy maps uniquely to an occupancy measure. Second, we observe that any RL algorithm learns by sequentially updating policies starting from an initial policy to reach an optimal policy. The search for an optimal policy is influenced by the exploration-cxxloitttioo strategy and functional approximators, both of which impact the overall performance of the agent by determining the quality of experiences from which it learns (Zhang et al., 2019; Ladosz et al., 2022). Hereby, we term collectively the learning strategy and the exploration-exxloitation interplay as the exploratory process.
Contributions. I. A Framework. Motivated by our observations, we abstract any RL algorithm as a trajectory of occupancy measures induced by a sequence of policies between an initial and a final (optimal) policy. The occupancy measure of a policy given an environment corresponds to the generatnn dis- tribution of state-actions. Thus, we can quantify the effort of each policy update, i.e. the effort to shift the state-action data distributions, as the transportation distance between their occupancy measures. The total effort of learning by the algorithm can be measured the total distance covered by its occupancy measure trajectory. We provide mathematical basis for this quantification by proving that the space of occupancy measures is a differentiable manifold for smoothly parameterized policies (Section 3). Hence, we can compute the length of the occupancy measure trajectory on this manifold using Wasserestein distance as the metric (Villani 2009).
2. F.forrt of Sequenlial Leurning. In contrast to RL, if we knew the optimal policy we could update our initial policy directly via supervised or imitation learning. Effort of this learning is represented by a direct, shortest (geodesic) path from initial to optimal policy on the occupancy mcasure manifold. To quantify the cost of the exploratory process to learn the environment, we define the Effort of Sequential Learning (ESL) as the ratio of the (indirect) path traversed by an RL algorithm in the occupancy measure space to the direct distance between the initial and optimal policy (Section 3.1 Lower ESL implies more efficient exploratory process.
3. Efforts to learn that lead to Regret-analogue mindmisation. Regret is a widely used optimality measure for reward- maximiinn RL algorithms (Jaksch et al., 2010). It measures the total deviation in the value functions achieved by sequence of policies learned by an RL algorithm with respect to the optimal algorithm that always uses the optimal policy (Sinclair et al. 2023). We show that regret is related to the sum of distances between the optimal policy and each policy in the sequence learned by the RL algorithm, in the occupancy measure space. We can define an analogue of instantaneous regret. (al any one step during learning rather than cumulative), in the occupancy measure space, as the geodesic distance between the occupancy measure of the policy at this step in the learning soquence, and the optimal one. We find that not all policy updates lead to reduction in this analogue of immediate regret, and thus define another index Optimal Movement Rotio (OMR) that measures the fraction that do (Section 3.2
4. Computational and Numerical Insights. We prove sample complexity guarantees to approximate ESL and OMR in practice as we do not have access to the occupancy measures but collection of rollouts from the corresponding policies (Section Ap. We show the relation of empirical OMR and EST. to the true ones if the optimal policy is never reached by an algorithm. We conduct experiments on multiple environments, both discrete and continuous, with sparse and dense rewards, comparing state-of-the-arr algorithms. We observe that by visualizing aspects of the path traversed (and by comparing FST and OMR), we are able to compare and provide insights into their exploratory processos and the impact of task hardness them (Section 5
2

---

Published in Transactions on Machine Learning Research (05/2025)
2 Preliminaries
Markov Decision Processes. Consider an agent interacting with an environment in discrete timesteps. At each timestep E N, the agent observes state St: executes an action ae: and receives scalar re- ward R(st,a)) Qz). The behaviour of the agent is defined by a policy <(als:)), which maps the observed states to actions. The environment is modelled as a Markov Decision Process (MDP) M with a state space S, action space A, transition dynamics T S x A + S, and reward function R S x A - R. Dur- ing task execution, the agent issues actions in response to visited states, and hence a sequence of states and actions h; = (sg, dn, $1; Oi,.... St-1; 1, st). here called rollout, observed. In infinite-horizon set- tings, the state value function for a given policy x is the expected discounted cumulative reward over time (s)s) _ E,, > ax) so s); where y 10, is the discount rate. The goal is to learn policy that maximiscs the objective J a (V)) where p(s) the initial state distribution.
Occupancy Measure. The state-action occupancy measure is distribution over the S x A space that represents the discounted frequency of visits to each state-action pair when executing policy in the environment SSed et al. 2008). Formally, the occupancy measure of is (x - Et= is the normalizing factor.
PP((s LP 8,0,
a | H), where p p=
Stationary Markovian policies allow a bijective correspondence with their state-action occupancy mcasures Givchi 2 We express the objective J in lerms of the occupancy measure as
1 B(sa)) [R(s,a)]
J
(1)
P
where R(s, a) is the expected immediate reward for the state-action pair (s,a).
Wasserstein Distance. Let ,, E P(X) be probability measures on a complete and separable metric (Polish) space (X, The p- p-aasserstein distance between and v is (Villani 2009)
min / c(x, dm(()))) 1/P
Wp(u, (reminn.,
(2)
where the cost function is given by the metric as c(x,x') (dx 2'))² for some 1. II(,,)) v) is set of all admissible transport plans between a and i.c. probability measures on X x X space with marginals a and v. Wasserstein distances induce geodesic in well-behaved spaces of probability measures. For more discussion, we refer to Appendix A.9 For this work, we consider 1-Wasserstein distance, i.e. p p= 1, though
the results are generalisable to >
MDPs with Lipschitz Rewards. Following Pirotta et (0055) and Kallel al. (2024). we assume an MDP with Lg-Lipschitz rewards (ref. Appendix A.1 for elaboration) that satisfies [R(s,a) R(s' d')l I LRdsA((s, a), (s',a')) a')) for all s, S and a, E A. Here, dsa((s, a), (s',a') ds((s, s')) +da a')) is the metric defined on the joint state-action space S x A. This is weaker condition than assuming a completely
Lipschitz MDP. Pirotta et al. (2015) showed that for any pair of stationary policies π and n², the absolute difference between their corresponding objectives is
J J
L (i (
(3)
where Wi (V,, uxy) ) is the 1-Wasserstein distance between the occupancy measures of the policies (ref. Ap- pendix A.2 for details).
3 RL Algorithms as Trajectories of Occupancy Measures
The exploration process (i.e. the cetporaatioo-eplottttttiioi interplay and learning strategy) of an RL algorithm influences how the policy model updates its policies (Kaelbling et al. 1996 Sutton & Barto 2018). During training, policy trajectory, ie. sequence of policies (Tgg Wi,..., nN). ). is generated over policy updates due the exploratory process. We assume these policies belong to set of stationary Markov policies parameterized
3

---

Published in Transactions on Machine Learning Research (05/2025)
by 0 E O. For policies in this set To E Te, we define the space of occupancy measures corresponding to Te as .M- (um, (8, a) no E Te, f E )).
Proposition (Properties of M). If the policy a has a smooth parumelerization 0, then the space of occupancy measures M is a differentiable manifold. (Proof in Appendix A.3)
We can endow the manifold M with a 1-Wasserstein metric Wi to the compute the length of any path on M since (M, is geodesic space (ref. Appendix A.9 for details). The path distance between occupancy measures corresponding to policies parameterised by 0,0 + d0 E M is ds Wi (Vr, Uxyi Additionally, in imitation learning, the 1- L-Wasserstein distance between the occupancy measures of the learner and expert can be used as minimisable loss function to learn the expert's policy (Zhang et al.) 20220). Hence, the 1- L-Wasserstein distance reflects the effort required to achieve this imitation learning. Similarly, we propose the following quantification of the effort update from one policy to another.
Definition 1 (Effort of Learning). We define the 1 1-Wasserslein metric belween occupancy measures of two policies ग. and x², i.e. Wi(ua, U as the effort required to learn update from policy to the other.
When a learning process causes an update between occupancy measures in M, we attribute the resulting update effort to the learning process and refer to it as the effort of learning. In a learning process, first the initial policy To obtained lypically by randomly sampling the model parameters, then these parameters undergo updates until a predefined convergence criterion is satisfied, yielding the final optimal policy TN n². Since each policy has corresponding occupancy measure, this process yields a sequence of points M, which can be connected by geodesics between successive points, producing curve. The length of the curve is computed by the summation of the finite geodesic distances bet consecutive policies along it Lott 2008),
N-
C
Wi (U U )
(4)
R=
where On and ON are respectively the initial and final parameter values before and after learning.
3.1 Effort of Sequential Learning (ESL)
As saw above, RL generates trajectory in the occupancy measure manifold M, whose length is given by Equation f Compared to the long trajectory of sequential policies generated by the exploratory process, the geodesic L is the ideal shortest path to the optimal policy nN = x" from To, whose length is L T= Wi Uaw). This path would be taken by an imitation-leaminng oracle algorithm that knows n². Both these paths are schematically depicted in Figure [
Desinition 2 (Effort of Sequential Learning (ESL)). We define the effort of sequential learning incurred by trajectory of the exploratory process of RL algorithm, relative to the oracle that knows x T² TN) as,
N-1 Wi(Uays Ura 1)
(5)
Wi(UTO Vnx)
Due to the stochasticity of the exploratory process, introduce an expectation to obtain nj Eso.. # [n]]- We refer to oj as the effort of sequential learning (ESL).
i 1 and a larger n corresponds to a less efficient exploratory process of the RL algorithm. Hence, an RL algorithm with 7 2 closely mimics the oracle and has an efficient exploratory process.
3.2 Optimal Movement Ratio (OMR)
Regret measures the total deviation in value functions incurred by a sequence of policies learned by an RL algorithm with respect to the optimal algorithm that always uses the optimal policy (Sinclair al. 2023). We show that regret is connected to the sum of distances from each policy (in the sequence learned by an RL algorithm) to the optimal policy in the occupancy measure space.
Proposition 2 (Regret and Occupancy Mcasures). Given an MDP with Lx Lipschhtt rewards, we obtaón
Regret EN (J J
< La Σ Wi U. (Proof in Appendia A.4
4

---

Published in Transactions on Machine Learning Research (05/2025)
We refer to W1 (Vru, Ux-) the distance-to-optimen and analogously use it as the expected immediate regret
in the occupancy measure space. Furthermore, refer to Wi (Uz, Uo as slepurise-distance Interestingly, during training, the
and stepwise-distance distance share a relationship illustrated in Figure 2 From Figure 2 we observe that the change in distance-to-on -oaattmall-ppimma,, 6₃ - W1 (1 Ux-) Wi (W Un-) > 0, indicates that the agent got closer to the optimal. We define the set. K+ as containing indices & for which og > O, while K contains the rest.
YNu-XKa U
UT
UTN
yue
10 πT"
mo_
yua
Vnn
RNN= -11
80
UTN
X3
X2
M
Uni
m3
X1
0
m
y1
Yo
Schematic of the policy trajectory C in the space of occupancy measures M during RL training (solid line) vs the geodesic L (shortest path, dashed line) between the initial and final points (i.e. and TN- = x").
Figure 1:
Figure 2: Schematic of how saslance-lo-oplimad (de- noted by zạ) and stepwise- (denoted by ys) on the occupancy measure space describe exploratory process of an RL algorithm during training.
Definition 3 (Optimal Movement Ratio (OMR)). We define the proportion of policy transitions that effec- lively reduce the dislance-lo-ow oo-ommalll, in a learning trajectory, as
ΣKeK WilUre Una1s)
(6)
E Wi (U Uran1)
Due to the stochasticity of the exploratory process, we introduce an expectation to obtain R = Exoua refer to R as the optimal movement ratio (OMR).
[k]]. We
Note that R E [0, 1], and + indicates that nearly all the policy updates reduce the distance-to-optimu thus showing high efficiency. - implics low efficiency, since only a small fraction the policy updates contribute towards the reduction of the dislance- dislance-to-oplimadl lopommalll.
The definitions of ESL and OMR assume that the policy at the end of learning is optimal. In Section 4.2 define a version of ESL that is useful for the cases where an optimal policy is not reached. While this is not empirical proxy, we show in Section 5.3 and Appendix B.5 that it is useful when the final policy is closer to optimal than the initial one. While regret also depends on an optimal policy, it is related to cumulative rewards, whereas our metrics do not explicitly depend on rewards. Still, we show a bound with regret Proposition 2 and further discuss the possibility of extending our metrics to be reward-aware in Section g We show empirically that our metrics are complementary to regret Section 5.2 and discuss other connections with regret in Section 7
3.3
Extension to Finite-Horizon Episodic Setting
In the episodic fiiteehorizon MDP formulation of RL, in short Episodic RL (Osband et al. 2013 Azar al, 2017 Ouhamma et al. 2023), the agent interacts with the environment in multiple episodes of H steps. An episode starts by observing state $1, then for = 1...., H, the agent draws action a, from (possibly time- dependent) policy Te(- st), observes the reward r(st, az), and transits to a state st+1 ~ T(- St, a.). Here, the value function and the state-action value functions at step h E [H] are respectively defined as V7(()
Ex En r(8,, az) 8r. = and (7( (8,a) 2 E, E/ r(se, 0e) 8h 8, an = a) Following Alltman 1999

---

Published in Transactions on Machine Learning Research (05/2025)
can define hortte-hrizzon version of occupancy measures
M H P(s = 8, = a |T, p.)-)
H (s, a)
Σ
Following Syed et al. (200S), work by Kalagaria et al. 2021 shows that of can be used in the linear programming formulation for solving MDPs and satisfies the Bellman Flow Construints (in Equation 19 from Appendix A.3). We prove that under some assumptions, the fnite-horizon occupancy measures also construct manifold, referred to as MH.
Proposition 3 (Properties of M If the policy has smooth parometrization 6, then the space of finite-horizon occupancy measures MR is differentiable manifold. (Proof in Appendia A.5
This allows us to similarly define Wasserstein metric on this manifold, which in turn, allows us to compute ESL and OMR for evaluating different RL algorithms.
4 Computational Challenges and Solutions
Similar to regret, our method requires knowing the optimal policy. This is because the efficiency and effectiveness of exploratory processes of RL algorithms are highly coupled with their ability to reach optimal policy. ESS. and OMR depend on the policies being stationary and Markovian.
4.1 Policy datasets for computing occupancy measures
We consider approxinations of occupancy measures using datasets assumed to be drawn from these mea- sures. We estimate the Wasserstein distance between the occupancy measures using method introduced by Alvarez-Melis & Fusi (2020) known as the oplimal transport dalasel dislance (OTDD). OTDD uses dalasets to estimate the Wasserstein distance between the underlying distributions. See Appendix A.6 for a detailed account of OTDD.
Definition 4 (Policy dataset). A dalasel of policy Drr is set of slate-uction pairs drawn from the policy 's occupancy measure, i.e. Dx = {((()) ((8())((())()))))))))))))) a()))= Ug. These can be constituted from the rollouts generated by the policy during task ezecution.
We know from imitation learning that if we are given Dr, generated by an expert policy, we can train a policy model on in a supervised manner via behaviour cloning (Hussein et al. 2017). Thus, knowing Dx can allow converting an RI. task into a Supervised Learning (SL) task. Consider a sccnario when we have access to a sequence of datasets (D....... Dax), each corresponding to policy मi for 2 0. If we train (in a supervised manner) policy model sequentially evolution as the RL algorithm that generated the policy trajectory (mz))20- This allows us to conceptualise learning in RL as a sequence of SL tasks with sequential transfer learning across the datasets (D DaN). We employ OTDD to estimate W1 (1(u Ure+s) ) using these datasets, i.e. dor (D D Re Wi URE4s): based on Proposition
these datasets, the model will undergo a similar policy
Proposition 4 (Upper Bound on Estimation Error). Let an RL algorithm yield a sequence of policies NO...., nN while training. Now, we construct N dalasets D ,.,,,, each consisting of M rollouts of the corresponding policies. Then, we can use these datasets to approzimate ENE Wi Ux by Here, T is the total number of steps per episode, diam(SA) the diometer of the state-action space, and Eg is a positive-vale and polylogarithmic function of S and A. For finâte horizon case, we can further reduce the error bound to 2NE, VM
EN-I (irDD,,,DTu)) with an expected error upper bound. 2NE, + N VMI
Proof of Proposition in Appendix A.7 The results support that EST. and OMR can be estimated as
Ener dor(D (D Dan.)) dor('D.o Drx)
ΣtEK+ dor DTV_.)
7 = Ero.
and k = Ero..
(8)
EN- dor (Drg, D Dreaa)

---

Published in Transactions on Machine Learning Research (05/2025)
4.2 When optimal policy is not reached
So far we have assumed that the algorithms converge at the optimal policy, ie. AN = =². However, this is not always truc. We consider a scenario when IN o x², and define
Ek
Wi (Urr UNTR+T
M/uub
Wi Wi UnN)
TN # n
(9)
Proposition 5. Given N 2 2 and To + nN TT,,, we obtain
7 7aub
2WV1 2W1 (UT vx-)
(10)
7
Wi Unx)
Wi (1 Ug-) + Wi (U Un-) 2 Wi Bxy) and (i( Uzx) + Wi (Vxx: ux-) 2 (1(UeN-1 Ug-). The proof is provided in Appendix A.8 Note that Equation (10) shows that when nN is close to T², then VDant is good approximation of 1), and thus good quantifier to determine the efficiency of the algorithm's exploratory process. However, W1 (1
This is true due to the triangle inequalities:
is de- pendent on the RL algorithm and hence a bound cannot be provided here. Still, 7/su might be useful when TN is closer to x² than To. A fallible proxy for this could be when the performance of TN better than that of mo, i.c. JØ > J We show the usefulness of 7uub in our experimental results Section 5.3 and Appendix B.5 for simple environments. It remains be seen how useful Tuub is in complex environments.
5 Experimental Evaluation
In this section, we evaluate the proposed methods in the 2D-Gridworld and Mounlain Car MMoore 1990;
Brockman et al. 2016) environments, to analyze our methods discrete and continuous state-action spaces respectively. The 2D-Gridworld environment of size 5x5 with actions: fup, right, down, lcft). In the gridworld, we perform experiments on settings namely:- A) deterministic with dense rewards, B) deter- ministic with sparse rewards, and C) stochastic with dense rewards. Further details about these settings are provided in Appendix [..] The Mountain Car environment, in our experimentation, is deterministic MDP with dense rewards that consists of both continuous states and actions described in detail in (Brock. man et al. 2016). Note that we used L1 distance and L2 distance as metrics (dx) for the state spaces of the 2D-Gridworld and Mountain Car, respectively, which underpin the W1. Our code is available at:
wtw.la..oooo/innaaanaaeii wwtmm//khuisseree.l aaalysiaaaalissss...aaa wwc__ocppaccc_eeeeeeeeeee measure _ww.sreettraeeetorrrrrr trajectory
Our experiments aim to address the following questions:
1. What information can the visualization of the policy evolution during RL training provide about the exploratory process of the algorilhm?
2. How do ESL and OMR allow us to analyze the exploratory processes of RL algorithms?
3. Does ESL scale proportionally with task difficulty?
Summary of Results. In Section 5.1 we demonstrate that visualizing the evolution of dislance-looopllmaal and stepuosee-disannce of different RL algorithms during training reveals: 1) whether the agent stuck suboptimal policies, 2) the coverage area of the exploration processes, and 3) their varied characteristics over time. We further compare ESL and OMR of different algorithms few environments in Section 5.2) Finally, we show in Section 5.4 that ESS,, scales proportionally with task difficulty, and thus, reflects the effects of task difficulty on exploration and learning.
5.1 Exploration Trajectories of RL Algorithms
(I) DISCRETE MDP. To understand the utility of visualizing exploratory processes, we use the following RL
a algorithrns: 1) Tabular Q-learning with a) e-greedy (e -==00 0) and b) e-greedy (e = 1) stralegies; 2) UCRL2
(Jaksch et al. 2010); ; 3) PSRL (Osband et al. 2013); 4) SAC (Haarnoja et al. 2018 Christodoulou 2019
and 5) DQN MMii et al. 2013) with e-ecay.. The algorithms solve a simple 5x5 gridworld with dense
rewards, starting from lef-left (0,0) to reach bottom-right (4,4). Figure 3 presents exploratory behaviours of the algorithms in both the occupancy measure space and state space.
7

---

(NB. Larger versions of these plots are presented in Appendix D.1, D.3. and corresponding performance plots are in Published in Transactions on Machine Learning Research (05/2025)
digready
Argreedy
VCRLD
PILL
SAC
GOtebie opaiinal
Figure 3: Top row: 3D plots of fistance-to-optimne (x-axis) and stepwise-distares (y-axis) across number of updates (z-axis), illustrating policy evolution in the occupancy measure space for algorithms: e(=0)-greedy and c(=1)-greedy Q-learning, UCRL2, PSRL, SAC, and DQN (left to right). Bottom row: Corresponding state visitation frequencies over the full training. The problem setting is deterministic with dense rewards and 15 maximum number of steps per episode.
while their 2D projections are in Figure 15 al Appendix
Figure 12 at Appendix D.2)
Q-learning: E = 0 vs c = 1. Note that e = 0 updates the Q-table by only exploiting, while e = by exploring. From the state visitations, we observe expected charactoristics, like a preferred visit path for c= 0, versus e with visitation frequencies that are similar at states equidistant from the start-state and gradually decreasing as the distance from the start-state increases. From the policy evolution, we see how scattered and erratic the policy transitions are for O. Whereas e = I is dominated by unchanging or little-changing policics seen by straight vertical line segments (indicating being stuck in suboptimality). In this setting, e = 0 is characterised by transitioning between diverse policies (i.e. being aggressive with larger coverage arca) while c = 1 is likely to be stuck in suboptimality. The stuckness is due to high action randomness in = that causes the agent to select suboptimal actions, slowing the convergence of the Q-table and not changing the learning policy until the best actions are discovered.
UCRL2 vs PSRL. UCRL2 has nearly uniform state visits (with the exception of the start-state because the initial state distribution is at state (0,0)), thus being consistent with literature since the algorithm selects exploratory state-action pairs more uniformly (Jaksch et al. 2010). In contrast, PSRL has high visit frequencies along the diagonal states, because it selects actions according to the probability that they are optimal (Osband et al. 2013). We observe from the policy evolution plots that PSRL has smoother policy transitions that. are orientated towards optimality, while UCRL2 behaves more aggressively with policy transitions that do not taper as it approaches optimality. Osband et al. (2013) highlighted that exploration in PSRL is guided by the variance of sampled policics as opposed to optimism in UCRL2. We observe in Figure 3 that the guiding variance in PSRL reduces after every policy update until optimality is reached, while UCRL2 maintains high variance.
SAC vs DQN. The state visits of both the algorithmss appear to be similar. SAC has higher visitation frequencies at the corners than DQN. Surprisingly from the policy evolution plots, we learn that both algorithms have a reluctance to transition between policies hence the stuck in suboptimality vertical line segments, especially initially. This reluctance is due to the slow 'soft updates' of target networks {Lillierap et al. 2016) in the algorithms. We also observe that SAC approaches optimality more gradually than DQN.
All algorithms. Figure shows that UCRL2 was more meandering (with larger coverage arca) towards optimality than the rest. SAC and DQN approsched optimality more directly and smoothly (with smaller coverage area) than the rest. These characteristics are intuitively revealed by policy visualisation plots, and are aligned with literature, hence enhancing our understanding of the exploratory processes.
(II) CONTINUOUS MDP. We use DDPG (Liliierap al. 2016 and SAC to solve the Mountain Car. The policy evolutions of these algorithms are presented Figure
8

---

Published in Transactions on Machine Learning Research (05/2025)
DDPG SAC. Both exhibit short-distances (< between policy updates (i.e. small coverage area). They depict no sign of being stuck settling early on any particular policy, which shows their continuously ex- ploratory nature. While they begin with almost constant mean distances-fo-optimal and stepwise-distanee SAC drops its mean distance-to toaptimml--p earlier than DDPG.
Figure 4 illustrates how OMR changes with update number k. OMR(k) represents OMR starting with the 4th policy the initial policy, while OMR starts from the 0th policy (details of computing OMR(k) are in Appendix B.2 For both algorithms, OMR(k) remains near chance level (~ 0.5) initially, then sharply increases near the final updates. This suggests that early policy updates purely exploratory and oblivious to policy improvement but align with the optimal policy just before convergence. The efficiency of the algorithm depends on how early this transition occurs, e.g. starts earlier for SAC than DDPG, rendering SAC more efficient.
ddpg
sBc
0.0
3.00
0.5
5:35
0.50
1.0
0.75
1.00
#updates
1.5
1.50
2.25
2.0
1.75
2.00
stepwise d
0.5
stepwise distance
03
4
D.J
0.2
distance to optimal
D. 0.8
85
distance to optimal
42 de 2a
nnc
a.
0.60
0.60
0.55
OMR(K)
0.50
0.50
0.45
0.45
0.40
0.40
5000
10000
15000
20000
5000
10000
15000
20000
#updates
#updates
Figure 4: Top row: 3D plots of Distance-to-optinie and stepwise-distane distance vs. number of updates for DDPG and SAC. Bottom row: OMR(k) vs. #update, k. for the corresponding algorithms. Note that corresponding performance plots are in Figure 13 in Appendix D.2
5.2 Comparison of ESL and OMR across RL Algorithms and Environments, and their complementarity to number of updates and regret
Tables showcase how ESL and OMR are summary indices of the policy trajectories during learning by evaluating the algorithms in various settings.
Dense Rewards. We observe, Table 1 that PSRL took the lowest number of updates (UC) to reach the optimal policy in contrast with SAC. Yet, PSRL was meandering more than SAC (see Figure 3)- The relative directness of SAC is captured by lower ESI. and higher OMR compared SAC has larger UC than PSRL, it took shorter path to optimality than PRSL. This shows that UC does not necessary correlate with ESL and OMR, and it provides incomplete information about the exploratory processes. Indeed, two algorithms may have the same UC, but different ESL and/or OMR due to different step-wise distances and varied movement towards optimality.
PSRL. Even though
Sparse Rewards. In the sparse rewards setting (Table 2). low performance DQN is observed in both our indices and UC. However, SAC is more efficient with lowest ESL and highest OMR, yet UCRL2 has the lowest number of updates (UC). Note that UCRL2 provably regret-optimal, while SAC does not have such rigorous theoretioal guarantees but is known to be practically efficient, and this is well captured by ESL and

---

Published in Transactions on Machine Learning Research (05/2025)
Algo.
ESL
OMR
UC
SR%
Algo
ESL
OMR
UC
SR%
SAC
9.26415.54
0.58=0.14
980-670
100
Deterministic, sparse
UCRL2
47.2+8.200
0.49-0.04
60.7+11
100
SAC
27.8+22..9
0.57
4385-+3274
100
PSRL
23.2:11..5
0.52-0.06
34.149.34
100
UCRL2
73.3+0.0 0.45+0.0
93.0+0.0
100
DQN
12.4+7.13
0.54-0.11
161+933
98
PSRL
73.2 0.52=0.076
100167.3
100
<( ((=1)-greedy
6.27:2.22
0.61=0.09
672=385
DQN
137=154"
0.49
12638-14331
80
-eeay
8.10+33.3
0.61-0.10
389+138
100
Stochastic, dense
greedyy
15.545.28
0.53=0.06
17613379
84
SAC
445+245
0.501=0.004
2463+:2043
92
UCRL2
198:121
0.502=0.027
268+:155
32
Table 1: Evaluation of RL algorithms (over runs) in the deterministic, dense-rewards setting for 5x5 gridworld, including Effort of Sequential Learn- ing (ESL), Optimal Movement Ratio (OMR), num- ber of updates to convergence (UC), and success rate (SR). Lowest F.ST., lowest. UC, and highest OMR val- ues are bold. The highest ESL value is starred (+).
PSRL
55.4+33..6
0.52=0.04
76.16
92
DQN 458-311
0.502=0.01
1586+1077
24
Table 2: Evaluation of RL algorithms (over 40 runs) in the deterministic, sparse-rewards and stochas- tic, dense-rewards settings for 5x5 gridworld. Low- est ESL, highest OMR and lowest UC values are bold. The highest ESLs are starred.
OMR. So lar, Tables and 2 demonstrate how our indices provide clearer picture of exploratory processes than the number of updates.
Stochastic Transitions. the stochastic setting (Table 2 by observing only successful cases, we notice that the meandering characteristic of PSRL, and UCRL2 more suitable for this setting than SAC and DQN (based on better ESL and OMR values). PSRL and UCRL2 have similar regret bounds (Osband al., 2013 yet in Tables 1 and 2 PSRL has better ESL and OMR (along with higher success rate). This aligns with the regret analysis presented in (Osband et al. 2013).
Table corroborates with the policy evolution plots in Figure in that duc to SAC dropping its mean collance-lo-pplimad earlier than DDPG it exhibits a lower ESL. Additionally, we notice a trend of increasing ESL and decreasing OMR across algorithms when shifting from dense-rewards to sparse-rewards settings, from deterministic to stochastic transitions, from discrete to continuous environments, indicating an increase in the effort of the exploratory processes. We have shown that ESL and OMR enhance the understanding of exploratory processes by effectively summarizing the policy trajectories of algorithms during learning. They offer more insight than the number of updates, and align with regret when algorithms reach an optimal policy. In the next Scction, we demonstrate the utility of ESL when an optimal policy is not reached.
Algo. ESL DDPG 1881#500 0.501 23500-5268
OMR
UC
SR%
100
SAC 1619+189
0.5 22700-2971
100
Table 3: Evaluation of RL algorithms in the Mountain Car continuous MDP (over 5 runs). The variances for OMR are negligible.
5.3 Usefulness of ESL when optimal policy is not reached
When an optimal policy is not reached at the end of learning, ESL cannot be computed exactly. At this point, we propose to use an approxination of ESL, i.c. Neub (Equation gb. A natural question arises: When the optimal policy is not reached, does the Nwub still yield insights about the exploratory process of RL? In Table 4 we compare ESL when the opai policy was reached, i.e. 7), versus when it was not, i.e. JJub-
We observe that Naub values are always greater than n values. However, they both yield the same efficiency ranking (e.g. PSRL, UCRL2, SAC and DQN). This indicates that naus reliably prediets results provided by y for relative comparison of algorithms.
10

---

Published in Transactions on Machine Learning Research (05/2025)
Algo. "
d
SAC
4451246
8531127
5.6311.23 7.26=1.45
UCRL2
198#121
5101274
5.36+2.84
4.58=1.90
PSRL
35.4433.6
361443.6
4.97#1334
3.91=0.48
DQN 458+311
1971+-250
4.88+1.06 6.52-0.31
Table 4: Evaluation of algorithms in the stochastic, dense-rewards setting for 5x5 gridworld. When the algorithm converged lo optimality, y) is the Effort of Sequential Learning, and d Wi (Uroz Un-) is the distance between the initial and optimal policies. When the algorithm did not converge to the optimal policy but some NN. we used sub and Wi(wn, (xx) to denote the aforementioned quantitics. 40 training trials were used.
5.4
ESL Increases with Task Difficulty
Figure depicts the ESL values for Q-learning with e-decay strategy (for e = 0.9) across tasks with varying hardnoss. These tasks are deterministic 3D-Gridworld of sizes 5x5 and 15x15 matched with either dense sparse rewards (as specified in Appendix B.Ip. We chose assess the e-decay Q-learning algorithm because it is simple and yet completes all these tasks. We observe that the F.ST. is lowest for [5x5] dense (5x5 grid, dense rewards) and highest for [15x 15j sparse (15x15 15 grid, sparse rewards) as anticipated. The results demonstrate that ESL scales proportionally with task difficulty, matching expectations that more difficult tasks demand greater effort of the exploratory process.
10?
ESL
10
109.
[5x5]
[5x5]
[5x5]
[15x15] [15x15]
dense
sparse
sparse
dense
sparse
(easy)
(hard)
Figure Q-learning with e-greedy (e = 0.9 decaying, averaged over 40 runs) across deterministic 2D- Gridworld (5x5 and 15x15) tasks. The 1st and 4th (from left to right) have dense rewards, while the rest have sparse rewards (details in Appendix B.1
Remarks. When the optimal policy reached, we can use the visualisation of policy trajectories, as well ESL (1) and OMR to study exploratory processes of RL algorithm. When the optimal policy is not reached, we can still use the visualisation of policy trajectories and ESL-sub (Neat) to study exploratory processes because they still capture characteristies of exploratory processes (e.g. high coverage, smooth exploration). Additionally, ESL (1 or Nsub) captures the hardness of the task that we are solving. Thus, studying the occupancy measure trajectories and their corresponding indices can aid making a knowledge- able choice of an RI. algorithm that exhibits desired characteristics.
11

---

Published in Transactions on Machine Learning Research (05/2025)
6 Related Works
Several prior works have utilized various components leveraged in our work, namely Wasserstein distance, occupancy measures, and the trajectory of RL on a manifold, but for different purposes. Here, summarise them and elucidate the connections.
In supervised learning, Alvarez-Melis & Fusi 2020 proposed an optimal transport approach, namely Optimal Transport Dataset Distance (OTDD), to quantify the transferability between two supervised learning tasks by computing the similarity (i.e. distance) between the task datasets. Here, we conceptualise and define the effort of learning for RL, as a sequence of such supervised learning tasks. We observe that the total effort of sequential learning can be compuled as the sum of OTDD dislances between conseculive occupancy measures. Recently, Zhu et al.] (2024) have developed generalized occupancy models by defining cumulative features that are transferable across tasks. In future, one can generalize our indices for the cumulative features constructed from some invertible functions of the step-wise occupancy measures.
Optimal transport-bascd approaches are also explored in RL literature. These works broadly belong to two families. First line of works uses Wasserstein distance over a posterior distribution of Q-values (Metelli let al.) 2019 Likmeta al., 2023) or return distributions (Sun et al.) (022) to quantify uncertainty, and then to use this Wasserstein distance as a loss to learn better models of the posterior distribution of Q- values or return distributions, respectively. The second line of works uses Wasscrstein distance between feasible family of MDPs as an additional robustness const to design robust RL algorithms (Abdullahi et al. 2019 Derman & Mannor 2020; Hou et al.] 2020). Here, we introduce a concept of using Wasserstein distance between occupancy measures to understand the exploratory dynamics. Incorporating this insight into better algorithm design would be an interesting future work. Recently, Calo el al. (2024) related Wasserstein distance between reward-labelled Markov chains to bisimulation metrics which abstract state spaces. In the same spirit, we could use reward as the cost-function in computing our nested Wasserstein distance (OTDD) to obtain reward- value-aware OTDD to define broader bisimulation metrics with abstract state-action spaces, instead of just state spaces.
As a parallel approach to optimal transport, the information geometries of the trajectory of an RL algo- rithm under different settings are studied. These approaches use mutual information as a metric instead of Wasserstein distance. |Basu et al. (2020) studied the information geometry of Bayesian multi-armed bandit algorithms. They considered a bandit algorithm as a trajectory on belief-reward manifold, and proposod geometric approach to design a near-optimal Bayesian bandit algorithm. Eysenbach el al. (2021) ; Laskin et al. (202)) studied information geometry of unsupervised RL and proposed mutual information maximi- sation schemes over a set of tasks and their marginal state distributions. Yang et al. (2024) extended this approach with Wasserstein distance and demonstrated benefits of using Wasserstein distance than mutual information. We use Wasserstein distance as a natural metric in occupancy manifold which aligns with the hardness of different tasks. It would be interesting to extend our framework to understand the dynamics of unsupervised RL algorithms.
7 Discussion
Our work introduces methods to theoretically and quantitatively understand and compare the exploratory strategics of different RI. algorithms. Since learning in a typical RI. algorithm happens through a sequence of policy updates, we propose to understand the exploratory process by visualizing and analyzing the path traversed by an RL algorithm in the space of occupancy measures. We show the usefulness of this approach by conducting experiments on various environments and RL algorithms.
Our results show that ESL and OMR provide insight into the evolution of the agent's policy, revealing whether it is approaching the optimal policy in steady or meandering manner. Additionally, they allow us to understand how the learning process of the same algorithm changes with different rewards and transitions structures, and task hardness. We emphasize that ESL and OMR complement the number of updates to converge and regret (see Appendix B.7 rather than replace them.
We now discuss various practical aspects and future possibilities.
12

---

Published in Transactions on Machine Learning Research (05/2025)
Computational complexity. While efficiency is an important aspect, the primary focus of this work is to introduce a framework for analyzing exploration in RT. using occupancy measures. It is of interest to utilize our approach to benchmark and compare the learning dynamics of various RL algorithms in more environments, espocially large-scale or high-dimensional oncs. Computing Wasserstein distances in such environments would incur high computational costs, however in recent years several methods such as greody computation (Carlier et al., 2010), hicrarchical methods (Lee et al., 2019), and inexact proximal point methods (Xie al.) 2020) have been introduced to handle large-scale OT problems. For example, (Gao & Chaudhari 2021) leverages a block-diagonal approximation method to deal with high-dimensional probability distributions similar to ours, while anchor space OT (Huang et al. 2024) specifically addresses multiple OT problems with multiple distributions.
Use of 1- 1-Wasserstein metric W1. The W1 satisfies the Kantorovich- Rubinstein duality, making Equation (3) applicable and providing a basis for bounding regret in Proposition 2. In contrast, wps1 does not have such duality. Since (i(P.Q) Q) S Wps1(P.Q) Q) (Villani 2009). Wi yiclds tighter regret bounds. Morcover, Wi is less sensitive to outliers and sampling discrepancies than Wp>1 (Raghvendra et al.) 2024), making it well-suited for our setting. Nevertheless, our approach extends to Wps1. Compared to KL-divergence, Wi is a metric that satisfies symmetry and triangle incquality, which have been instrumental in our proofs and guarantees, while KL-divergence is not a metric. Additionally, W321 leverages the geometry of the underlying support space (Peyré. 2019). This allows it to capture distances between distributions with disjoint support (Peyré 2019), which KL-divergence cannot capture. Hence, we use Wi in our approach.
Choice of distance metric dx. The choice of the distance metric impacts the geometry of the support space (Tee 2009), consequently the Wasserstein distances, and thus ESL and OMR. In our case, the support space (i.e. state-action space) is reduced to the state space, as the action space maps back to the state space via OTDD. Thus, the choice of distance metric should reflect the effort of moving in the state space. For example, in the Gridworld, we used L1 distance because only vertical and horizontal displacements are allowed, and L2 distance in the Mountain Car, as applicable to real-world spaces.
Regret and sures the similarity of returns (with respect to optimal), disregarding behavioural differences, like variations in actions at the same states. This makes regret advantagcous in scttings without critical safety and physi- cal constraints, e.g. games, due to its computational elficiency. In contrast, the tiieoodddilancett-opolii focuses on behavioural differences between policies, is reward-agnostic, and is well-suited for environments where safety and physical constraints are critical, e.g. robotics. While minimizing regret prioritizes match- ing the performance of the optimal policy, minimizing the dessd-ddaannccoo-pppae can be used similarly to regret, especially where the
dliitances-tooopliinee
(Ei- Wi ("=, Comparing regret across policies mea-
focuses on mimicking
its behaviour. Thus, the behaviour of achieving good performance is essential.
Algorithm Selection and Design. Depending on the environment, we can select algorithm with promising characteristics and spend more time optimising it to improve performance. For example, rather than tuning the syper-parameters of multiple competing algorithms to find the best one, it may be more effective to first identify an algorithm best suited to an environment based on ESL and OMR, and then fine-tune it. The chosen algorithm might remain suitable across similar environments as well.
Furthermore, we could incorporate an online adaptation of the exploratory process of the RL algorithm itself, based on recent estimates of the ESL and OMR. For example, if the current policy gives better return than the initial policy, then we could adapt the exploratory parameters (at a slow rate) to optimise a running estimate of Neub and a suitable approximation of OMR, thus enabling better exploration. However, the feasibility and convergence of such a scheme remain open.
Acknowledgments
R. Nkhumise expresses his gratitude to Pawel Pukowski and Mohamed S. Talamali for their insightful discus- sions and valuable feedback on the ideas explored in this study. R. Nkhumise was supported by the EPSRC Doctoral Training Partnership (DTP)- - Early Carcer Rescarcher funding awarded to A. Gilra. A. Gilra acknowledges the CHIST-ERA grant for the Causal Explanations in Reinforcement Learning (CausalXRL) project (CHIST-FERA-II-XAA100g) by the Engineering and Physical Sciences Research Council, United
13

---

Published in Transactions on Machine Learning Research (05/2025)
Kingdom (grant reference EP EP/VO55720/1) for supporting the work. D. Basu acknowledges the CHIST-ERA grant for the CausalXRL, project CCHIST-FRAA-1--XXT--OO by L'Agence Nationale de la Recherche, France the ANR JCJC for the REPUBLIC project AANR-22-CE23-0000--00
(grant reference ANR-21-CHHR4-0000 and the PEPR project FOUNDRY (ANR23-PEIA-0003) for supporting the work. We thank Eleni Vasilaki and Philippe Preux for their support.
References
M. A. Abdullah, H. Ren, H. B. Ammar, V. Milenkovic, R. Luo, M. Zhang, and J. Wang. Wasserstein robust reinforcement learning. Xrxiv preprint or Xiv: Xiw:1907.13190 2019.
A. Agarwal, N. Jiang, S. M. Kakade, and W. Sun. Reinforcement learning: Theory and algorithms, 2022. URL lttpps//rritheoryboott git.ub. io/
M. Aleksandrowicz and J. Jaworek Korjakowkk... Metrics for assessing generalization of deep reinforcement learning in parameterized environments. JAISCR, 14(1):45-60 61, 2023.
E. Altman. Constrained Markov Decision Processes. Routledge, 1999.
D. Alvarez-Melis and N. Fusi. Geometric dataset distances via optimal transport. In Proceedings of the 34th International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
S. Amin, M. Gomrokchi, H. Satija, H. Hoof, and D. Precup. A survey of exploration methods in reinforcement learning, 2021.
M. G. Azar, 1. Osband, and R. Munos. Minimax regret bounds for reinforcement learning. In International conference on machine learning, pp. 263-272. PMLR, 2017.
D. Basu, X. Wang, Y. Hong, H. Chen, and S. Bressan. Learn-as-you-go with megh: Efficient live migration virtual machines. JEEE Transactions on Parallel and Distributed Systems, 30($):1786-1801, 2019.
D. Basu, P. Senellart, and S. Bressan. Belman: An informatioon-geonetrie approach to stochastic bandits. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019, Würzburg, Germany, September 16-20, 2019, Proceedings, Part III, pp. 167-183. Springer, 2020.
M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471- 1479. Curran Associates, Inc., 2016.
H. Bojun. Steady state analysis of episodic reinforcement learning. In H. Larochelle, M. Ranzato, R. Had- sell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9335 9345. Curran Associates, Inc., 2020. URL httttp:://prcceednngge neurips. cc/paper files/ Paper. pdf
S. Boyd and Vandenberghe. Convez Optimization. Cambridge University Press, 2004.
G. Brockman, V. Cheung, I.. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016. URL https://ww/ httpp:://ww..mmlbbrrrrry dwtp::/deeeeeeeeeeeeeeeeee httpr//ddeeeeiirreeetaaeeeeeeeee http://ww.oatttt.ooaaaaaaooattta
ternational Conference on Learning Representations, 2019. URL control/mountain H11JJnH5Ym
car/
Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. In In-
S. Calo, A. Jonsson, G. Neu, L. Schwartz, and J. Segovia- Bisimulation metrics are optimal transport distances and can be computed efficiently, 2024. URL
G. Carlier, A. Galichon, and F. Santambrogio. From knothe's transport brenier's map and continuation method for optimal transport. STAM Journal Mathematical Analysis, 41(():2554 (16626254-2576, 2010.
14

---

Published in Transactions on Machine Learning Research (05/2025)
P. Christodonlon. Soft actor-critic for discrete action settings, 2019.
E. Derman and S. Mannor. Distributional robustness and regularization in reinforcement learning. arXiv preprint SKfe::2033028944 2020.
M. P. Drazin. Pscudo-inverses in associative rings and semigroups. The American mathematioal monthly, 65 (7):506 514, 1958.
B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019.
B. Eysenbach, R. Salakhutdinov, and Levine. The information geometry of unsupervised reinforcement learning. arXiv preprint arXiv:2110.0277 00.01719, 2021.
R. Flamary, N. Courty, A. Gramfort, M. Z. Alaya, A. Boisbumon, S. Chambon, L. Chapel, A. Corenflos, K. Fatras, Fournier, L. Gautheron, N. T.H. Gayraud, H. Janati, A. Rakotomamonjj, I. Redko, A. Rolet,
A. Schutz, V. V.Seguy, D. J. Sutherland, R. Tavenad,, A. Tong, and T. Vayer, Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1-8, 2021. URL http://jmlr.. wtt.:/rr.rrgpppss//2////// html
Y. Gao and P. Chaudhari. An informatioo-geommerii distance on the space of tasks. In Proceedings of the 38th International conference machine learning, pp. 3553 3563.. PMLR, 2021.
C. Gclada, S. Kumar, J. Buckman, O. Nachum, and M. G. Bellemare. Decpmdp: Learning continuous latent space models for representation learning. In Proceedings of the 36th International Conference Machine Learning, pp. 2170 2179. PMLR, 2019.
A. Gibbs and E. Su. On choosing and bounding probability metrics. International Statistical Review Revue Internationale de Stalistique, 70(3):419-435, 2002.
A. Givchi. Optimal Transport in Reinforcement Learning. PhD thesis, Graduate School-Newark Rutgers, The State University of New Jersey, 2021. CRL https:// wwbraries............... wwbrariss.rutgerrsccco hwt.gurrrr.dr/uugerreeeeee
66700/PDE/17
X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics, 2010.
T. Haarnoja, A. Zhou, P. Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, 2018.
T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine. Soft actor-critic algorithms and applications, 2019.
L. Hou, Pang, X. Hong, Y. Lan, Z. Ma, and D. Yin. Robust reinforcement learning with wasserstein constraint. arxiv preprint arXiv:2006.009) 2020.
J. Huang, X. Su, Z. Fang, and Kasai. Anchor space optimal transport as a fast solution to multiple optimal transport problems. IEEE Transactions on Neural Networks and Learning Systems, 2024.
A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys, 50:1 35, 2017.
J. Ibarz, J. Tan, C. Finn, M. Kalakrishnan, P. Pastor, and Levine. How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotícs Research, 40(4-5); 698-721, 2021.
M. Jaderberg, W. M. Czarnecki, I. Dunning, L. Marris, G. Lever, A. G. Castaneda, C. Beattie, N. C. Rabinowitz, A. Morcos, A. Ruderman, N. Sonnerat, T. Green, L. Deason, Z. Leibo, D. Silver, D. Hassabis, K. Kavukeuoglu, and T. Gracpel. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science, 364(6443))855 1865595-55666 2019.
15

---

Published in Transactions on Machine Learning Research (05/2025)
T. Jaksch, R. Ortner, and P. Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, (1(11)156331660 1600, 2010. URL http://jmlr ort/ppaprrss/t//asssshol. html
L. P. Kaelbling, M. L. Littman, and A. W. Moore. Journal of artificial intelligence research. Reinforcement Learning: A Survey, 4:237 -285, 1996.
K. Kalagarla, R. Jain, and P. Nuzzo. A sample-efficient algorithm for episodic finite-horizon mdp with constraints. In The 35th AAAI Conference on Artificial Intelligence, pp. 8030 8037, 2021.
M. Kallel, D. Basu, R. Akrour, and C. D'Eramo. Augmented bayesian policy search. In The 12th International Conference of Learning Representations, volume 139, 2024.
E. Kaufmann, L. Baucrsfeld, A. Loquercio, M. Müller, V. Koltun, and D. Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620(7976):982 987, 2023.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization, 2017.
S. Kolouri, S. R. Park, M. Thorpe, D. Slepcev, and G. K. Rohde. Optimal mass transport: Signal processing and machino-learning applications. IEEE Signal Processing Magazine, $4:43-59, 2017.
P. Ladosz, L. Weng, M. Kim, and H. Oh. Exploration in deep reinforcement learning: A survey, Information Fusion, 85:1-22, 2022.
R. Laroche and R. T. des Combes. On the occupaney measure of non-markovian policies in continuous mdps. In Proceedings of the 40th International Conference on Machine Learning, volume 202, 2023.
M. Laskin, H. Liu, X. Bin Peng, D. Yarats, A. Rajeswaran, and P. Abbeel. CIC: contrasttvv intrinsic control for unsupervised skill discovery. CoRR, abs 22022.0011, 2022. URL sttps://arxiv..rg/abb... 00161
T. Lattimore and C. Szepesvári. Bondit algorithms. Cambridge University Press, 2020.
A. Lazaridis, A. Fachantidis, and I. Vlahavas. Deep reinforcement learning: A state-of-the-art walkthrough. Journal of Artificial Intelligence Research, 69:1421 1471, 2020.
J. Lee, M. Dabagia, F. Dyer, and C. Rozell. Hicrarchical optimal transport for multimodal distribution aligmment. Advances in Newrol Information Processing Systems, 32, 2019.
J. M. Lee. Manifolds and differential geometry, volume 107. American Mathematical Society, 2009.
S. E. Li. Deep reinforcement learning. In Reinforcement learning for sequential decision and optimal control, pp. 365-402. Springer, 2023.
A. Likmeta, M. Sacco, A. M. Metelli, and M. Restelli. Wasserstein actor-critic: directed exploration via optimism for continnous-actions control. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 8782 8790, 2023.
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcementt learning. In International Conference on Learning Representations, 2016. URL httggs//aardd..org//////116 02971v6.
S. Liu. An evaluation of ddpg, td3, sac, and ppo: Deep reinforcement learning algorithms for controlling continuous system. In International Conference on Data Science, Advanced Algorithm and Intelligent Compaling, 2023.
J. Lott. Some geometric calculations wasserstein space. Communications in Mathematical Physics, 277: 423 -437, 2008.
M. Memmel, P. Liu, D. Tateo, and J. Peters. Dimensionality reduction and prioritized exploration for polic search. In 25th International Conference Artificial Intelligence and Statistics, 2022.
16

---

Published in Transactions on Machine Learning Research (05/2025)
A. M. Metelli, A. Likmeta, and M. Restelli. Propagating uncertainty in reinforcement learning via wasserstein barycenters. Advances in Newral Information Processing Systems, 32, 2019.
C. D. Meyer. The role the group generalized inverse in the theory of finite markov chains. SIAM Review, 17(3))443 464, 1975.
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning, 2013.
A. W. Moore. Efficient Memory-ben based Learning for Robot Control. PhD thesis, University of Cambridge, 1990.
S. Müller, A. von Rohr, and S. Trimpe. Local policy search with bayesian optimization. Advances Neural Information Processing Systems, 34:20708 20720, 2021.
G. Neu and C. Pike-Burke. A unifying view of optimism in episodic reinforcement learning. Advances in Newral Information Processing Systems, 33:1392-1403, 2020.
I. Osband, B. V. Roy, and D. Russo. (more) efficient reinforcement learning via posterior sampling. Advances Neural Information Processing Systems, pp. 3003 -3011, 2013.
R. Ouhamma, D. Basu, and O. Maillard. Bilinear exponential family of mdps: frequentist regret bound with tractable exploration & planning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 9336 9344, 2023.
V. M. Panaretos and Y. Zemel. Statistical aspects of wasserstein distances. Annual Review of Statistics and its Applications, 6:405-431, 2019. URL https: /tttpp::/doi.rgggg/ 1116///mmrrrrrtttaaiiii1--030111
G. Peyré. Computational optimal transport. Foundations and Trends in Machine Learning, 11(5-6):355 607, 2019.
M. Pirotta, M. Restelli, and L. Bascetta. Policy gradient in lipschitz markov decision processes. Machine Learning, 100(2-3):255 283, 2015.
M. L. Puterman. Markoe decision processes: Discrele stochaslic dynamic progrummingg John Wiley and Sons, 1994.
S. Raghvendra, P. Shirzadian, and K. Zhang. A new robust partial p- p-wasserstein- based metric for comparing distributions, 2024.
G. O. Ramos, B. C. da Silva, and L. C. Bazzan. Learning to minimisc regret route choice. In Proceedings of the 16th Conference on Autonomous Agents and MulliAgent Systems, pp. 846 855, 2017.
G. O. Ramos, A. L. C. Bazzan, and B. C. da Silva. Analysing the impact of travel information for minimising the regret of route choice. Transportation Research Part C: Emerging Technologies, 88:257-22 27577777,1 2018.
F. Santambrogio. Optimal Transport for Applied Mothematicians. Bikhauser Cham, 2015. ISBN 978-3-319- 20828-2.
H. Van Soijen, H. Nekoci, E. Racah, and S. Chandar. The loca regret: a consistent metric to evaluate model-based behavior in reinforcement learning. Advances in Neural Information Processing Systems, 33: 2020.
D. Silver, J. Schrittwieser, K. Simonyan, I. Antonogiou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis. Mastering the game go without human knowledge. nature, 350(7676):354-3335) 2017.
S. R. Sinclair, S. Banerjee, and C. L. Yu. Adaptive discretization in online reinforcement learning. Operations Research, 656:11633 166211666,,,,,,, 2023.
17

---

Published in Transactions on Machine Learning Research (05/2025)
M. Sommerfeld, J. Schrieber, Y. Zemel, and A. Munk. Optimal transport: Fast probabilistic approximation with exact solvers. Journal of Machine Learning Research, 20:1-23, 2019.
K. Sun, Y. Zhao, Y. Liu, B. Jiang, and L. Kong, Distributional reinforcement learning via sinkhorn iterations. arXiv preprint ar Xiv:2200.. 200-20000079, 2022.
R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction, 2nd Edition. MIT press, 2018.
C. Syed, M. Bowling, and R. E. Schapire. Apprenticeship learning using linear programming: In Proceedings of the 25th International Conference on Machine Learning, pp. 1032-1039. PMTR, 2008. URL https: //doi org/10. 1145/1390156 1390286
H. Tang, R. Houthooft, D. Foote, X. Chen A. Stooke, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel. #exploration: A study of count-based exploration for deep reinforcement learning. In Advances in Neural Information. Processing Systems, pp. 7753-2762. Curran Associates, Inc., 2017.
C. Villani. Oplimaal Transport Old and New. Springer Berlin, Heidelberg, 2009. ISBN 978-3-662-50180-11
J. Wang, I. Trummer, and D. Basu. Udo: universal database optimization using reinforcement learning. Proceedings of the VLDB Endoument, 14(13):3402-341) 2021.
P. N. Ward. Linear programming in reinforcement learning, 2021. URL htsp:://eschollarriip..ggggl 2a//domloads/2e555mm772 MSc thesis.
Y. Xie, X. Wang, R. Wang, and Zha. A fast proximal point method for computing exact wasserstein distance, In Uncertainty in artificial intelligence, pp. 433 453. PMLR, 2020.
Y. Yang, T. Zhou, Q He, L. Han, M. Pechenizkiy, and M. Fang. Task adaptation from skills: Information geometry, disentanglement, and new objectives for unsupervised reinforcement learning. In The Twelfth International Conference on Leurning Represenlations, 2024.
L. Zhang, K. Tang, and X. Yao. Explicit planning for efficient exploration in reinforcement learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 32, pp. 7488-7 1988-7497. Curran Associates, Inc., 2019.
M. Zhang, Y. Wang, X. Ma, L. Xia, J. Yang, Z. Li, and X. Li. Wasserstein distance guided adversarial imitation learning with reward shape exploration. In IEEE 9th Data Driven Control and Learning Systems Conference, pp. 1165-1170, 2020.
C. Zhu, X. Wang, T. Han, S. S. Du, and A. Gupta. Transferable reinforcement learning via generalized occupancy models, 2024.
18

---

Published in Transactions on Machine Learning Research (05/2025)
A Theoretical Analysis
A.1 MDP with Lipschitz Rewards
Given two metric spaces dx) and dy), function f X y is called 1-Lipschitz continuous if Villani 2009):
(f(()) dx(x,xx), v(r,x') E
(11)
This implies that the Lipschitz semi-nore over the function space F(X, Y), defined as
dy (dx((()),((a))) (Vxxx²) =x} x). dx(xxx²)
Iffl = sup aft
(12)
is When dx) Polish space and n.v E P(X), the Kantorovich-Rubinnn formula states that (Villani 2009):
(((un)) sup 1511.51 UsSdi 1 sdv (dde
(13)
= sup { (f()) - TE, [[(x))}
1
where Wi (1( is the 1-Wasserstein distance between y and with f the cost function.
Note that when ||/]|, < LR for any Lx > 0, then the function f is called LR-Lipschitz continuous, and Equation 13 becomes (Gelada et al. 2019
Wi(uv)) Le S
sup E [s()] E. [S(X))}
(14)
Now, we consider X = S x A, i.e. the state-action space, y R, i.e. the real line, and the function f to be the reward function R. Then, we can call the reward function R to be LR-Lipschitz if
Læds R((s a) R(s', (6
for all s E S, and a E A, and dsa((s, a), a')) = ds((s, + dalla, d')) being the metric on the state-action space S x A. If the reward function R of an MDP is LR Lipschitz, we refer it as an MDP with Lipschitz rewards.
A.2 Performance Difference and Occupancy Measures
We know that
JH 1 ( [R(s,a)] p
(15)
Using Equation
15 we write for two policies प and n', with u(s) as the initial state distribution,
1 JH J B(sn)an, [((s.a) Eaaaon a))s
(16)
Given an MDP with LR- tre-Lipchitz rewards, the Kantorovich-Rubinstei formula dictates that (Gelaaa ct al.
2019):
[Be,a)yv [R(8, a)] - B(e,o)= [R(s, = T.R (Vn: Unr)
sup
(17)
[[
By dividing both sides of Equation follows:
by A, and due an upper bound by the supremum, this inequality
LR W W (Vn var)
J Jr <
(18)
19

---

Published in Transactions on Machine Learning Research (05/2025)
A.3 Proof of proposition 1
The Linear Programming formulation for solving MDPs, assuming discrete state and action spaces, is Put-
erman 1994):
max Σ ()s,
8,0
subject to
Ux (s, -o((s) +7 Σ T(s 8 (s) s'.0
(19)
(8,0) 20 0 V(s,a) ESSA,
where Po(s) is the initial state distribution and T(s a' is the transition probability, The constraints of this optimization problem are often referred to as Bellman Flow Constraint.
A stationary policy π has a corresponding occupancy measure un (s, a) that satisfies the Bellman flow con- straint. (Syed et al. 2008), and hence æ and (=(5,a) share a bijective relationship (Syed et al. 2008 2021),
Givehi
x(a 8) = (n(s,a) a)
(20)
(s)
with
(=(()) = Σ Un (8, = po(s) + Σ
T(s
(21)
8'.a'
By rearranging Equation 20 to
(8. a) x(a (s)
(22)
and substituting Equation
22 into Equation
21
can rewrite Equation 21
(defining D
Σ. T(s
s', a))ar(a))) s')),
po()) = u (x) - 7 Σ
T(s 8, ()) IaaaasssJuaa (')))
.'.0
(23)
A (s) - pd (s (s')
which in matrix form is
P U (I - pP*) )
(24)
u,
where Po, Uy E IRIS| are column vectors and p ||s|i|S| are matrices. Solving for uy we get
u (I - -1-pp-)1 Po
(25)
PP exists because for < 1, (I PP*) is a strictly diagonally dominant ma- trix (Syed et al. 2008 Thus, (I -P-)- ( where E forms valid Neumann
The inverse matrix (II
series (Ward 2021). We let A = (r so Equation 25 can be written as u, = A We can
therefore express Equation 22 in matrix form as:
V, II @ (u 1)"
(26)
II (A 8 61)"
where II, V E SI 1 AR is column vector of ones, @ presents the Kronecker product, and c denotes the Hadamard product.
If we consider the case of a parameterized policy II),), then the derivative va with respect to
Vov, - II (p [ T @ =
O (p. (A 8 1)" +I +IO (P) 81)" =V (p) (A 8 1)a
(27)
+I (p AVo 01)"
20

---

Published in Transactions on Machine Learning Research (05/2025)
The first term in Equation 27 is differentiable since the policy is parameterized by 8. We expand V follows:
80
V R V
Vo (I+ + ( +...) - =0+ ((1"))))) +3( "0)p" +... (01" +...
+ + Vo(?P")
(1+ (1)"))))) Vo(TP=)
(28)
(t + (++1))"""/VVE
(t+1) (**))
Σ Tslbs a))((ass)
t=
p'a
= Σ (1+1))7PT)T
a) [ VoT((l'))
0
8'a
Since Equation 28 shows that A= is differentiable, VaV, is also differentiable based on Equation 27 Proceeding similarly, given the same conditions, we see that all higher derivatives of Va also exist with respect to 0. Thus, the space of parametrized occupancy measures u2 forms a differentiable manifold.
A.4 Proof of proposition 2
Regret a common metric for evaluating agents, that measures the total loss an agent incurs over policy updates by using its policy in lieu of the optimal one, defined as (Osband et al. 2013),
- Vsn(s)) -Vart)))va(t))
Regret = Beny
(29)
where V* Va is the value function of the optimal policy क² while Vaa (s) is the value function of policy Thy and a the initial state distribution.
Since J = B.-nl (V))) we can conclude from Equation 29 that
Regret = Eang
(vr(s)) - (s)
Ennp (V*(s) - Vv, (s))
Σ (Jn - -jh
(30)
JH -J
LR W/ Wi (Uxr, Uzc)
The last inequality due to Equation
18
21

---

Published in Transactions on Machine Learning Research (05/2025)
A.5 Proof of proposition 3
Let us begin the proof by defining the visitation probability at any step h E [H] in an episode, following policy {[]ss). Spocifically,
(s, a) - - 8, aq a) vh E [H] and ()) 0 vh M H.
8,
(31)
Thus, we rewrite Equation 7 of (s,a) + Σm= (s),))
Then, following (Kalagarla et al. 2021 can write the Linear Programming formulation for solving episodic MDP M as
rr(
max ( 2
R,,,a
subject to
Σ (s, = 10 th A
(32)
(s) = ={(as)p((s)
20
wh E (s, xSxAx
where u(s) is the initial state distribution and T(s aa) the transition probability. The constraints of this optimization problem are often referred to as Bellman Flow Constroints.
This implics that
H+1
Σ
(s, Σ
Σ
T(s
h=2
h=2 s'aa
H+1
#+1
(s) +
Σ
(s,a) = Σ
Σ (' (s'.a) + 2
1=2
h=2 8/
H-1
H+1
(8, = Σ
Σ
T8818 h- (d,,a) + Σ (3,
1=1
1=2 8'.0
H+1
(s a) Σ
E(( a)
Σ
1 (5', a)) + p(s)
8'
1=2
H Σ (8, a) = H Σ
T(s 2 + u(8)
8
Σ (s)a) = Σ
T(s a)u (s a) + 1
H m(s)
s¹.a
#
(s, a)
Σ T(s ET(s| 8 (a (5') + m(())
(33)
Now, denote / and P as corresponding column vectors and the transition matrix p Ev,a T(s ((s1s)) Thus, obtain
H (I -I- P J
H n
(34)
We can therefore express the finite horizon occupancy measure in matrix form as
H = II +
(35)
where VI,,, v E IXAAI RIAl is a column vector of ones, & presents the Kronecker product, @ denotes the Hadamard product.
22

---

Published in Transactions on Machine Learning Research (05/2025)
If we consider the case of parameterized policy II)),, the derivative of v with respect to 0 is
Vov
=V II (u @ 1)
=V,I ( ) 1) + V ( @1)*
(36)
=Va11 (u @ 1) + (Vou 81)"
The first term in Equation 36 is differentiable since the policy is parameterized by 0. We show that Vou exists using Equation 34 as follows:
Vo [(T - P Ju H Vo (11-P")uu + (I P V
V
(37)
- -VPP" + (1 V =0
(I - P" Vou V PoP"
we observe that (7- p*)-1 may may exist. To address this, we the group inverse, generalized matrix inverse that extends the concept of inversion to both singular and invertible matrices Drazin 1958 For square matrix C E Raxx, its group inverse C# satisfies the conditions CC# C = C, C CC# = C# and CC# = C#C
In Equation 37
The group inverse useful in analysing Markov chains Meyer 1975 and coincides with the standard inverse when C is invertible. Using group iuverse (II - P* p*)# Equation 37 can expressed as,
H o = (T- Vou
u w
(38)
where w is a vector in the null space of (I P"). Note that w = for ergodic Markov chains (Meyer) An episodic MDP induces an ergodic Markov chain that admits a unique stationary distribution Bojun 2020 (see also Appendix B.3 Thus, Equation 38 simplifies to
1975
Vou = -1--p V
(39)
= (I-P** p #
Σ a(sss, u s/.a
(40)
which shows that Vou is differentiable and so is Vov Procceding similarly, given the same conditions, we see that all higher derivatives of v also exist with respect to 0. Thus, the space of parametrized finite-horizon occupancy measures of forms a differentiable manifold M
A.6 Optimal Transport Dataset Distance (OTDD)
Suppose we have two datasets, each consisting of feature-label pairs, DA = ((t),, w ~ Pa u) and DB = (tt) u/)) ~ Pt(t, u) with ta, E T and uA, up E UA, These datasets can used to create empirical distributions PA u) and Pt(t, u). OTDD is the p- D-Wasserstein distance between the datasets D and DB which essentially the distance between their empirical distributions PA and PB with the cost function defined as the metric the joint space x U (Alvarez-Melis & Fusi 2020
Naturally, the metric on this joint space can be defined as drzu((t, u), (t', u()) = (d, t'p + dulu, )'')
for 2 1. However, in most applications dr is readily available, while du might be scarce, especially in supervised learning (SL) between labels from unrelated label sets (Alvarez Melis & Fusi 2020p. Further, we want dy and dy have the same units be addable. To overcome these issues, dy expressed in terms of dr by mapping labels u to distributions over the feature space P(T) as u (T) & P(T U u) E P(T). Therefore, the distance between the labels and u' is defined as the p-Wasserstein distance between a. (T)
23

---

Published in Transactions on Machine Learning Research (05/2025)
and au (T),
du(u, u') (n
( min
(elllaa / (dr ((((())))(((((()) 1/P
(41)
The metric on the joint space becomes,
drullt, u), (drr), o (T)))
(42)
Let 2 = x U, then the p- D-Wassersteinn distance between Pa(t, u) and Pra(t u) is 'nested" Wasserstein
distance:
min =EU(PA.PP) Jzzz I (dz(z,2))) dm
Wg(PA, PB)=
(43)
min RETT(A.PP)
I (dT(t, + W (au, (aur, Ow)) dr
Wp(PA, PB)) the OTDD between datasets DA and DB, often expressed (Dr, This is used in transfer learning to determine the distance (or similarity) between datasets. Figure t illustrates OTDD when applied to RL using datasets of state-action pairs.
D. = {si, a=)
Dr = ((5)/
a²)
n J==
aa' (S)
Create Empirical Distributions
aaa (S)
Wp(:,-)
da = Wp ag)
Un (s, a)
Wp(.,.)
WP U
Un (s' as)
Figure 6: Illustration of OTDD when applied RL.
A.7 Proof of Proposition
We compute the error in occupancy mcasure for both the infinite and finite horizon cases. In infinite horizon MDPs, the occupancy measure is defined as the expected discounted nunber of visits of a state-action pair (s, a) in a trajectory (Laroche des Combes 2023): f = (1-7) Ei y'pt, where μ4 = P(st, az T,17) the state-action probability distribution at time step with the initial state distribution r following the policy w. In finite horizon MDPs, the occupancy measure is the expected number of visits of state-action pair (s, a) in episode of length H (Altman 1999): μ E
24

---

Published in Transactions on Machine Learning Research (05/2025)
First, derive error bounds for the infinite horizon MDP in which y 1 and the occupancy measure is approximated using finite number of samples collected up to a finite number of time steps T. Later, we derive error bounds for the finite horizon MDP.
A.7.1 Infinite Horizon MDPs
Estimated Occupancy Measure. For convenience, express the occupancy measure as / (1 n) Et=o E where 14 P(st,a; π) is the state-action probability distribution at time step with the initial state distribution following the policy æ. To compute μ, we roll out N episodes (each of multiple time steps) using π, and take number of samples at t to approximate Pt. Thus, the empirical occupancy mcasure i is given by f = P Et=o T where Ei
Note that the total number of samples in the
policy dataset Dr is [D+] = 11).
Occupancy Measure Estimation Error. Consider two occupancy measures 1 == (1 (1-7) Er 7 Wh and Ei I'v (with estimates E I'p N₂ and & p >' For independent sets (22 and (u4)120, the Wasserstein distance has the following additive property (Panarctos & Zemel 2019)
Wo Me, Σ vc) s Σ mplum,
(44)
While for a E IR Paanaretos Zemel 2019).
Wp(ap, av) |a|Mi(u,, v)
(45)
Therefore, for our scenario where p 1, the Wasserstein distance between μ and given by:
Wi(un)) W1((1
y' Me, (1 - 7) Σ v'v))
1=U
t=0
(46)
$ (1-7) Σ t=0
7" (1rs)))
while for f and D,
T w((,D) S Σ t=
7 Wi (2t ,,N))
(47)
In the RL problems we consider, the state-action space Z = S x A is commonly defined as the subset of the Euclidean space R², where usually B 2 2. Theorems and (Sommerfeld et al., 2019) establish the following error bounds between the true and empirical probability distributions,
(A (e) < N (DN v1) S EN
(48)
where
2 + (1/2)log, |2|
B 2 if B > 2
&g S 1/2 di)
/211/211/1 1 [2 1/(22/2-1
Note that 2 and d2am(Z) denote the cardinality and diameter of 2, respectively. Suppose (a, P), b (M, n), Wi(P,u). d = Wi v), and Wi v). Then by performing two reverse triangle inequalitics,
la cl < b and 1c-
(49)
la - d < bee
25

---

Published in Transactions on Machine Learning Research (05/2025)
Equation 49 implies that,
(I) = = BMI(A + Wi(
T Σ
=E/Wi(P Σ
YR N 10) + Wi(p
yDD .)]
1=
1=0
= =e[Wi(p) Σ
1/4 No -1) (D Σ
2D, .
(50)
1=0
+ EMMi((1 7)
A N ,1) -Wi(11 -7) Σ
2' Ni .)]
1=1
1=0
+ [mWi((1-1)
Σ
2/ ,D) - W1((1 - 7) Σ
y
1/
No v)
t=0
1=0
By virtue of triangle inequalities, get
T
T
Wile
t Nu.
M4 (1 7) Σ
7 Ar No) 2 mi
Σ
p'p .4) - Wi((1 77) Σ
7
Nu
14)
t=
1=0
1=
t=0
(51)
Wile
No. (1 77) Σ "DNN 2 mi Σ
ToNN -
Σ
/ v)
t=0
(=0
1=0
1=0
Therefore, the right-hand-side (R.H.S) of Equation 50 can be further simplified as
T
20
R.H.S & (p Σ
7 U4
No
(1 - 7)) Σ
- Em Σ
2DN No (1 -17) 7) Σ 7 ('
t=0
1=0
t=0
t=0
(52)
20
+ +E|W - 7) Σ
No [d] + B( (117) Σ 2'DiN U1]
1=0
1=
For simplicity, we denote P0 = (1 7) Er= r Ar (similarly 2xg and Pur = p Et P'p
No (similarly Pr).
where p Ei. 7t 1- Using Theorem 4 in Gibbs & Su 2002), the 1-Wasserstein metric Wi and the total variation distance drv satisfy the following,
Wisss AT) S diam( Z) drv dro(poo
diam (2) 2 2E2
1
Σ (2) - PT(2))
(53)
26

---

Published in Transactions on Machine Learning Research (05/2025)
However,
T
A100 - PT = (1-7)
Σ
44"
1- TT+
1 1 7
t N
14
1=0
t=0
= (1 =(1- (1-7)
30 Σ
1 1/4 N2
1 - 7
7 P4
No.
7 xT+1
1=
t=0
+(1 7)
Σ
7 Pt
No
-(1 7) Σ
? "
Nr
T
T
= -(1-7)
Σ F
Σ
+ (1-1))
1 1-7 γ
1 - T++
(54)
1=0
1=0
T
= (1 7) Σ
2 Pr No -g T+1 - 7
Σ
p'p N,
1 y T+1
T=
t=0
(1 - 7)
1'P Np
(=T+1
T+1
?'p No
I=
-T fr1+1.00
where normalizes ΣiSTH 7' No. We utilize Equation
"r"
54 in Equation 53 as,
(i()o,, diam(2) 2)
Σ
| px)())
2
=E2
S diam(2 2)
Σ
17
T+1 (x)
(55)
=E2
? T+
diam(z)
2
Equation 55 also applies for Wi (D0, 0T), therefore by substituting these into Equation 52
R.H.S < E[ 7)
Σ
7 P No .H]] + E[MWi((1 77
Σ
"ONN, + T+ +1 diam(Z)
1=0
1=0
8
= [(1 7) Σ
M No ,(1 7))
Σ
1=
1=
(56)
20
20
+E|Mi((1 - 7) Σ
7 A0N0
,, (1 7)
Σ
7 vvy] + T+2 d2
1=0
S 7) 7' (E ",p)) + EM No. (20)) + + diam
By substituting Equation 48 into Equation 56
R.H.S < (1 7) Σ 7dt (E2N + + t) +
1=0
(57)
= (Nz 1/2 + No )) + "3ddmdam(2)
27

---

Published in Transactions on Machine Learning Research (05/2025)
Therefore, Equation
50 becomes:
- < & (N + No $) + diam(
+
(58)
Over the full trajectory in the occupancy measure space. The true distance between consecutive policies n. and u441 after an update is Wi (1x 1,11)), ( which is induced by the 7th policy updale. We estimate this distance using datasets of the policies, i.c. approximated distributions, using Wi (i 0m41))
For M roll out episodes of cach Ri, use Equation 58 with N₁ N M, to derive the following error bounds,
E (Va, Umpn)) - Wi (6 Pm ) < 28 + 2)
diam
(59)
which consistent with learning from Dr; and then D By summing sequentially through policies encountered during RL training, we compute the total distance over a path of N segments obtained via policy updates:
E Wi U Wi Wi 0m44 )l s 2NE diam Z))
(60)
1=0
Since 0e S E. |xce| then,
2NE VM
E
Wi (V URa1)
Σ
Wi(Oo, 0Rs11)
NN diam (
(61)
A.7.2 Finite Horizon MDPs
Occupancy Measure Estimated Error. Consider two occupancy measures / =
B E ue and v
ΣE i. with estimates f 1 Σt NN and & 1/ Etn Di From Equation 49 have
E WI D) - Wi
< (A, 11) + (1,
= IMM _
N2 H
1
Σ
mc)
+ Wi(
1 No.
1 F
Σ
mll
t=1
t=1
(62)
1 H
E|Wi(At No [c)
Σ
11
s & N + No
Therefore for the total path in the occupancy measure space with M roll out episodes of each T; the error bound is
N
N-1
2
E
Wi(ueys Uaet1) - Σ
Wi Dres))
(63)
VM
by assigning N₂ N, M in Equation
62 which concludes the proof.
28

---

Published in Transactions on Machine Learning Research (05/2025)
A.8 Proof of Proposition 5
By definition of naub, we get
2 Wi(Un, Un(Ti) ) + Wi (UTNNNI UAN)
vauh
Wi (W UNN)
EN= Wi (V Mx(+1) + W1 (UTTNN-- VTN)
Wi (Uioo Ux+)
x
Wi (1 vx-)
Wi (Uro Vayy) )
2i=o Wi Wi Mret1) W1 (U Ux+)
Wi W1(UNN Unr)
Wi (Uaoo uxx)
Wi (Uror
Wi (Wnos UxN
WilUrN 1xr) 11 (1 (ay)
Wi (Uzo: Up
(64)
W1 UgN)
The inequality above is true due to the triangle inequality W1 (1(UxN-1 UnN) + Wi (UNN (xr) 2 Wi (1 Ug-). By applying triangle inequality, we also get
Wi Ux-) + WI(UNN Un-) ) 2 W1 (Uio: Unn)
This implies that
Wi (UmO: Unr)
Wi (U Un-)
(65)
Wi (U UnN)
W1 (U UnN)
Equation
64 and Equation
65
together yield
Wi(UnN Ux-))
Wi WilUaN Umr)
nsub
11
Wi(UTO UnN)
Wi ViTN
Wi Uxr)
W Wi(UNN Upre)
(W) (Uryy Uxr)
=
Wi UxN)
Wi (Uxor Uay)
+
Wi (Uigg Uxx)
(V Uxr)
Wi (Ungg Unr)
2 1) 1
Wi(UTOO vxx)
Wi UnN)
2W, (VAN Un-))
2 1
Wi Uxx)
Wi yo Wi (Ung (ix
2
The second last inequality is due to non- non-negativity that 7) 1.
The last inequality is due to the fact
Thus, conclude that
1 Noud
2W (WaNs Un-)
1
Wi (Uro, Vax)
A.9 Wasserstein Spaces as Geodesic Spaces
Given probability measures v, E P(X) on a metric space X' C RB with metrie dx(x, x²) ('), the Wasserstein distance (,(,,,) v) is the minimal transport cost for c(z, z') = (dx(x,2)) z') with p 2 (Villani 2009 The Wasserstein distance V)(,,v) takes distance on X' and creates out of it distance on P(X)) Peyré 2019 Proposition 5.1 in (Santambogioo 2015) assorts that Wp is distance over P(X))
Definition A.9 Wasserstein Space). (Santambrogio 2015) Given Polish space X,, for each E (1),,),
the space P(X)
endowed with the distance Wp is a Wasserstein space W, of order p.
Theorem 5.27 in (Santambrogio 2015 states that if X is a convex space, then the space Wp is geodesic space (length space). Thus, the geodesic (shortest path distance) between u, E P((X)) is given by Up(u,v
29

---

Published in Transactions on Machine Learning Research (05/2025)
Kolouri et al 2017 It mentioned in Appendix A.7.1 that the RL problems we consider consist of the state-action space 2 = S x A E RB B > 2 (subsets of the Euclidean space). Given that Euclidean spaces are convex spaces (Boyd & Vandenberghe 2004), our space of occupancy measures M is a Wasserstein space W1 = (M, Wi) and thus a geodesic space. Therefore, Wi (1, 2/) measures the shortest path on the surface of the manifold M between probability distributions l and v.
30

---

Published in Transactions on Machine Learning Research (05/2025)
B Additional Experimental Analysis and Results
B.1 Environment Description
2D-Gridworld environment of size 5x5 with actions: {up, right, down, left). The start and goal states are always located at top-left and bottom-right, respectively. In the gridworld, we perform experiments on three settings namely:- A) Determúnústic, dense rewards setting: State transitions are deterministic. The reward function is given by ||s: -syl1,, where is the agent state at timestep and 8y is the goal state. B) Delerministic, sparse rewards selting: State transitions are deterministic and all states issue a reward of -0.04 except the goal state with reward of 1. C) Stochastic, dense rewards setting: Actions have a probability of 0.8 in the instructed direction and 0.1 in each adjacent direction. Reward function is as defined in setting A.
2D-Gridworlds (Task Difficulty). Figure 7 depicts the configurations of the 5 tasks that were used to assess ESL with respect to task hardness. They are all deterministic with actions: {up, right, down, left), and mostly have the start-state at the top-left and the goal-state at the bottom-right with only one task that has the goal-state at the center. In the order of appearance: a) [5x5] dense: has size 5x5 and dense rewards, b) [5z5] sparse (hard): has size 5x5 and sparse rewards, c) [5a5] sparse (easy): has size 5x5, sparse rewards, and goal-state at the center, d) [15x15) dense: has size 15x15 and dense rewards, and e) [15a15] sparse: has size 15x15 and sparse rewards. The reward functions for both dense and sparse rewards are previously described above for 2D-Gridworld.
a) Denze Rewards
b) Sporse Dewards
c) Sparee Newards
d/Dense Rewards
Sparse Rewards
start state
goal state
Figure 7: Five gridworld tasks with the same action space, but different rewards, state space and location of the goal state.
B.2 OMR(K): OMR over number of updates
OMR is defined for the entire policy trajectory by Equation 0 as,
k A ΣtEK: Wi(UEN Uoy 1)
EN-1 Wi Urats)
To observe how it changes with respect to updates, compute OMR from update i onwards till the end of the learning trajectory, i.e. over subsequences with a decreasing mumber of policy updates with increasing i, using:
ΣteK! k2i Wy Wi Uwets) NN1 Wi(Wre Uroez)
x(i)=
such that i i N0 T]
(66)
31

---

Published in Transactions on Machine Learning Research (05/2025)
where T = 0.9.N to ensure that the last subsequence of policy updates have at least 10% of the total updates in the trajectory.
B.3 Computation of Occupancy Measures
The finite-horizon occupancy measure is defined as A Altman 1999
1 H
of (8,
P(s: 8, az n.
for which the probability of the state-action pair selected is time-dependent. If we restrict analysis to stationary policies where T(als)) <{ass), then the probability of the state-action pair becomes time- independent and thus,
# (s, P(8, 14
This implies that the of stationary policies in finite-horizon MDPs, observed in practice with many
episodic MDPs (Memmel et al. 2022 Aleksandrowicz & Jaworek-Kon Korjakowska 2023 Liu 2023), induces stationary occupancy measures- where the expected number of stale-action pair visits are independent of the time-step. Work by [Bojun 2020) provides extensive details about the existence of stationarity in episodic MDPs and shows (in Theorem 4) that,
( [R(s.a)] = Ec-M= E1 R(s;, at)
(67)
BxnM, [H())
where C is the episodic state-action pair trajectory, H(S) is the episode length corresponding to C, and Mr is the markov chain induced by policy т. We verified the correctness of our v computation by calculating the relative error derived from Equation 67 to check its validity. The relative error given as
E (e,a))ool
R(ss, a)] Be-M, [H(()] - -ECMM
ΣnO R(s at)
Rel. Error %
1000
(68)
EswM, II(O) R(st,at)
Figure depicts Rel. Error% vs number of updates in the stochastic 2D- DD-Gridworld environment with dense rewards. We observe that increasing the number of rollouts M reduces the estimation error of v. For M 10, the absolute relative error can be as high as 50% with the mean less than 10%. While for M = 500, the maximum absolute relative error is 4%.
-5
-0.5
-10-
Rel. Error %
-1.0
Rel. Error
-15
-1.5
-20
-2.0
-25
-2.5
-30
-3.5
-00
200
400
600
800
200
400
600
800
#updates
#updates
Figure 8: Rel. Error% vs number of updates plots in the 2D-Gridworld environment where "f is estimated using M = 10 rollouts (left) and M = 500 rollouts (right).
32

---

Published in Transactions on Machine Learning Research (05/2025)
B.4 Effects of the number of rollouts SAC
The policy dataset Dr, in a deterministic environment is made up of (s,a) pairs generated from a single episode of the policy n.. In deterministic environment, this sequence remains the same across repeats of episodes, for each policy क, (deterministic) update step Therefore, a single rollout is sufficient estimate the occupancy measure Ux,- In stochastic environment, rollouts impacted by the environment's stochasticity. Thus, multiple rollouts are needed to estimate the occupancy measure accurately, As the number of rollouts increases, the occupancy measure should converge and become less noisy.
Table 5 shows that, in a stochastic setting, the EST, values converge as the number of rollouts increases. OMR appears to be invariant across various the number of rollouts, and the mean number updates appear be consistent around 2900 (with exception for #rollouts = 1). The results indicate that from about 6 rollouts, the estimated occupancy measures become less noisy. This aligns with Equation 61
which shows
that increasing the number of rollouts reduces estimation error.
#rollouts
ESL
OMR
UC
849.1+-46855
0.500=#0.004
1849:4742.2
618.6=257.3
0.501=00005
2413+1397
245.8
0.501=:0002
2462-+2043
423.14
0.503=0.004
2281
Table 5: Evaluation SAC algorithm the stochastic, dense-rewards setting for 5x5 gridworld with 40 maximum steps per episode across various number of rollouts. The effects of #rollouts on the Effort of Sequential Learning (EST.), Optimal Movement Ratio (OMR), and number of updates to convergence (UC) are observed.
B.5 Neab can be a reasonable proxy for 7), when optimal policy not fully reached
We compare ESL when the optimal policy was reached, denoted 7,, versus when it was not, denoted 7sdd; in Tables and First, we observe that the number of rollouts impacts the metric values. Second, T/ub values are always greater than values. Note that UCRL2 and PSRL update their policies only at the end of each episode, whereas SAC and DQN update theirs after each time step. Hence, UC = 499 for both UCRL2 and PSRL.
The ESL values (both 7) and Nasb)) in Table are lower than those in Table expected since more data samples reduce estimation error. The distance from the initial policies to the final polices are not different. Using both Tables 0 and 7 we notice that comparing algorithms with /sub yields the same efficiency ranking (e.g. PSRL, UCRL, SAC and DQN) as This indicates that Noub reliably predicts results provided by т for comparing a algorithms.
The results presented in Table P for stochastic dense-rewards setting are consistent with those Table because the number of rollouts used was Nr 6.
B.6 Effects of Hyperparameters- UCRL2
Table g illustrates the effects of hyperparamcter values in the UCRL2 algorithm. The environment is deterministic dense-rewards setting with 200 training episodes. We observe that high exploration rates (5 0) appear to align with high ESL and UC, while high exploitation rates (8 1) appear to align with low ESL and UC. OMR appears to be invariant across various d values.
33

---

Published in Transactions on Machine Learning Research (05/2025)
Algo.
7/
Naanh
d
UC
UCaub
849
3623
5.631
5.263
1850
745
SAC
468
4166
1.50
2.10
742
3535
230E
613E
5.651
5.451
284
499E
UCRL2
155
999
0.93
2.15
180
0.0
86.24
389
4.96
5.294
97.24
499
PSRL
44.4
102
1.26
1.49
52.5
0.0
5641
39111
5.521
6.541
12131
9U9TE
DQN
478
1710
1.39
2.05
1061
1904
Table 6: Evaluation of algorithms in the stochastic, dewse-rewards setting for 5x5 gridworld with 40 maximum steps per episode with the number of rollouts Nr = 1. The total number of training episodes is 500. When the algorithm converged at optimality, 7) is the Effort of Sequential Learning, d = Wi (T x²) is distance from initial policy to the optimal policy, and UC is the number of updates to convergence. When the algorithm did not converge at the oplimal policy, rather a non-oplimal NN, we use T/sub; C = Wi (To. and UCeub denote the aforementioned quantities. 40 training trials were used.
A Algo.
Jkub
c
UC
UCsub
445
8532
5.63
7.26
2463
6293
SAC
246
127
1.23
1.45
2043
441
1984
5104
3.36
4.58
268
499
UCRL2
121
274
0.84
1.90
155
0.0
50.4=
361
4.974
3.91 +
76.1
499
PSRL.
33.6
43.6
1.34
0.48
50.6
0.0
4584
1971E
4.88#
6.521
1586E
13713E
DQN
311
250
1.06
0.31
1077
6907
Table 7: Evaluation of algorithms in the stochastic, dense-rewards setting for 5x5 gridworkd with 40 maximum steps per episode with the number of rollouts Nr 6. The total number of training episode is 500. When the algorithm converged at optimality, 7 is the Effort of Sequential Learning, W x²) is distance from initial policy to the optimal policy, and UC is the number of updates convergence. When the algorithm did not converge the optimal policy however some nN; we Tssub (T Tx), and UCrub to denote the aforementioned quantities. 40 training trials were used.
ESL.
OMR
UC
SR%
0.1
47.7611.776 0.512:10033
62.26119.977
100
0.3
39.29+5.860
0.51540.034
58.0847.746
100
0.5
6..26
0.511+0.036
56.92+99111
100
0.7
37.4815.004
0.50740.029
56.6847.460
100
0.9
36.40-5.301
0.51040.036
54.86+7.326
100
Table 8: Evaluation of UCRL2 algorithm in the deterministic, dense-rewards setting for 5x5 gridworld with 15 maximum steps per cpisode. Different confidence parameter 5 E (0, were evaluated to see their effects on Effort of Sequential Learning (ESL), Optimal Movement Ratio (OMR), number of updates to convergence (UC), and success rate (SR). Note that å 0, the agent approaches absolute exploration, and with 8 absolute exploitation.
B.7 Extended Discussion of Usefulness of ESL and OMR
The quantities like regret and number of updates (UC) are outcomes of the exploratory processes, and thus reflect only a partial view of the underlying exploration mechanisms. We propose ESL and OMR to complement regret and number of updates as metrics but not to replace them.
1. Complementarity ESL and OMR with respect to UC:
a. Case 1. Let us consider two RL algorithms that reach optimality with the same number of updates, i.c. they have the same UC. How would be able to distinguish the exploratory processes of these algorithms?
34

---

Published in Transactions on Machine Learning Research (05/2025)
ESL and OMR are the summary metrics of the policy trajectory during learning. These can reveal which algorithm's 's exploratory process is more direct versus meandering, smooth versus noisy, or has large versus small coverage area in the policy space (Figures 3 and 4 top rows), Therefore, ESL and OMR quantify with granularity the characteristics of the exploratory process of an RL algorithm for any given environment.
b. Case 2. Let us consider the case when optimality is not reached but the maximum number of up- dates is attained by two RL algorithms. How would one be able to evaluate the exploratory processes of these algorithms and systematically uncover which exploratory process demonstrates desired characteristics? Looking into the training trajectories of RL algorithms in environment and corresponding higher /lower ESLs (Mou., Section 4.2) we can make knowledgeable choice of an RL algorithm exhibiting desired charac- teristics (e.g. high coverage, smooth exploration). We have shown in Section 4.2 and results in Section 5.3 (also Appendix B.5) that ranking based on suboptimal ESL aligned with true ESL, and additionally, the visualization of the training trajectories (Figures and can indicate the characteristics of corresponding RL algorithms even when optimal policy is not reached.
Experimental Evidence. UCRL2 is known to be provably regret-optimal and is designed to continuously explore. SAC does not have such rigorous theoretical guarantees but is known to be practically efficient. In Table 1 by UC, we observe that SAC is significantly suboptimal than UCRL2. But SAC has lower ESL than UCRL2 as its exploration is smoother. Additionally, OMR for SAC is higher than that of UCRL2 They together indicate that SAC takes smoother but larger number of policy transitions aligned to optimal direction for exploration, while UCRL2 exhibits bigger policy changes and in diverse manner trying to cover the environment faster.
2. Complementarity of ESL and OMR with respect to Regret:
UCRL2 and PSRL have the same order of regret bound (Osband et al. 2013). But PSRL leads to smoother policy transitions that are much more orientated towards optimality (as shown in Figure 3). while UCRL2 leads to less smooth policy transitions that do not taper as it approaches optimality. This information is not evident from regret but from corresponding ESLs and OMRs (Table 1
3. Insights for Algorithm Design:
Knowing ESL (or suboptimal ESL) and OMR can assist with developing algorithms that emphasize certain exploratory characteristics. We can develop algorithms with grades of coverage or directness, while also being able to visualize this. Ultimately, depending on the environment, we can choose which characteristies of exploratory process are well suited. In contrast, looking only at the final outcomes RL algorithms like regret and number of updates does not include these nuances.
35

---

Published in Transactions on Machine Learning Research (05/2025)
C Specifications of the RL Algorithms under Study
C.1 Methods for simulation results (Discrete MDP)
Model parameter initialisation. We initialised model parameters for deep learning RL algorithms like DQN and SAC by uniformly sampling weight values between -3 10-4 and 3- 10 and the biases al O. For tabular Q-learning algorithms, randomly initialized the Q-values between -1.0 and 1.0. For UCRL and PSRL, the policy model was randomly initialized. Note that all Wasserstein distances were computed using python package POT (Flamary et al. 2021 Additionally, L1 norm was used in our Wassenstein metric cost function as the ground metric for the 2D gridworld environment.
Results in Figure The problem sctting was deterministic with denso-rewards and 15 maximum number of sleps per episode. The total number of episodes was 200. The convergence criterion satisfied when maximum returns were produced by algorithm over 5 consecutive updates. The results showcase single representative run of each algorithm. The confidence parameter 8 = 0.1 was utilized for UCRL2. The parameter for SAC was autotuned using the approach in (Haarnoja et al. 2019) along with hyperparameters described Table g While DQN began with = 1.0 and the value decayed as <(t + 1] max (0.9999 x
e[t], 0.0001). Table shows hyperparameters for DQN. Note that the ADAM (Kingma & Ba 2017 optimizer was used all the neural network models.
Table 9: SAC Hyperparameters.
Parameter
Value
learning rate discount(7)
10
0.99
replay buffer size number of hidden layers (all networks) number of hidden units per layer number of samples per minibatch nonlincarity
10a
1
32
64
ReLU
entropy target
-4
target smoothing coefficient (r)
0.01
target update interval
1
gradient steps initial exploration steps
1
before model starts updating
500
Table 10: DON Hyperparameters
Parameter
Value
learning rate discount(7)
10 2
0.99
replay buffer size number of hidden layers (all networks)
10a
1
number of hidden units per layer number of samples per minibatch nonlinearity
32
64
ReLU
target smoothing coefficient (T) target update interval gradient steps initial exploration steps before decays
0.001
1
i
500
Results in Tables 1 and 2 The problem settings had 40 maximum number of steps per episode, and the convergence criterion satisfied when maximum returns were produced by algorithm over consecutive
36

---

Published in Transactions on Machine Learning Research (05/2025)
updates. The means and standard deviations for each algorithm were computed over 50 runs. The total number of episodes was 200 for results in Table and 500 in Table 2 For results in Figure 5 the Q-learning with decaying e-greedy where = 0.9 was employed in the gridworld tasks described in Appendix B.I A convergence criterion of 50 consecutive model updates with maximum returns was utilized. We aggrogated the result over 40 training trials and the maximurn number of steps per episode was 60.
C.2 Methods for simulation results (Continuous MDP)
Model parameter initialisation. We initialised model parameters for the deep learning SAC algorithm by uniformly sampling weight values between -3 10-4 and 3 10 and the biases at For the DDPG algorithm, the output layer weight values were initialised using Xavier Initialization (Glorot & Bengio 2010), while the rest were uniformly sampled between -3 10 and 3 10 This was done both the actor and critic networks. The ADAM (Kingma & Ba 2017 optimizer was used in all the neural network models. In both algorithms, 1) discount factor y = 0.99 was used. 2) 500 initial steps were taken before updating
model weights, and 3) replay buller size was 10 Tables 11 and 12 display hyperparameters for DDPG and SAC, respectively.
Results in Figure 4. The problem sctting was Mountain Car continuous (Moore 1990) with 999 maximum number of steps per episode (Brockman et al. 2016). The total number of training episodes 100. The convergence criterion updates. The results showcase single representative run of each algorithm. For results in Table 3 the mean and standard deviations for each algorithm were computed over 5 runs. While RL training was conducted in a continuous state-action space, we ized it for Wasserstein distance calculations between occupancy measures, using bins for actions and bins for states. Note that all Wasserstein distances were computed using a python package POT (Flamary el al.] (021).. Additionally, L2 norm was used in our Wasserstein metric cost function the ground metric for the Mountain Car environment.
satisfied when maximum returns
produced by algorithm
10 consccutive
Table 11: DDPG Hyperparameters..
Parameter
Value
number of samples per minibatch
128
nonlinearity
ReLU
target smoothing coefficients (7)
0.001
target update interval
1
gradient steps
number of hidden layers (all networks)
2
number of hidden units per layer
64
Actor learning rale
5 10-4
Critic learning rate
5 10-3
Table 12: SAC Hyperparameters.
Parameter
Value
learning rate
3. 10
number of hidden layers (all networks)
2
number of hidden units per layer number of samples per minibatch nonlinearity
64
128
ReLU
target smoothing coefficient, (T)
0.001
target update interval
1
gradient steps
1
37

---

Published in Transactions on Machine Learning Research (05/2025)
D
Supplementary Results
In this section present enlarged versions of results in Figure (see Section D.1 and additional plots that support the results in the main paper (see Section D.3.
D.1 Enlarged Visualisation of the Occupancy Measure Trajectories
Figures 9
enlarged versions of enlarged versions of Figure For each algorithm, there is visuali- sation of the policy trajectory and visualisation of the state visitation below it.
O)-greedy
(E = 1)-greedy
50
500
000
100
150
2000
2500
stepwise
stepwise-de Hepw/se-distan distance
1440g 520
2 wewpiis--dsstaee distance
distance-te tooopimaa
12 14
distance-to-optimg optimal
In
17.22
11:79
7.52
116
11
32.31
0.6
0:52
0224
014
3
5:38
5.1
2
208
33.43
G.54
0:47
6:7
012
2:40
1.71
1:44
Figure 9: Top row: Scatter plots of distance-fo-opptmmal and stepwiseedistance over updates for c(=0)-groody and e(=l)-greedy Q-learning. Bottom row: State visitations.
38

---

Published in Transactions on Machine Learning Research (05/2025)
UCRL2
PSRL
dupdeece
Sagepdig
20
1.0
60
stepwise distend disptiseddsttanc
8 420
14, 14 6 A2d stepwise wetpw//eddstanee.
distance- distance-to-opoee to-ottimal
10 12
distance
10 12 distance-to- tottiia
2
Vistat 1%
State Vidiation (P)
5.70
91
21 63
30c
100
222
110
6:33
541
4.40
2.35
2.26
2.27
1
2:53
3.30
13c
3.34
4.4
1.20
0.71
0.16
3:35
0
Figure 10: Top row: Scatter plots of distaacce-lo-pliine and slepuiiee-disance over updates for UCRL2 and PSRL. Bottom row: State visitations.
SAC
DQN
sajepdni
Fapeddng
20
20
50
60
stepwise Seepwis--ddiate distance
14
stepwise distance
34408
2
distance- toptimal
2 10 12 14
distance to-tiila
10 12
2
State Visiatt on (%)
State Vistation (%)
10
4:4
9.11
7.24
1.11
9
9.46
6:35
124
1.17
5
232
1.74
2.31
1:9
3.70
431
0.15
1:59
3/12
Figure 11: Top row: Scatter plots of distance-to-optimal and stepwise- distance over updates for SAC and DQN. Bottom row: State visitations.
39

---

Published in Transactions on Machine Learning Research (05/2025)
D.2 Performance Plots
This section contains Return plots of the algorithms. This allows us to assess the learning algorithms from the performance perspective. Figures 12 and T3 depict performance cvolution that corresponds settings in Figures and respectively. Note that while all algorithms find the optimal policy, UCRL2 and cg=1)-greddy Q-learning fail to remain there if training continues without truncation. As result, their performance does not improve over time compared to those that stabilize at the optimal policy.
(c- (i-00) greedy
(e-1)y
DQN
-20
60
Return
-00
-140
50
100
150
200
250
Episodes
Figure 12: Return plots of algorithms: -reodyr and (=1))-reody Q-lcarning, UCRL2, PSRL, SAC, and DQN averaged over runs in the deterministic 5x5 Gridworld with dense rewards.
ddpg
SBc
60
Return
-20
40
-60
Episodes
Figure 13: Return plots of algorithms: DDPG and SAC averaged over 5 runs in the continuous Mountain Car problem.
40

---

Published in Transactions on Machine Learning Research (05/2025)
D.3 Evolution of stepwise- disiisee,, distance- to opoopiial,, and OMR(k)
In this section we present 2 dimensional versions the policy trajectories Figures and along with cor- responding OMR evolution plots. These are stepwise- disttace vs. updates, distance-to-optimen to-optimal vs. updates, and (M)(k) plots for the algorithms. Figure 14 presents plots for the continuous environment Mountain Car, while Figure 15 presents plots for the discrete environment 2D Gridworld.
DDPG
distance to aptimal
0.8
0.4
0.6
0.55
0.50
0.1
0.45
5000
10000
15000
20000
5000
10000
15000
20000
5000
30000
15000
20000
#updates
#updates
#updates
SAC
distance to optooaall
0.5
0.8
0.60
0.4
0.55
0.50
0.2
500n
10000
15000
20000
5000
10000
15000
20000
5000
10000
15000
20000
#updates
#updates
#updates
Figure 14: Plots in the first column stepwise- distance number of updates, second column distance- to- optimal vs. number of updates, and third OMR(k) vs. number of updates. Top row plots belong to DDPG algorithm, while bottom row plots belong to SAC.
41

---

Published in Transactions on Machine Learning Research (05/2025)
c=0) 0)-geedy
distance to optimal 2
1.0
0.9
0.0
stapwise
0.6
0.4
175
100
100
@updates
#updates
@updates
<(=1)-greedy
stapwise distance
distance to aptimal
1.0
0.9
OMR(M
0.8
0.7
0.6
0.5-
H
500
1.000
3500
2000
2500
500
1000
1500
2000
2500
500
1000
1500
2000
2500
@updates
#updates
@updates
UCRL2
distance to opt.onallll
stopwise, aiizance
12.5
Mm WMMM MM MMM
10.0-
OMR(K)
0.9
7.5-
0.8
5.0-
2
2.5
0.5
0.0
10
2b
30
40
50
60
70
80
10
20
30
40
60
70
80
10
20
300
40
50
60
70
#updates
#updates
#updates
PSRL
distance to opppiialll
1.0
OMR(K)
44
0.8
stapwise
0.7
0.5
0.4
tup dates
#updates
@updates
stapwise distance
distance to optimal
1.0
0.7-
0.6
20
0.0
15
20
supdates
#updates
#updates
DQN
stepwise distance
distance to optimal 2
1.0
0.9
OMR(K)
0.8
2-
0.7
0.6
0.4
@updates
#updates
#updates
Figure 15: Plots the first column are disptise vs. number of updates, second column distance- to to-optimal vs. number of updates, and third OMR(K) vs. number of updates. The plots the row belong algorithms in the following order from top bottom: c(=0)-greody, c(=1)-greody, UCRL2, PSRL, SAC, and DQN.
42


Reviews:
[{"review_id": "ZqdPLHj8d8", "reviewer": "Reviewer_CbTT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper give an interesting framework to visualize and analyze the policy evolution during RL training.\n\n2. The paper rigorously formalizes occupancy measures as points on a differentiable manifold and employs the Wasserstein distance to measure trajectories, providing a strong theoretical underpinning.\n\n3. ESL and OMR are introduced to help analyze the characteristics of algorithms, which is a different evaluation perspective from regret. This direction is promising and worthy of further research.\n\n4. The experiment part is precise and well-explained, making the advantages and functions of this proposed framework clear.\n\n\nWeaknesses:\n\n1. **Reliance on knowledge of an optimal policy** may hinder the direct computation of distance-to-optimal, which can complicate the framework’s practical usage.\n\n2. Estimating Wasserstein distance and occupancy measures can be expensive for large-scale or high-dimensional environments, potentially limiting scalability.\n\n3. It's good that Proposition 2 gives a connection between regret and occupancy measure, but further analysis is lacking. For example, it is interesting to discuss whether the summation of distance-to-optimal can become a similar evaluation way to regret of an algorithm.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a framework for analyzing the learning process in reinforcement learning. By representing algorithms as trajectories of occupancy measures, it defines two metrics—Effort of Sequential Learning (ESL) and Optimal Movement Ratio (OMR)—to quantify exploration efficiency. The authors establish theoretical guarantees and provide empirical evidence that these metrics capture meaningful insights into exploration behavior. The most interesting point is that this paper provides a new perspective on analyzing the characteristics of algorithms, which is promising and worthy of further research.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "This paper is generally a good and interesting paper that provide a framework to visualize and analyze the policy evolution during RL training, thus giving a new perspective on analyzing the characteristics of algorithms. **I recommend accepting this paper**.\n\nHowever, I still have some suggestions for this paper and some problems need to be clarified from the authors:\n\n1. As is stated in weaknesses (3), it is interesting to discuss whether the summation of distance-to-optimal can become a similar evaluation way to regret of an algorithm. I suggest adding some further discussion on this point.\n\n2. In page 5, how to understand the statement \"we can show that $v_\\pi^H$ satisfies the linear programming description of value function maximization along with the Bellman flow constraints\"?\n\n3. Is it necessary to pre-know optimal policy for the calculation of ESL and OMR? In experiment part, is the optimal policy pre-known? (Otherwise we cannot calculate distance-to-optimal while training). If so, I think the reliance on knowledge of an optimal policy will complicate the framework’s practical usage. **I hope the author can clearly explain this question.**\n\n4. Does this framework only work in MDP with $L_R$-Lipschitz reward setting? Can it be applied in more general settings?\n\n5. The experiments to analyze the characteristics of algorithms are interesting. I suggest implementing on more popular algorithms to make the paper more convincing. Moreover, I think the 3D scatter plots in Figure 3 is not very intuitional though it conveys some phenomenon. I suggest the author also providing projection plots (i.e. y-axis is updates and x-axis is stepwise-distance or distance-to-optimal), which may be more direct for readers to understand the characteristics of algorithms.\n\n6. Drawing Q-functions plot is a common way to record the performance of algorithms. I suggest the author also provide Q-function plot to compare with existing plots. I would like to know the unique advantages of your provided methods to analyze algorithms' characteristics.\n\n7. The paper states that \"Ultimately, depending on the environment, we can select an algorithm with promising characteristics and spend more time optimizing it to improve performance.\" Can you explain more about this idea? Do you mean that we can use this framework to analyze characteristics of algorithms first then choose the most promising algorithm that can best adapt the environment? Will the characteristics will change if the environment changes?\n\n8. In the definition of $\\mathcal{M}$, that is, $\\mathcal{M}=\\lbrace v_{\\pi_\\theta}(s, a) \\mid \\pi_\\theta \\in \\Gamma_\\theta, \\theta \\in \\mathbb{R}^{N_\\theta} \\rbrace$ in page 3. I don't think there should be a $\\theta$ in policy set $\\Gamma$. Is this a typo?", "broader_impact_concerns": "This paper does not have any concerns on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NHwcMTqDTG", "reviewer": "Reviewer_rnie", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. **Novel Theoretical Framework:** The paper introduces a mathematically rigorous framework for understanding RL algorithms through the lens of occupancy measures and optimal transport, providing a new perspective on exploration. The two metrics, ESL and OMR, proposed in this paper are novel in the literature.\n2. **Impacts other than Comparing Algorithms:** By connecting their occupancy measure-based metrics to regret (a standard performance metric), the authors bridge theoretical understanding with practical performance evaluation. The paper also demonstrates that ESL scales proportionally with task difficulty, providing a way to quantify the relative hardness of different environments.\n3. **Consideration of Practical Cases:** The authors provide methods to estimate ESL and OMR, and give upper bound on the estimation error. Moreover, when the optimal policy is not reached, the surrogate quantity $\\eta_{\\textup{sub}}$ for ESL can also be bounded near the true value.\n\n### Weaknesses\n\n1. **Computational Complexity:** Computing the Wasserstein distance between occupancy measures can be computationally intensive, especially for large state-action spaces, making practical application challenging.\n2. **Dependence on Optimal Policy:** While the authors provide methods to approximate their metrics when the optimal policy is unknown, the most accurate measurements still require knowledge of the optimal policy, which is rarely available in real-world applications. Meanwhile, though Proposition 5 gives an approximation factor of $\\eta_{\\textup{sub}}$ with respect to $\\eta$, this multiplicative approximation is not very informative, as $\\mathcal{W}\\_1 (v_{\\pi_N}, v_{\\pi^*})$ may be large when $\\pi_N$ is far away from $\\pi^*$. This results in high-level requires the algorithm to reach a neighborhood of the optimal policy to estimate $\\eta_{\\textup{sub}}$ accurately.\n3. **Limited Experiments:** The experiments are primarily conducted on relatively simple environments (GridWorld and Mountain Car). More complex domains with high-dimensional state spaces would better demonstrate the framework's scalability.\n4. **Strong Assumptions:** The assumption on existence of inverse transition matrices (Proposition 1) may not hold in practical settings. The authors didn't verify that for the simple environments in Section 5, such assumption holds.\n5. **Limited Practical Guidelines:** While the metrics provide insights into exploration processes, the paper doesn't explicitly guide how to design better RL algorithms based on these insights.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel framework for analyzing and comparing reinforcement learning algorithms based on the trajectories they take in the space of state-action occupancy measures. The key insight is that any RL algorithm generates a sequence of policies during training, which can be viewed as a path in the manifold of occupancy measures.\nThe authors define two primary metrics:\n- **ESL (Effort of Sequential Learning):** Measures the ratio of the path traversed by an RL algorithm to the direct distance between initial and optimal policies\n- **OMR (Optimal Movement Ratio):** Measures the proportion of policy updates that effectively reduce the distance to optimal policy\n\nThe paper establishes a mathematical foundation by proving that the space of occupancy measures forms a differentiable manifold for smoothly parameterized policies. They also connect their metrics to regret, showing that regret is related to the sum of distances between the optimal policy and each policy in the learning sequence.\n\nFor practical implementation, the authors derive approximation guarantees for estimating ESL and OMR with finite samples, including when the optimal policy is not reached. Through experiments across various environments (discrete and continuous) and algorithms (Q-learning, UCRL2, PSRL, SAC, DQN, DDPG), they demonstrate that their metrics provide insights into exploration processes and task difficulty.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In general, I think this paper provides interesting insights into a new method of comparing algorithms and tasks in RL problems. Regarding each weakness I mentioned, I suggest the authors make changes correspondingly.\n1. **Computational Complexity:** Give details of how to approximate Wasserstein distance when state-action space is large.\n2. **Dependence on Optimal Policy:** Make clear arguments about the applicable scenarios when Proposition 5 gives meaningful approximation factor. Say, how can we guarantee $\\mathcal{W}\\_1 (v_{\\pi_N}, v_{\\pi^*}) < c$ for some constant $c$?\n3. **Limited Experiments:** Maybe add experiments on harder tasks. This can be related to the first weakness.\n4. **Strong Assumptions:** Verify that for the simple environments in Section 5, such assumption holds (or not).\n5. **Limited Practical Guidelines:** Add more discussions on how will ESL and OMR related to algorithm design for RL tasks.", "broader_impact_concerns": "I don't think this work has ethical issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7zMzB08EDi", "reviewer": "Reviewer_5wrR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. This work provides a novel geometric perspective on RL exploration with Wasserstein distances, which enables a principled comparison of policy trajectories.  \n2. This work has a strong theoretical foundation, which links policy dynamics to regret bounds and provides approximation guarantees.  \n3. The experiments clearly demonstrate ESL/OMR can differentiate exploration behaviors (e.g., direct vs. meandering paths) across algorithms like Q-learning, UCRL2, and SAC.  \n4. The work is well-written with clear contributions and demonstration.\n\n**Weaknesses**\n1. The proposed metrics rely on knowing the optimal policy ($\\pi^*$) to compute the geodesic. Approximate $\\pi^*$ with suboptimal $\\pi_N$ sounds valid but is not validated in complex settings, which may limit applicability to environments where $\\pi^*$ is tractable.  \n2. I am concerned about the computational scalability. Calculating Wasserstein distances between occupancy measures incurs high computational costs, especially for high-dimensional/continuous spaces. Can the authors provide some insights on this?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. The paper introduces a geometric framework to analyze exploration in reinforcement learning (RL) by modeling the learning process as trajectories in the space of state-action occupancy measures.  \n2. It proposes two novel metrics: Effort of Sequential Learning (ESL), which quantifies exploration efficiency as the ratio of an algorithm’s trajectory length to the shortest path (geodesic) between initial and optimal policies, and Optimal Movement Ratio (OMR), which measures the fraction of policy updates that reduce regret-like \"distance-to-optimal.\"  \n3. Theoretical guarantees are provided for estimating ESL and OMR with finite samples and suboptimal final policies.  \n4. Empirical validation on discrete and continuous environments demonstrates the metrics’ ability to diagnose exploration efficiency and task difficulty.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Can the authors demonstrate the reason for using W1 distance? Do the authors also consider W2 distance or KL divergence? How does the choice of distance metrics affect ESL/OMR?\n\n**Typos**\n  - Abstract: \"finite number of samples\" $\\rightarrow$ \"a finite number of samples.\"  \n  - Introduction: \"Comparing the exploratory processes of these eclectic algorithms across the multi-directional space of RL algorithm design, emerges as a natural question\" includes an unnecessary comma after \"design.\"\n  - Keep consistent when using \"behaviour\" and \"behavior\"", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a framework for analyzing the learning process in reinforcement learning by representing algorithms as trajectories over occupancy measures. It introduces two novel metrics—Effort of Sequential Learning (ESL) and Optimal Movement Ratio (OMR)—to quantify exploration efficiency. The authors provide theoretical guarantees and empirical evidence that these metrics offer meaningful insights into exploration dynamics.

A particularly interesting aspect of the work is its new perspective on understanding the behavior of learning algorithms through occupancy measure trajectories and the use of Wasserstein distance. This approach opens up a promising direction for further research in characterizing and comparing RL algorithms beyond standard performance metrics.

The authors have addressed most of the reviewers’ concerns in their response. One remaining point is that both ESL and OMR require knowledge of the optimal policy. While the authors draw an analogy to regret and discuss optimal policy approximation in Section 4.2, this comparison is not entirely precise: regret can be practically visualized using cumulative rewards, whereas the proposed metrics depend more directly on the unknown optimal policy and lack a straightforward empirical proxy. It would be beneficial if the authors could explicitly discuss this limitation where the metrics are defined, in Section 3.

Despite this minor concern, the paper is well-structured and provides a thoughtful and theoretically grounded contribution to the understanding of learning dynamics in RL. I recommend acceptance with minor revisions to address the above clarification.
Decision: Accept (poster)