SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jose Blanchet\*
Miao Lu $^{1*}$
Tong Zhang*
Han Zhong $^{3*}$
$^{1}$  Department of Management Science and Engineering, Stanford University
$^{2}$  Department of Mathematics, The Hong Kong University of Science and Technology
<sup>3</sup> Center for Data Science, Peking University
We study distributionally robust offline reinforcement learning (RL), which seeks to find an optimal robust policy purely from an offline dataset that can perform well in perturbed environments. We propose a generic algorithm framework Doubly Pessimistic Model-based Policy Optimization  $(\mathbb{P}^2\mathbb{M}\mathbb{P}\mathbb{O})$  for robust offline RL, which features a novel combination of a flexible model estimation subroutine and a doubly pessimistic policy optimization step. Here the double pessimism principle is crucial to overcome the distribution shift incurred by i) the mismatch between behavior policy and the family of target policies; and ii) the perturbation of the nominal model. Under certain accuracy assumptions on the model estimation subroutine, we show that  $\mathbb{P}^2\mathbb{M}\mathbb{P}\mathbb{O}$  is provably sample-efficient with robust partial coverage data, which means that the offline dataset has good coverage of the distributions induced by the optimal robust policy and perturbed models around the nominal model. By tailoring specific model estimation subroutines for concrete examples including tabular Robust Markov Decision Process (RMDP), factored RMDP, and RMDP with kernel and neural function approximations, we show that  $\mathbb{P}^2\mathbb{M}\mathbb{P}\mathbb{O}$  enjoys a  $\tilde{\mathcal{O}}(n^{-1/2})$  convergence rate, where  $n$  is the number of trajectories in the offline dataset. Notably, these models, except for the tabular case, are first identified and proven tractable by this paper. To the best of our knowledge, we first propose a general learning principle — double pessimism — for robust offline RL and show that it is provably efficient in the context of general function approximations.
Reinforcement learning (RL) [52] aims to learn an optimal policy that maximizes the cumulative rewards received in an unknown environment. Typically, deep RL algorithms learn a policy in an online trial-and-error fashion using millions to billions of data. However, data collection could be costly and risky in some practical applications such as healthcare [56] and autonomous driving [38]. To tackle this challenge, offline RL (also known as batch RL) [21, 22] learns a near-optimal policy based on a dataset collected a priori without further interactions with the environment. Although there has been great progress in offline RL [72, 20, 16, 54, 62, 6], these works implicitly require that the offline dataset is generated by the real-world environment, which may fail in practice. Taking robotics [18, 37] as an example, the experimenter trains agents in a simulated physical environment and then deploy them in real-world environments. Since the experimenter does not have access to the true physical environment, there is a mismatch between the simulated environment used to generate the offline dataset and the real-world environment used to deploy the agents. Such a mismatch is
*Alphabetical order. Email to miaolu@stanford.edu
commonly referred to as the sim-to-real gap [42, 76]. Since the optimal policy is sensitive to the model [31, 9], the potential sim-to-real gap may lead to the poor performance of RL algorithms.
A promising solution to remedy this issue is robust RL [13, 9, 32] – training a robust policy that performs well in a bad or even adversarial environment. A line of work on deep robust RL [43, 44, 41, 30, 53, 75, 19] demonstrates the superiority of the trained robust policy in real world environments. Furthermore, the recent work of Hu et al. [11] theoretically proves that the ideal robust policy can attain near optimality in dealing with problems with sim-to-real gap, but this work does not suggest how to learn a robust policy from a theoretical perspective. In order to understand robust RL from the theoretical side, robust Markov decision process (RMDP) [13, 9] has been proposed, and many recent works [77, 47, 29] design sample-efficient learning algorithms for robust offline RL. These works mainly focus on the tabular case, which is not capable of tackling large state spaces. Meanwhile, in the non-robust setting, a line of works [16, 54, 62, 73, 45] show that “pessimism” is the general learning principle for designing algorithms that can overcome the distributional shift problem faced by offline RL. In particular, in the context of function approximation, Xie et al. [62] and Uehara and Sun [54] leverage the pessimism principle and propose generic algorithms in the model-free and model-based fashion, respectively. Hence, it is natural to ask the following questions:
Q1: What is the general learning principle for robust offline RL?
Q2: Based on this learning principle, can we design a generic algorithm for robust offline RL in the context of function approximation?
To answer these two questions, we need to tackle the following two intertwined challenges: (i) distributional shift, that is, the mismatch between offline data distribution and the distribution induced by the optimal robust policy. In robust offline RL, the distributional shift has two sources - behavior policy and perturbed model, where the latter is the unique challenge not presented in non-robust RL; and (ii) function approximation. Existing works mainly focus on the tabular case, and it remains elusive how to add reasonable structure conditions to make RMDPs with large state spaces tractable. Despite these challenges, we answer the aforementioned two questions affirmatively.
Contributions. We study robust offline RL in a general framework, which not only includes existing known tractable  $S \times \mathcal{A}$ -rectangular tabular RMDPs, but also subsumes several newly proposed models (e.g.,  $S \times \mathcal{A}$ -rectangular factored RMDPs,  $S \times \mathcal{A}$ -rectangular kernel RMDPs, and  $S \times \mathcal{A}$ -rectangular neural RMDPs) as special cases. Under this general framework, we propose a generic model-based algorithm, dubbed as Doubly Pessimistic Model-based Policy Optimization (P2MP0), which consists of a model estimation subroutine and a policy optimization step based on doubly pessimistic value estimators. We note that the model estimation subroutine can be flexibly chosen according to structural conditions of specific RMDP examples. Meanwhile, the adoption of doubly pessimistic value estimators in the face of model estimation uncertainty and environment uncertainty plays a key role in overcoming the distributional shift problem in robust offline RL.
From the theoretical perspective, we characterize the optimality of  $\mathrm{P}^2\mathrm{MP0}$  with partial coverage. In particular, we show that the suboptimality gap of  $\mathrm{P}^2\mathrm{MP0}$  is upper bounded by the model estimation error (see Condition 3.2) and the robust partial coverage coefficient (see Assumption 3.3). For concrete examples of RMDPs, by customizing specific model estimation mechanisms and plugging them into  $\mathrm{P}^2\mathrm{MP0}$ , we show that  $\mathrm{P}^2\mathrm{MP0}$  enjoys a  $n^{-1/2}$  convergence rate with robust partial coverage data, where  $n$  is the number of trajectories in the offline dataset. In summary, we identify a general learning principle — double pessimism — for robust offline RL. Based on this principle, we can perform sample-efficient robust offline RL with robust partial coverage data via general function approximation. See Table 1 for a summary of our results and a comparison with mostly related works.
Robust reinforcement learning in robust Markov decision processes. Robust RL is usually modeled as a robust MDP (RMDP) [13, 9], and its planning has been well studied [13, 9, 65, 60, 57]. Recently, robust RL in RMDPs has attracted considerable attention, and a growing body of works studies this problem in the generative model [68, 39, 49, 58, 69, 66, 7], online setting [59, 3, 8], and offline setting [77, 40, 47, 29]. Our work focuses on robust offline RL, and we provide a more in-depth comparison with Zhou et al. [77], Shi and Chi [47], Ma et al. [29] as follows. Under the full coverage condition (a uniformly lower bounded data distribution), Zhou et al. [77] provide the first sample-efficient algorithm for  $S \times A$ -rectangular tabular RMDPs. After, Shi and Chi [47]
Table 1: A comparison with closely related works on robust offline RL.  $\checkmark$  means the work can tackle this model with robust partial coverage data,  $\checkmark!$  means the work requires full coverage data to solve the model, and  $\pmb{x}$  means the work cannot tackle the model. Lightblue color denotes the models that are first proposed and proved tractable in this work.
Zhou et al. [77]	Shi and Chi [47]	Ma et al. [29]	This Work
S × A-rectangular tabular RMDP	✓!	✓	X	✓
d-rectangular linear RMDP	X	X	✓	✓
S × A-rectangular factored RMDP	X	X	X	✓
S × A-rectangular kernel RMDP	X	X	X	✓
S × A-rectangular neural RMDP	X	X	X	✓
leverage the pessimism principle and design a sample-efficient offline algorithm that only requires robust partial coverage data for  $S \times \mathcal{A}$ -rectangular tabular RMDPs. Ma et al. [29] propose a new  $d$ -rectangular RMDP and develop a pessimistic style algorithm that can find a near-optimal robust policy with partial coverage data. In comparison, we provide a generic algorithm that can not only solve the models in Zhou et al. [77], Shi and Chi [47], Ma et al. [29], but can also tackle various newly proposed RMDP models such as  $S \times \mathcal{A}$ -rectangular factored RMDP,  $S \times \mathcal{A}$ -rectangular kernel RMDP, and  $S \times \mathcal{A}$ -rectangular neural RMDP. See Table 1 for a summary. Moreover, we propose a new pessimistic type learning principle "double pessimism" for robust offline RL. Although Shi et al. [48] and Ma et al. [29] adopt the similar algorithmic idea in tabular or linear settings, neither of them have identified a general learning principle for robust offline RL in the regime of large state space.
Non-robust offline RL and pessimism principle. The line of works on offline RL aims to design efficient learning algorithms that find an optimal policy given an offline dataset collected a priori. Prior works [33, 2, 5] typically require a dataset of full coverage, which assumes that the offline data have good coverage of all state-action pairs. In order to avoid such a strong coverage condition on data, the pessimism principle – being conservative in policy or value estimation of those state-action pairs that are not sufficiently covered by data – has been proposed. Based on this principle, a long line of works [see e.g., 16, 54, 62, 63, 45, 73, 71, 64, 48, 24, 26, 74, 28, 46] propose algorithms that can learn the optimal policy only with the partial coverage data. The partial coverage data only requires to cover the state-action pairs visited by the optimal policy. Among these works, our work is mostly related to the work of Uehara and Sun [54], which proposes a generic model-based algorithm for non-robust offline RL. Our algorithm for robust offline RL is also in a model-based fashion, and our study covers some models such as  $S \times \mathcal{A}$ -rectangular kernel and neural RMDPs whose non-robust counterparts are not studied by Uehara and Sun [54]. More importantly, our algorithm is based on a newly proposed double pessimism principle, which is tailored for robust offline RL and is in parallel with the pessimism principle used in non-robust offline RL. Also, we show that the performance of our proposed algorithm depends on the notion of robust partial coverage coefficient, which is also different from the notions of partial coverage coefficient in previous non-robust offline RL works [16, 62, 54].
1.2 Notations
For any set  $A$ , we use  $2^A$  to denote the collection of all the subsets of  $A$ . For any measurable space  $\mathcal{X}$ , we use  $\Delta(\mathcal{X})$  to denote the collection of probability measures over  $\mathcal{X}$ . For any integer  $n$ , we use  $[n]$  to denote the set  $\{1, \dots, n\}$ . Throughout the paper, we use  $D(\cdot \| \cdot)$  to denote a (pseudo-)distance between two probability measures (or densities). In specific, we define the KL-divergence  $D_{\mathrm{KL}}(p \| q)$  between two probability densities  $p$  and  $q$  over  $\mathcal{X}$  as
$$
D _ {\mathrm {K L}} (p \| q) = \int_ {\mathcal {X}} p (x) \log \left(\frac {p (x)}{q (x)}\right) \mathrm {d} x,
$$
and we define the TV-distance  $D_{\mathrm{TV}}(p\| q)$  between two probability densities  $p$  and  $q$  over  $\mathcal{X}$  as
$$
D _ {\mathrm {T V}} (p \| q) = \frac {1}{2} \int_ {\mathcal {X}} | q (x) - p (x) | \mathrm {d} x.
$$
Given a function class  $\mathcal{F}$  equipped with some norm  $\| \cdot \|_{\mathcal{F}}$ , we denote by  $\mathcal{N}_{[]}(\epsilon, \mathcal{F}, \| \cdot \|_{\mathcal{F}})$  the  $\epsilon$ -bracket number of  $\mathcal{F}$ , and  $\mathcal{N}(\epsilon, \mathcal{F}, \| \cdot \|_{\mathcal{F}})$  the  $\epsilon$ -covering number of  $\mathcal{F}$ .
2.1 A Unified Framework of Robust Markov Decision Processes
We introduce a unified framework for studying episodic robust Markov decision processes (RMDP), which we denote as a tuple  $(S, \mathcal{A}, H, P^{\star}, R, \mathcal{P}_{\mathrm{M}}, \Phi)$ . The set  $S$  is the state space with possibly infinite cardinality,  $\mathcal{A}$  is the action space with finite cardinality. The integer  $H$  is the length of each episode. The set  $P^{\star} = \{P_h^{\star}\}_{h=1}^H$  is the collection of transition kernels where each  $P_h^{\star}: S \times \mathcal{A} \mapsto \Delta(S)$ , and  $R = \{R_h\}_{h=1}^H$  is the collection of reward functions where each  $R_h: S \times \mathcal{A} \mapsto [0,1]$ . We use  $\Delta(S)$  to note the probability simplex on  $S$  (i.e. the space of probability measures with support on  $S$ ).
We consider a model-based perspective of reinforcement learning, and we denote  $\mathcal{P} = \{P(\cdot |\cdot ,\cdot):$ $S\times \mathcal{A}\mapsto \Delta (S)\}$  as the space of all transition kernels. The space  $\mathcal{P}_{\mathrm{M}}\subseteq \mathcal{P}$  of the RMDP is a realizable model space which contains the transition kernel  $P^{\star}$ , i.e.,  $P_{h}^{\star}\in \mathcal{P}_{\mathrm{M}}$  for any step  $h\in [H]$ . Finally, the RMDP is equipped with a mapping  $\Phi :\mathcal{P}_{\mathrm{M}}\mapsto 2^{\mathcal{P}}$  that characterizes the robust set of any transition kernel in  $\mathcal{P}_{\mathrm{M}}$ . Formally, for any transition kernel  $P\in \mathcal{P}_{\mathrm{M}}$ , we call  $\Phi (P)$  the robust set of  $P$ . One can interpret the transition kernel  $P_{h}^{\star}\in \mathcal{P}_{\mathrm{M}}$  as the transition kernel of the training environment, while  $\Phi (P_h^\star)$  contains all the possible transition kernels of the test environment.
Remark 2.1. The mapping  $\Phi$  is defined on the realizable model space  $\mathcal{P}_{\mathrm{M}}$ , while for generality we allow the image of  $\Phi$  to be outside of  $\mathcal{P}_{\mathrm{M}}$ . That is, a  $\widetilde{P} \in \Phi(P)$  for some  $P \in \mathcal{P}_{\mathrm{M}}$  might be in  $\mathcal{P}_{\mathrm{M}}^c$ .
Policy and robust value function. Given an RMDP  $(\mathcal{S},\mathcal{A},H,P^{\star},R,\mathcal{P}_{\mathrm{M}},\Phi)$ , we consider using a Markovian policy to make decision. A Markovian policy  $\pi$  is defined as  $\pi = \{\pi_h\}_{h = 1}^H$  with  $\pi_h:\mathcal{A}\mapsto \Delta (\mathcal{S})$  for each step  $h\in [H]$ . For simplicity, we use policy to refer to a Markovian policy.
Given any policy  $\pi$ , we define the robust value function of  $\pi$  with respect to any set of transition kernels  $P = \{P_h\}_{h=1}^H \subseteq \mathcal{P}_{\mathrm{M}}$  as the following, for each step  $h \in [H]$ ,
$$
V _ {h, P, \Phi} ^ {\pi} (s) = \inf  _ {\widetilde {P} _ {h} \in \Phi \left(P _ {h}\right), 1 \leq h \leq H} V _ {h} ^ {\pi} \left(s; \{\widetilde {P} _ {h} \} _ {h = 1} ^ {H}\right), \quad \forall s \in \mathcal {S}, \tag {2.1}
$$
$$
Q _ {h, P, \Phi} ^ {\pi} (s, a) = \inf  _ {\widetilde {P} _ {h} \in \Phi \left(P _ {h}\right), 1 \leq h \leq H} Q _ {h} ^ {\pi} (s, a; \{\widetilde {P} _ {h} \} _ {h = 1} ^ {H}), \quad \forall (s, a) \in \mathcal {S} \times \mathcal {A}. \tag {2.2}
$$
Here  $V_h^\pi (\cdot ;\{\widetilde{P}_h\}_{h = 1}^H)$  and  $Q_{h}^{\pi}(\cdot ;\{\widetilde{P}_{h}\}_{h = 1}^{H})$  are the state-value function and the action-value function [52] of policy  $\pi$  in the standard episodic MDP  $(S,\mathcal{A},H,\{\widetilde{P}_h\}_{h = 1}^H,R)$ ,
$$
V _ {h} ^ {\pi} \left(s; \{\widetilde {P} _ {h} \} _ {h = 1} ^ {H}\right) = \mathbb {E} _ {\{\widetilde {P} _ {h} \} _ {h = 1} ^ {H}, \pi} \left[ \sum_ {i = h} ^ {H} R _ {i} \left(s _ {i}, a _ {i}\right) \mid s _ {h} = s \right], \quad \forall s \in \mathcal {S}, \tag {2.3}
$$
$$
Q _ {h} ^ {\pi} \left(s, a; \{\widetilde {P} _ {h} \} _ {h = 1} ^ {H}\right) = \mathbb {E} _ {\{\widetilde {P} _ {h} \} _ {h = 1} ^ {H}, \pi} \left[ \sum_ {i = h} ^ {H} R _ {i} \left(s _ {i}, a _ {i}\right) \mid s _ {h} = s, a _ {h} = a \right], \quad \forall (s, a) \in \mathcal {S} \times \mathcal {A}, \tag {2.4}
$$
where the expectation  $\mathbb{E}\{\widetilde{P}_h\}_{h=1}^H, \pi[\cdot]$  is taken with respect to the trajectories induced by the transition kernel  $\{\widetilde{P}_h\}_{h=1}^H$  and the policy  $\pi$ . Intuitively, the robust value function of a policy  $\pi$  given transition kernel  $P$  is defined as the least expected cumulative reward achieved by  $\pi$  when the transition kernel varies in the robust set of  $P$ . This is how an RMDP takes the perturbed models into consideration.
$\mathcal{S} \times \mathcal{A}$ -rectangular robust set and robust Bellman equation. Ideally, we would like to consider robust value function that has recursive expressions, just like the Bellman equation satisfied by (2.3) in a standard MDP [52]. To this end, we focus on a generally adopted kind of robust set in our unified framework, which is called the  $\mathcal{S} \times \mathcal{A}$ -rectangular robust set [13].
Assumption 2.2 ( $S \times \mathcal{A}$ -rectangular robust set). We assume that the mapping  $\Phi$  induces  $S \times \mathcal{A}$ -rectangular robust sets. Specifically, the mapping  $\Phi$  satisfies, for  $\forall P \in \mathcal{P}_{\mathrm{M}}$ ,
$$
\Phi (P) = \bigotimes_ {(s, a) \in \mathcal {S} \times \mathcal {A}} \mathcal {P} _ {\rho} (s, a; P), \quad \mathcal {P} _ {\rho} (s, a; P) = \left\{\widetilde {P} (\cdot) \in \Delta (\mathcal {S}): D (\widetilde {P} (\cdot) \| P (\cdot | s, a)) \leq \rho \right\},
$$
for some (pseudo-) distance  $D(\cdot \|\cdot)$  on  $\Delta(S)$  and some  $\rho \in \mathbb{R}_+$ . Intuitively,  $S \times \mathcal{A}$ -rectangular requires that  $\Phi(P)$  gives decoupled robust sets for  $P(\cdot | s, a)$  across different  $(s, a)$ -pairs. The (pseudo-)distance  $D(\cdot \|\cdot)$  can be chosen as a  $\phi$ -divergence [68, 10] or a  $p$ -Wasserstein-distance [35].
Thanks to the  $S \times \mathcal{A}$ -rectangular assumption on the mapping  $\Phi$ , the robust value functions (2.1) of any policy  $\pi$  then satisfy a recursive expression, which is called robust Bellman equation [13, 36].
Proposition 2.3 (Robust Bellman equation). Under Assumption 2.2, for any  $P = \{P_h\}_{h=1}^H$  where  $P_h \in \mathcal{P}_{\mathrm{M}}$  and any  $\pi = \{\pi_h\}_{h=1}^H$  with  $\pi_h : S \mapsto \Delta(\mathcal{A})$ , the following robust Bellman equation holds,
$$
V _ {h, P, \Phi} ^ {\pi} (s) = \mathbb {E} _ {a \sim \pi_ {h} (\cdot | s)} \left[ Q _ {h, P, \Phi} ^ {\pi} (s, a) \right], \quad \forall s \in S, \tag {2.5}
$$
$$
Q _ {h, P, \Phi} ^ {\pi} (s, a) = R _ {h} (s, a) + \inf  _ {\widetilde {P} _ {h} \in \Phi (P _ {h})} \mathbb {E} _ {s ^ {\prime} \sim \widetilde {P} _ {h} (\cdot | s, a)} \left[ V _ {h + 1, P, \Phi} ^ {\pi} \left(s ^ {\prime}\right) \right], \quad \forall (s, a) \in \mathcal {S} \times \mathcal {A}. \tag {2.6}
$$
To be self-contained, in Appendix B we provide a detailed proof of the robust Bellman equation in our framework under Assumption 2.2. Equation (2.5) actually says that the infimum over all the transition kernels (recall the definition of  $V_{h,P,\Phi}^{\pi}$  in (2.1)) can be decomposed into a "one-step" infimum over the current transition kernel, i.e.,  $\inf_{\widetilde{P}_h\in \Phi (P_h)}$ , and an infimum over the future transition kernels, i.e.,  $V_{h + 1,P,\Phi}^{\pi}$ . Such a property is crucial to the algorithmic design and theoretical analysis for RMDPs.
2.2 Examples of Robust Markov Decision Processes
In this section, we give concrete examples for the general RMDP framework proposed in Section 2.1. Most existing works on RMDPs hinge on the finiteness assumption on the state space, which fails to deal with prohibitively large or even infinite state space. In our framework, RMDPs can be considered in the paradigm of infinite state space, for which we adopt various powerful function approximation tools including kernel and neural functions. Also, we introduce a new type of RMDP named robust factored MDP, which is a robust extension of standard factored MDPs [17].
Remark 2.4. Besides  $S \times \mathcal{A}$ -rectangular-type robust sets (Assumption 2.2), our unified framework of RMDP can also cover other types of robust sets considered in some previous works as special cases, including  $S$ -rectangular robust set [61] and  $d$ -rectangular robust set for linear MDPs [29]. See Section A for a discussion about these two types of robust sets.
In the sequel, we introduce concrete examples of our framework of RMDP.
Example 2.5 ( $S \times \mathcal{A}$ -rectangular robust tabular MDP). When the state space  $S$  is a finite set, we call the corresponding model an  $S \times \mathcal{A}$ -rectangular robust tabular MDP. Recently, there is a line of works on the  $S \times \mathcal{A}$ -rectangular robust tabular MDP [77, 68, 39, 27, 47, 40, 8, 10, 35, 58, 69, 66, 7]. For  $S \times \mathcal{A}$ -rectangular robust tabular MDPs, we choose  $\mathcal{P}_{\mathrm{M}} = \mathcal{P}$  containing all possible transitions.
Remark 2.6. We point out that our framework of RMDP under  $S \times \mathcal{A}$ -rectangular assumption covers substantially more model than  $S \times \mathcal{A}$ -rectangular robust tabular MDP since our state space  $S$  can be infinite. The model space  $\mathcal{P}_{\mathrm{M}}$  can be adapted to function approximation methods to handle the infinite state space. Thus any efficient algorithm developed for our framework of RMDPs can not be covered by algorithms for  $S \times \mathcal{A}$ -rectangular robust tabular MDPs. Example 2.7 and 2.8 are infinite state space  $S \times \mathcal{A}$ -rectangular robust MDPs with function approximations.
Example 2.7 ( $S \times \mathcal{A}$ -rectangular robust MDP with kernel function approximations). We consider an infinite state space  $S \times \mathcal{A}$ -rectangular robust MDP whose realizable model space  $\mathcal{P}_{\mathrm{M}}$  is in a reproduced kernel Hilbert space (RKHS). Let  $\mathcal{H}$  be a RKHS associated with a positive definite kernel  $\mathcal{K} : (S \times \mathcal{A} \times S) \times (S \times \mathcal{A} \times S) \mapsto \mathbb{R}_+$  (See Appendix D.3.1 for a review of the basics of RKHS). We denote the feature mapping of  $\mathcal{H}$  by  $\psi : S \times \mathcal{A} \times S \mapsto \mathcal{H}$ . With  $\mathcal{H}$ , an  $S \times \mathcal{A}$ -rectangular robust MDP with kernel function approximation is defined as an  $S \times \mathcal{A}$ -rectangular robust MDP with
$$
\mathcal {P} _ {\mathrm {M}} = \left\{P \left(s ^ {\prime} \mid s, a\right) = \langle \psi (s, a, s ^ {\prime}), f \rangle_ {\mathcal {H}}: f \in \mathcal {H}, \| f \| _ {\mathcal {H}} \leq B _ {\mathrm {K}} \right\}, \tag {2.7}
$$
for some  $B_{\mathrm{K}} > 0$ . Here we implicitly identify  $P(\cdot |\cdot ,\cdot)$  as the density of the corresponding distribution with respect to a proper base measure on  $\mathcal{S}\times \mathcal{A}\times \mathcal{S}$ .