SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Laixi Shi* Caltech
Gen Li†  
CUHK
Yuting Wei†  
UPenn
Matthieu Geist $^{\S}$   
Google
This paper investigates model robustness in reinforcement learning (RL) via the framework of distributionally robust Markov decision processes (RMDPs). Despite recent efforts, the sample complexity of RMDPs is much less understood regardless of the uncertainty set in use; in particular, there exist large gaps between existing upper and lower bounds, and it is unclear if distributional robustness bears any statistical implications when benchmarked against standard RL. In this paper, assuming access to a generative model, we derive the sample complexity of RMDPs — when the uncertainty set is measured via either total variation or  $\chi^2$  divergence over the full range of uncertainty levels — using a model-based algorithm called distributionally robust value iteration, and develop minimax lower bounds to benchmark its tightness. Our results not only strengthen the prior art in both directions of upper and lower bounds, but also deliver surprising messages that learning RMDPs is not necessarily easier or more difficult than standard MDPs. In the case of total variation, we establish the minimax-optimal sample complexity of RMDPs which is always smaller than that of standard MDPs. In the case of  $\chi^2$  divergence, we establish the sample complexity of RMDPs that is tight up to polynomial factors of the effective horizon, and grows linearly with respect to the uncertainty level when it approaches infinity.
Reinforcement learning (RL) deals with the problem of learning to make sequential decisions based on trial-and-error interactions with some unknown environment. As a fast-growing subfield of artificial intelligence, it has achieved significant success in a variety of domains such as large language model alignment (OpenAI, 2023; Ziegler et al., 2019), healthcare (Liu et al., 2019; Fatemi et al., 2021), robotics and control (Kober et al., 2013; Mnih et al., 2013). Due to the high dimensionality of the state-action space, achieving sample efficiency lies at the core of modern RL practice, especially in various sample-starved applications. As a result, a large portion of efforts in RL has been put in designing sample-efficient algorithms and understanding the fundamental statistical difficulty for diverse RL problems (Azar et al., 2013; Li et al., 2020).
While standard RL has been heavily invested recently, its use can be significantly hampered in practice due to the sim-to-real gap, where an agent trained in an ideal, nominal environment might be extremely sensitive and fail catastrophically when the deployed environment is subject to small changes in task objectives or unexpected perturbations (Zhang et al., 2020a; Klopp et al., 2017; Mahmood et al., 2018). Consequently, in addition to maximizing the long-term cumulative reward,
*Department of Computing Mathematical Sciences, California Institute of Technology, CA, USA.
†Department of Statistics, The Chinese University of Hong Kong, Hong Kong, China.
$^{\ddagger}$ Department of Statistics and Data Science, Wharton School, University of Pennsylvania, PA, USA.
$\S$  Google Research, Brain Team, Paris, France.
Department of Electrical and Computer Engineering, Carnegie Mellon University, PA, USA.
robustness becomes another critical goal for an RL agent, especially in high-stake applications such as robotics, autonomous driving, clinical trials, financial investments, and so on. To address this, distributionally robust RL (Iyengar, 2005; Nilim and El Ghaoui, 2005), which leverages insights from distributionally robust optimization and supervised learning (Rahimian and Mehrotra, 2019; Gao, 2020; Bertsimas et al., 2018; Duchi and Namkoong, 2018; Blanchet and Murthy, 2019), becomes a natural and versatile framework with the goal of learning a policy that performs well even when the deployed environment deviates from the nominal one in the face of environment uncertainty.
In this paper, we are particularly interested in understanding whether, and how, the choice of distributional robustness bears statistical implications in learning the desired policy, by studying the sample complexity in the widely-used generative model (Kearns and Singh, 1999). Suppose that one has access to a generative model which draws samples from a Markov decision processes (MDP) with a nominal transition kernel. Standard RL aims to learn the optimal policy tailored for the nominal kernel based on this set of samples, where the sample complexity has been well understood with matching upper and lower bounds developed recently (Azar et al., 2013; Li et al., 2020). In contrast, distributionally robust RL — leveraging the same set of samples — aims to learn the optimal robust policy whose worst-case performance is maximized when the transition kernel is from some prescribed uncertainty set around the nominal kernel, a setting that is referred to as the robust MDP (RMDPs). Clearly, this ensures that the performance of the learned policy is robust and does not fail catastrophically as long as the sim-to-real gap is not too large. It is then natural to wonder how the robustness consideration impacts the RL performance: should we always prefer to learn a robust policy for a given set of samples? Is there a statistical premium when asking for additional robustness?
Compared with standard MDPs, RMDPs is a richer class of models since one additionally needs to prescribe the shape and size of the uncertainty set, which is usually hand-picked as a small ball around the nominal kernel measured with respect to some distance measure  $\rho$  and uncertainty level  $\sigma$ . To ensure the tractability of solving RMDPs, the uncertainty set is usually assumed to obey certain structures, where the uncertainty set can be decomposed as a product of independent uncertainty subsets over each state or state-action pair (Zhou et al., 2021; Wiesemann et al., 2013), denoted as the  $s$ - and  $(s,a)$ -rectangular rectangularity respectively; in particular, our paper adopts the second choice by assuming the uncertainty set satisfies the  $(s,a)$ -rectangularity. An additional challenge with RMDPs arises from distribution shift, where the transition kernel drawn from the uncertainty set can be different from the nominal kernel, leading to complicated nonlinearity and nested optimization in the problem structure not present in standard MDPs.
1.1 Prior art and open questions
In this paper, we focus on understanding the sample complexity of learning the optimal robust policy of RMDPs in the infinite-horizon setting assuming access to a generative model, when the uncertainty set is measured using one of the  $f$ -divergence: total variation (TV) distance and  $\chi^2$  divergence. These two choices are motivated by their practical appeal: easy to implement, and already adopted by empirical RL (Lee et al., 2021; Pan et al., 2023).
A popular learning approach is model-based, which first estimates the nominal transition kernel using a plug-in estimator based on the collected samples, and then runs a planning algorithm such as a robust variant of value iteration on top of the estimated RMDP. Despite the surge of recent activities, however, existing statistical guarantees for the above paradigm remain highly inadequate, as we shall elaborate momentarily (see Table 1 and Table 2 respectively for a summary). For concreteness, let  $S$  be the size of the state space,  $A$  be the size of the action space,  $\gamma$  be the discount factor (and the effective horizon  $\frac{1}{1 - \gamma}$ ), and  $\sigma$  be the uncertainty level. We are interested in the sample complexity — the number of samples needed for an algorithm to output a policy whose robust value function (the worst-case value over all the transition kernels in the uncertainty set) is at most  $\varepsilon$  away from the optimal robust one — with respect to all salient problem parameters.
- Large gaps between existing upper and lower bounds. There remained large gaps between the sample complexity upper and lower bounds established in prior literature, regardless of the divergence metric in use. Specifically, considering the cases using either TV distance or  $\chi^2$  divergence, the state-of-the-art upper bounds (Panaganti and Kalathil, 2022) scales quadratically with the size  $S$  of the
<sup>6</sup>While it is straightforward to incorporate additional uncertainty of the reward in our framework, we do not consider it here for simplicity, since the key challenge is to deal with the uncertainty of the transition kernel.
Result type	Reference	Sample complexity
0 < σ ≈ 1 - γ	1 - γ ≈ σ < 1
Upper bound	Yang et al. (2022)	S2A(2+σ)2/σ2(1-γ)4ε2
Panaganti and Kalathil (2022)	S2A(1-γ)4ε2
Ours	SA(1-γ)3ε2	SA(1-γ)2σε2
Lower bound	Yang et al. (2022)	SA(1-γ)3ε2	SA(1-γ)/σ4ε2
Ours	SA(1-γ)3ε2	SA(1-γ)2σε2
Table 1: Comparisons between our results and prior arts for finding an  $\varepsilon$ -optimal robust policy in the infinite-horizon RMDPs with an uncertainty set measured with respect to the TV distance, where we ignore logarithmic factors in the sample complexities. Here,  $S$ ,  $A$ ,  $\gamma$ , and  $\sigma \in (0,1)$  are the state space size, the action space size, the discount factor, and the uncertainty level, respectively.
(a) TV distance
(b)  $\chi^2$  distance
Figure 1: Illustrations of the obtained sample complexity upper and lower bounds for learning RMDPs with comparisons to state-of-the-art and the sample complexity of standard MDPs, where the uncertainty set is specified using the TV distance (a) and the  $\chi^2$  distance (b).
state space, while the lower bound (Yang et al., 2022) exhibits only linear scaling with  $S$ . Moreover, in the  $\chi^2$  divergence case, the state-of-the-art upper bound grows linearly with the uncertainty level  $\sigma$  when  $\sigma \gtrsim 1$ , while the lower bound (Yang et al., 2022) is inversely proportional to  $\sigma$ . These lead to unbounded gaps between the upper and lower bounds as  $\sigma$  grows. Can we hope to close these gaps for RMDPs?
- Benchmarking with standard MDPs. Perhaps a more pressing issue is that, past works failed to provide an affirmative answer regarding how to benchmark the sample complexity of RMDPs with that of standard MDPs regardless of the chosen shape (determined by  $\rho$ ) or size (determined by  $\sigma$ ) of the uncertainty set, given the large unresolved gaps mentioned above. Specifically, existing sample complexity upper (resp. lower) bounds are all larger (resp. smaller) than the sample size requirement for standard MDPs. As a consequence, it remains mostly unclear whether learning RMDPs is harder or easier than learning standard MDPs.
To address these questions, we have developed new upper bounds on learning RMDPs with TV distance and  $\chi^2$  distance in the infinite-horizon setting using the model-based approach called distributionally robust value iteration (DRV1), as well as minimax lower bounds to help gauge their
<sup>7</sup>Let  $\mathcal{X} \coloneqq \left(S, A, \frac{1}{1 - \gamma}, \sigma, \frac{1}{\varepsilon}, \frac{1}{\delta}\right)$ . The notation  $f(\mathcal{X}) = O(g(\mathcal{X}))$  or  $f(\mathcal{X}) \lesssim g(\mathcal{X})$  indicates that there exists a universal constant  $C_1 > 0$  such that  $f \leq C_1g$ , the notation  $f(\mathcal{X}) \gtrsim g(\mathcal{X})$  indicates that  $g(\mathcal{X}) = O(f(\mathcal{X}))$ , and the notation  $f(\mathcal{X}) \asymp g(\mathcal{X})$  indicates that  $f(\mathcal{X}) \lesssim g(\mathcal{X})$  and  $f(\mathcal{X}) \gtrsim g(\mathcal{X})$  hold simultaneously. Additionally, the notation  $\widetilde{O}(\cdot)$  is defined in the same way as  $O(\cdot)$  except that it hides logarithmic factors.
Result type	Reference	Sample complexity
0 < σ ≤ 1 - γ	1 - γ ≤ σ ≤ 1/1-γ	σ ≥ 1/1-γ
Upper bound	Panaganti and Kalathil (2022)	S2A(1+σ)/(1-γ)4ε2
Yang et al. (2022)	S2A(1+σ)2/(√1+σ-1)2(1-γ)4ε2
Ours	SA(1+σ)/(1-γ)4ε2
Lower bound	Yang et al. (2022)	SA(1-γ)3ε2	SA(1-γ)2σε2
Ours	SA(1-γ)3ε2	SAσ/(1-γ)4(1+σ)4ε2	SAσ/ε2
Table 2: Comparisons between our results and prior arts for finding an  $\varepsilon$ -optimal robust policy in the infinite-horizon RMDPs with an uncertainty set measured with respect to the  $\chi^2$  distance, where we ignore logarithmic factors in the sample complexities. Here,  $S$ ,  $A$ ,  $\gamma$ , and  $\sigma \in (0, \infty)$  are the state space size, the action space size, the discount factor, and the uncertainty level, respectively.
tightness and provide benchmarking with standard MDPs. As shall be outlined, these new analyses lead to new insights on the interplay between the geometry of uncertainty sets and the statistical sample complexity.
Sample complexity of RMDPs under the TV distance. We summarize the results and comparisons to prior works in Table 1; see Figure 1(a) for an illustration.
- Minimax-optimal sample complexity. We prove that DRVI reaches  $\varepsilon$  accuracy as soon as the sample complexity is on the order of
$$
\widetilde {O} \left(\frac {S A}{(1 - \gamma) ^ {2} \varepsilon^ {2}} \min  \left\{\frac {1}{1 - \gamma}, \frac {1}{\sigma} \right\}\right)
$$
for all  $\sigma \in (0,1)$ , assuming that  $\varepsilon$  is small enough. In addition, a matching minimax lower bound (modulo some logarithmic factor) is established to guarantee the tightness of the upper bound. To the best of our knowledge, this is the first minimax-optimal sample complexity for RMDPs, which was previously unavailable regardless of the divergence metric and uncertainty level in use and is over the full range of the uncertainty level.
- RMDPs are easier than standard MDPs under the TV distance. Given the sample complexity  $O\left(\frac{SA}{(1 - \gamma)^3\varepsilon^2}\right)$  of standard MDPs, it can be seen that RMDPs under the TV distance is never harder than standard MDPs, where it matches that of standard MDPs when  $\sigma \lesssim 1 - \gamma$ , and becomes simpler by a factor of  $\sigma / (1 - \gamma)$  when  $1 - \gamma \lesssim \sigma < 1$ . Therefore, in this case, the robustness comes almost for free since we do not need to collect more samples to reach the same accuracy.
Sample complexity of RMDPs under the  $\chi^2$  distance. We summarize the results and comparisons to prior works in Table 2; see Figure 1(b) for an illustration.
- Near-optimal sample complexity. It is established that DRVI reaches  $\varepsilon$  accuracy as soon as the sample complexity is on the order of  $\widetilde{O}\left(\frac{SA(1 + \sigma)}{(1 - \gamma)^4\varepsilon^2}\right)$  for all  $\sigma \in (0,\infty)$ , which is the first sample complexity that scales linearly with respect to the size of the state space  $S$  in the infinite-horizon setting, breaking the quadratic scaling bottleneck presented in prior works (Panaganti and Kalathil, 2022; Yang et al., 2022). We have also developed a strengthened lower bound that is optimized by leveraging the geometry of the uncertainty set under different ranges of  $\sigma$ . Comparing the two bounds, they match at  $\widetilde{O}\left(\frac{SA}{(1 - \gamma)^4\varepsilon^2}\right)$  when  $\sigma \asymp 1$ , and have a bounded gap no larger than a polynomial factor of the effective horizon  $1 / (1 - \gamma)$  over the entire range of the uncertainty level, again significantly improving prior art that exhibits an unbounded gap.
- RMDPs can be harder than standard MDPs under the  $\chi^2$  distance. Somewhat surprisingly, the new lower bound suggests that RMDPs in this case can be much harder than standard MDPs, at least for certain ranges of the uncertainty level. We single out two regimes of particular interest. First, when  $\sigma \asymp 1$ , the sample size requirement of RMDPs is  $\widetilde{O}\left(\frac{SA}{(1 - \gamma)^4\varepsilon^2}\right)$ , which is provably harder than
standard MDPs by a factor of the effective horizon  $\frac{1}{1 - \gamma}$ . Second, when  $\sigma \gtrsim \frac{1}{(1 - \gamma)^3}$ , the lower bound exceeds the sample complexity of standard MDPs and keeps growing linearly with the uncertainty level  $\sigma$ .
In sum, our sample complexity bounds not only strengthen the prior art in both directions of upper and lower bounds, but also reveal new insights on how the additional consideration of distributional robustness fundamentally changes the sample complexity of RL in a surprising manner. It turns out that RMDPs are not necessarily harder nor easier than standard MDPs, but the answer is far more nuanced and highly dependent on both the size and shape of the uncertainty set, which are not elucidated in prior analyses.
In this section, we introduce the model of distributionally robust Markov decision processes (RMDPs) focusing on the discounted infinite-horizon setting, the sampling mechanism, as well as our goal.
Standard MDPs. To begin, we first introduce the standard Markov decision process (MDP), which facilitates the understanding of RMDPs. A discounted infinite-horizon MDP is represented by  $\mathcal{M} = (\mathcal{S},\mathcal{A},P,\gamma ,r)$ , where  $\mathcal{S} = \{1,\dots ,S\}$  and  $\mathcal{A} = \{1,\dots ,A\}$  are the finite state and action spaces, respectively,  $\gamma \in [0,1)$  is the discount factor,  $P:S\times \mathcal{A}\to \Delta (\mathcal{S})$  denotes the probability transition kernel, and  $r:S\times \mathcal{A}\rightarrow [0,1]$  is the immediate reward function which is assumed to be deterministic. A policy is denoted as  $\pi :S\to \Delta (\mathcal{A})$ , which specifies the action selection probability over the action space in any state. When the policy is deterministic, we overload the notation and refer to  $\pi (s)$  as the action selected by policy  $\pi$  in state  $s$ . To characterize the long term cumulative reward, the value function  $V^{\pi ,P}$  for any policy  $\pi$  under the transition kernel  $P$  is defined by
$$
\forall s \in \mathcal {S}: \quad V ^ {\pi , P} (s) := \mathbb {E} _ {\pi , P} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r \left(s _ {t}, a _ {t}\right) \mid s _ {0} = s \right], \tag {1}
$$
where the expectation is taken over the randomness of the trajectory  $\{s_t, a_t, r_t\}_{t=0}^{\infty}$  generated by executing policy  $\pi$  under the transition kernel  $P$ , namely,  $a_t \sim \pi(s_t)$ , and  $s_{t+1} \sim P(\cdot | s_t, a_t)$ . Similarly, the Q-function  $Q^{\pi, P}$  associated with any policy  $\pi$  under the transition kernel  $P$  is defined as
$$
\forall (s, a) \in \mathcal {S} \times \mathcal {A}: \quad Q ^ {\pi , P} (s, a) := \mathbb {E} _ {\pi , P} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r \left(s _ {t}, a _ {t}\right) \mid s _ {0} = s, a _ {0} = a \right], \tag {2}
$$
where the expectation is again taken over the randomness of the trajectory.
Distributionally robust MDPs. In this work, we focus on the discounted infinite-horizon distributionally robust MDP (RMDP), denoted as  $\mathcal{M}_{\mathrm{rob}} = \{\mathcal{S},\mathcal{A},\mathcal{U}_{\rho}^{\sigma}(P^{0}),\gamma ,r\}$ , where  $\mathcal{S},\mathcal{A},\gamma ,r$  are defined the same as those in the above standard MDP. A key distinction from the standard MDP, is that rather than assuming a fixed transition kernel  $P$ , it postulates the transition kernel lies in an uncertainty set  $\mathcal{U}_{\rho}^{\sigma}(P^{0})$  centered around a nominal kernel  $P^0:\mathcal{S}\times \mathcal{A}\to \Delta (\mathcal{S})$ , where the uncertainty set is specified using some distance metric  $\rho$  of radius  $\sigma >0$ . In particular, given the nominal transition kernel  $P^0$  and some uncertainty level  $\sigma$ , the uncertainty set—with divergence  $\rho :\Delta (S)\times \Delta (S)\to \mathbb{R}^{+}$ —is specified as
$$
\mathcal {U} _ {\rho} ^ {\sigma} \left(P ^ {0}\right) := \otimes \mathcal {U} _ {\rho} ^ {\sigma} \left(P _ {s, a} ^ {0}\right), \quad \mathcal {U} _ {\rho} ^ {\sigma} \left(P ^ {0}\right) := \left\{P _ {s, a} \in \Delta (\mathcal {S}): \rho \left(P _ {s, a}, P _ {s, a} ^ {0}\right) \leq \sigma \right\}, \tag {3}
$$
where we denote a vector of the transition kernel  $P$  or  $P^0$  at state-action pair  $(s,a)$  respectively as
$$
P _ {s, a} := P (\cdot \mid s, a) \in \mathbb {R} ^ {1 \times S}, \quad P _ {s, a} ^ {0} := P ^ {0} (\cdot \mid s, a) \in \mathbb {R} ^ {1 \times S}. \tag {4}
$$
In other words, the uncertainty is imposed in a separate manner for each state-action pair, obeying the so-called  $(s,a)$ -rectangularity (Zhou et al., 2021; Wiesemann et al., 2013). In RMDPs, we are interested in the worst-case performance of a policy  $\pi$  over all the possible transition kernels in the uncertainty set. This is measured by the robust value function  $V^{\pi,\sigma}$  and the robust  $Q$ -function  $Q^{\pi,\sigma}$  in  $\mathcal{M}_{\mathrm{rob}}$ , defined respectively as
$$
\forall (s, a) \in \mathcal {S} \times \mathcal {A}: \quad V ^ {\pi , \sigma} (s) := \inf  _ {P \in \mathcal {U} _ {\rho} ^ {\sigma} (P ^ {0})} V ^ {\pi , P} (s), \qquad Q ^ {\pi , \sigma} (s, a) := \inf  _ {P \in \mathcal {U} _ {\rho} ^ {\sigma} (P ^ {0})} Q ^ {\pi , P} (s, a).
$$
Optimal robust policy and robust Bellman operator. Generalizing standard MDPs, it is well-known that there exists at least one deterministic policy that maximizes the robust value function and
the robust  $Q$ -function simultaneously (Iyengar, 2005; Nilim and El Ghaoui, 2005). Therefore, we denote the optimal robust value function (resp. optimal robust  $Q$ -function) as  $V^{\star,\sigma}$  (resp.  $Q^{\star,\sigma}$ ), and the optimal robust policy as  $\pi^{\star}$ , which satisfies
$$
\forall s \in \mathcal {S}: V ^ {\star , \sigma} (s) := V ^ {\pi^ {\star}, \sigma} (s) = \max  _ {\pi} V ^ {\pi , \sigma} (s), \tag {5a}
$$
$$
\forall (s, a) \in \mathcal {S} \times \mathcal {A}: Q ^ {\star , \sigma} (s, a) := Q ^ {\pi^ {\star}, \sigma} (s, a) = \max  _ {\pi} Q ^ {\pi , \sigma} (s, a). \tag {5b}
$$
The robust Bellman operator (Iyengar, 2005; Nilim and El Ghaoui, 2005) is denoted as  $\mathcal{T}^{\sigma}(\cdot):\mathbb{R}^{SA}\to \mathbb{R}^{SA}$ , which is defined as follows: for all  $(s,a)\in S\times \mathcal{A}$
$$
\mathcal {T} ^ {\sigma} (Q) (s, a) := r (s, a) + \gamma \inf  _ {\mathcal {P} \in \mathcal {U} _ {\rho} ^ {\sigma} \left(P _ {s, a} ^ {0}\right)} \mathcal {P} V, \quad \text {w i t h} \quad V (s) := \max  _ {a} Q (s, a). \tag {6}
$$
Given that  $Q^{\star, \sigma}$  is the unique fixed point of  $\mathcal{T}^\sigma$ , one can recover the optimal robust value function and Q-function using a procedure termed distributionally robust value iteration—which generalizes the standard value iteration—by recursively applying the robust Bellman operator from some fixed initialization. In addition, this procedure converges rather fast due to the nice  $\gamma$ -contraction property of  $\mathcal{T}^\sigma$  (Iyengar, 2005; Nilim and El Ghaoui, 2005) with respect to the  $\ell_{\infty}$  norm.
Specification of the divergence  $\rho$ . We consider two popular choices of the uncertainty set measured in terms of the  $f$ -divergence: total variation and  $\chi^2$  divergence, given respectively by
$$
\rho_ {\mathrm {T V}} \left(P _ {s, a}, P _ {s, a} ^ {0}\right) := \frac {1}{2} \left\| P _ {s, a} - P _ {s, a} ^ {0} \right\| _ {1} = \frac {1}{2} \sum_ {s ^ {\prime} \in \mathcal {S}} P ^ {0} \left(s ^ {\prime} \mid s, a\right) \left| 1 - \frac {P \left(s ^ {\prime} \mid s , a\right)}{P ^ {0} \left(s ^ {\prime} \mid s , a\right)} \right|, \tag {7}
$$
$$
\rho_ {\chi^ {2}} \left(P _ {s, a}, P _ {s, a} ^ {0}\right) := \sum_ {s ^ {\prime} \in \mathcal {S}} P ^ {0} \left(s ^ {\prime} \mid s, a\right) \left(1 - \frac {P \left(s ^ {\prime} \mid s , a\right)}{P ^ {0} \left(s ^ {\prime} \mid s , a\right)}\right) ^ {2}. \tag {8}
$$
Note that  $\rho_{\mathrm{TV}}(P_{s,a},P_{s,a}^{0})\in [0,1]$  and  $\rho_{\chi^2}(P_{s,a},P_{s,a}^0)\in [0,\infty)$  in general. As we shall see, the two choices convey drastically different messages in the statistical complexity of RMDPs.
Sampling mechanism: a generative model. Following Zhou et al. (2021); Panaganti and Kalathil (2022), we assume the access to a generative model or a simulator (Kearns and Singh, 1999), which allows us to collect  $N$  independent samples from the nominal kernel  $P^0$  at each state-action pair: