SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[62] Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust Markov decision processes. Mathematics of Operations Research, 38(1):153-183, 2013.
[63] Huan Xu and Shie Mannor. Distributionally robust Markov decision processes. In Advances in Neural Information Processing Systems, pages 2505-2513, 2010.
[64] Zaiyan Xu, Kishan Panaganti, and Dileep Kalathil. Improved sample complexity bounds for distributionally robust reinforcement learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 9728-9754, 2023.
[65] Mengjiao Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Dichotomy of control: Separating what you can control from what you cannot. arXiv preprint arXiv:2210.13435, 2022.
[66] Wenhao Yang, Liangyu Zhang, and Zhihua Zhang. Toward theoretical understandings of robust markov decision processes: Sample complexity and asymptotics. The Annals of Statistics, 50(6):3223-3248, 2022.
[67] Runyu Zhang, Jincheng Mei, Bo Dai, Dale Schuurmans, and Na Li. On the global convergence rates of decentralized softmax gradient play in markov potential games. Advances in Neural Information Processing Systems, 35:1923-1935, 2022.
[68] Ruida Zhou, Tao Liu, Dileep Kalathil, PR Kumar, and Chao Tian. Anchor-changing regularized natural policy gradient for multi-objective reinforcement learning. Advances in Neural Information Processing Systems, 2022.
[69] Zhengqing Zhou, Qinxun Bai, Zhengyuan Zhou, Linhai Qiu, Jose Blanchet, and Peter Glynn. Finite-sample regret bound for distributionally robust offline tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 3331-3339, 2021.
A Experimental Details and Additional Experimental Results
In this section, we provide details of the RNAC algorithm (Algorithm 1) implemented in the experiments - robust natural actor-critic proximal policy optimization (RNAC-PPO) algorithm (Algorithm 4). We also demonstrate further experimental results evaluated on different perturbations of physical hyperparameters in Hopper-v3, Walker2d-v3, and HalfCheetah-v3 from OpenAI Gym [7] compared with soft actor-critic (SAC) [18] and soft-robust [11] PPO (SRPPO). Finally, we introduce experimental details of the TurtleBot navigation task.
A.1 RNAC-PPO Algorithm
We provide the RNAC-PPO algorithm in Algorithm 4, where the robust critic is minimizing squared robust TD error (MSRTDE) and the robust natural actor is performing the clipped version of robust PPO (RPPO) for computational efficiency, and we name it RNAC-PPO for simplicity. We implement RNAC-PPO employing neural network (NN) function approximation, where the robust natural actor is utilizing a neural Gaussian policy [48] with two hidden layers of width 64, and the value function in the robust critic is also parameterized by an NN with two hidden layers of width 64. Compared with the canonical empirical Bellman operator in the PPO algorithm, we adopt the robust empirical Bellman operator in the Robust PPO algorithm based on the double-sampling (DS) uncertainty set and the integral probability metric (IPM) uncertainty set, which can be efficiently computed. For a better comparison with SAC in the next subsection, we adopt several modifications (e.g., state normalization, reward scaling, gradient clip, etc) in the implementation of PPO-based algorithms to improve their performance under the nominal model. Note that for fairness we employ the same modification for both PPO and RNAC-PPO algorithms.
We use the same hyperparameters across different MuJoCo environments. Specifically, we select  $\gamma = 0.99$  for the discount factor,  $\eta^t = \alpha^t = 3 \times 10^{-4}$  for learning rates of both actor and critic updates implemented by ADAM [24],  $T = 3 \times 10^6$  for the maximum training steps, and  $B = 2048$  for the batch size.
Output:  $\theta^T$
A.2 Perturbed Mujoco Environments with Other Hyperparameters
In this subsection, we provide more experimental results on different perturbations of physical hyperparameters (e.g., leg_joint_damping, leg_joint_stiffness, and front_joint_stiffness) in Hopper-v3, Walker2d-v3, and HalfCheetah-v3. Figure 4 shows that the RNAC-PPO algorithm is consistently robust compared to the PPO algorithm.
A.3 Comparison with Soft Actor-Critic and Soft-Robust PPO
In Section 7, we demonstrate the robust behavior of the RNAC-PPO algorithm compared with the PPO algorithm in Figure 1. In this subsection, we add two more baselines for robust algorithms: soft actor-critic (SAC) [18] and soft-robust [11] PPO (SRPPO). SAC is regarded as one of the robust baselines since maximum entropy RL (e.g. SAC) was shown to solve some robust RL problems by maximizing the lower bound on a robust RL objective [13]. Soft-robust RL learns an optimal policy based on a distribution over an uncertainty set instead of considering the worst-case scenario [11]. For a fair comparison, we implement the idea of soft robustness into the framework of PPO. Specifically, we build an uncertainty set wrapping up four environments, including one nominal environment and three perturbed environments with leg_joint_stiffness=5.0 (default=0.0), gravity=-9.50 (default=-9.81), and actuator_ctrlrange=(-0.95, 0.95) (default=(-1.0, 1.0)), respectively. Additionally, we select [0.85, 0.05, 0.05, 0.05] as the distribution over the above uncertainty set for the SRPPO algorithm.
Figures 5c and 5d show that the cumulative rewards of RNAC-PPO decay much slower compared with those of PPO, but similar to those of SAC under the perturbation of gravity. This verifies the claim that SAC can solve some robust RL problems [13]. However, as shown in Figures 5a and 5b, SAC is not a panacea for all robust RL problems. Under the perturbation of leg_joint_stiffness, SAC suffers faster cumulative rewards decay compared with RNAC-PPO. Since SRPPO considers a distribution over an uncertainty set instead of only sampling from the nominal model, it doesn't perform well under the nominal model, but it shows a fairly robust behavior in Figures 5a and 5b when leg_joint_stiffness values are perturbed.
Since the cumulative rewards of SAC ( $>14000$ ) are much higher than those of PPO-based algorithms ( $<3000$ ) in HalfCheetah-v3, we only report the results of SAC in this subsection for Hopper-v3 to prevent the information of robustness from being blurred. Additionally, the training time of RNAC-PPO, PPO, and SRPPO is similar, which is at least 5 times less than that of SAC to
end the training of 3 million steps. This is due to the fact that PPO-based algorithms require fewer updates for critic  $V$ , while SAC requires more updates for critic Q.
A.4 TurtleBot Experiment Details
The goal of this TurtleBot navigation task is to guide the robot to any desired target within a 2-meter range. The state space is a 2-dimensional continuous space, comprising the distance and relative orientation between the robot and the target. The robot is fixed to move towards its front with a linear velocity of  $15~\mathrm{cm / s}$ , while its angular velocity is controlled by the algorithm. Action space is 1-dimensional and continuous, ranging from  $-2$  to 2. The action signal is then linearly scaled into  $[-1.5\mathrm{rad / s}, 1.5\mathrm{rad / s}]$ . The reward function is designed to be proportional to the product of distance and action-scaled orientation between the robot and the target. Hitting the boundary or reaching the target would cause a reward of  $-200$  or  $200$  respectively. One trajectory would end when the robot hits the boundary, reaches the goal, or the elapsed time is more than 150 seconds.
We train both PPO and RNAC-PPO(IPM) under the simulator Gazebo. To introduce stochasticity into the originally deterministic Gazebo environment, we apply uniform action noise perturbation. PPO and RNAC-PPO(DS) are trained under a balanced Unif[-0.5, 0.5] noise-perturbed environment. All algorithms undergo 400 epochs of training, and the policy with the highest speed is saved. Subsequently, we evaluate all policies in an unbalanced Unif[1.0, 1.5] noise-perturbed environment. We employ such a high noise since all learned algorithms are aggressive at fast turning, with action close to the limit of -2 or 2. The trajectories of the robot under Gazebo simulator and a video for the real-world experiment are demonstrated in Figure 2 and [Video Link].
Specifically, both PPO and RNAC-PPO are implemented by neural networks. The actor network is defined by the Gaussian policy, with one hidden layer and tanh activation function, projecting the
input state into a feature with dimension 100, then into the 1-dimensional action. Critic network consists of one hidden layer, with a width 100, and a ReLU activation function. We choose discount factor as 0.99, learning rate as  $2.5 \times 10^{-4}$ , batch size as 64, and optimizer ADAM.
Additionally, we also add more baselines (i.e., dynamics randomization [44] and action noise envelope [20]) to demonstrate the robustness of the proposed methods in the real-world TurtleBot environment. As shown in Figure 6, the proposed algorithm RNAC-PPO (DS / IPM) enjoys higher target reaching rates  $(100\% / 100\%)$ , compared with action noise envelope  $(67.5\% / 13\%)$ , dynamic randomization  $(9\% / 2\%)$ , and PPO  $(0\% / 0\%)$ , under perturbed testing environments, which illustrates the robustness of the RNAC-PPO algorithm.
We end this section by illustrating our hardware configurations. All experimental results are carried out on a Linux server with 48-core RTX 6000 GPUs, 48-core Intel Xeon 6248R CPUs, and 384 GB DDR4 RAM.
B Discussions on Uncertainty Sets
$R$ -contamination For  $R$ -contamination [60], the uncertainty set is  $\mathcal{P} = \otimes_{s,a}\mathcal{P}_{s,a}$ , where  $\mathcal{P}_{s,a} = \{Rq + (1 - R)p_{s,a}^o : q \in \Delta_S\}$ . It can be shown that
$$
\inf  _ {p \in \mathcal {P} _ {s, a}} p ^ {\top} V = (1 - R) \left(p _ {s, a} ^ {\circ}\right) ^ {\top} V + R \min  _ {s ^ {\prime}} V \left(s ^ {\prime}\right). \tag {8}
$$
Its corresponding robust Bellman operator requires searching the entire state space  $\mathcal{S}$  to calculate the minimum value in  $V$ , and thus intractable for large state space.
$\ell_{\mathfrak{p}}$  norm [28] For any nominal model  $p^{\circ}$ , the uncertainty set is  $\mathcal{P} = \otimes_{s,a}\mathcal{P}_{s,a}$  with  $\mathcal{P}_{s,a} = \{p:\| q - p_{s,a}^{\circ}\|_{\mathfrak{p}}\leq \delta, \sum_{s'}q(s') = 1\}$ , where  $\| \cdot \|_{\mathfrak{p}}$  is an  $\ell_{\mathfrak{p}}$  norm with its dual norm  $\| \cdot \|_{\mathfrak{q}}$  satisfying  $\frac{1}{\mathfrak{p}} +\frac{1}{\mathfrak{q}} = 1$ , and the domain of  $q\in \Delta_S$  is relaxed to hyperplane  $\sum_{s'}q(s') = 1$  due to the difficulty in handling the boundary of  $\Delta_S$ . It was shown [28] that
$$
\inf  _ {p \in \mathcal {P} _ {s, a}} p ^ {\top} V = \left(p _ {s, a} ^ {\circ}\right) ^ {\top} V - \min  _ {w} \| V - w \mathbf {1} \| _ {\mathbf {q}}, \tag {9}
$$
which requires solving a minimization problem via binary search. The minimum value may have a closed-form representation (c.f. Table 1 in [27]). For example, the minimum value is the average of  $V$  for  $\ell_2$  norm, the median of  $V$  for  $\ell_1$  norm, and the average peak of  $V$  (i.e., the average of the maximum and minimum value of  $V$ ) for  $\ell_{\infty}$  norm.  $\ell_{\mathfrak{p}}$  norm-based uncertainty set is thus intractable in the large-scale scenario.
$f$ -divergence Given a continuous strict convex function  $f: \mathbb{R}_+ \to \mathbb{R}$  with  $f(1) = 0$ ,  $f$ -divergence is defined by  $\mathrm{d}_f(q, p) = \sum_s p(s) f\left(\frac{q(s)}{p(s)}\right)$ . By distributionally robust optimization literature [12], we have
$$
\inf  _ {p \in \mathcal {P} _ {s, a}} p ^ {\top} V = \sup  _ {\lambda > 0, \eta \in \mathbb {R}} \mathbb {E} _ {s ^ {\prime}} \left[ - \lambda f ^ {*} \left(\frac {- V \left(s ^ {\prime}\right) - \eta}{\lambda}\right) - \lambda \delta - \eta \right], \tag {10}
$$
where  $f^{*}$  is the Fenchel conjugate of  $f$ , i.e.,  $f^{*}(y) = \sup_{x > 0}(yx - f(x))$ . Given such a formula, the robust Bellman operator can be estimated given samples from the nominal model but requires the optimal dual variables  $\lambda, \eta$  for each  $(s, a)$  pair.
Wasserstein distance Given some metric  $d(\cdot, \cdot)$  on state space  $S$ , Wasserstein- $\sigma$  distance is defined for any  $p, q \in \Delta_S$  that
$$
W _ {\sigma} (p, q) := (\inf _ {\mu \in U (p, q)} \mathbb {E} _ {(s, s ^ {\prime}) \sim \mu} [ d (s, s ^ {\prime}) ^ {\sigma} ]) ^ {\frac {1}{\sigma}},
$$
where  $U(p,q)$  is the set of all couplings of  $p$  and  $q$ . Wasserstein RMDP has uncertainty set  $\mathcal{P} = \otimes_{s,a}\mathcal{P}_{s,a}$  with  $\mathcal{P}_{s,a} = \{q:W_{\sigma}(q,p_{s,a}^{\circ})\leq \delta \}$  and we have
$$
\inf  _ {p \in \mathcal {P} _ {s, a}} p ^ {\top} V = \sup  _ {\lambda > 0} \sum_ {s ^ {\prime}} p _ {s, a} ^ {\circ} \left(s ^ {\prime}\right) \inf  _ {\tilde {s}} \left[ V \left(\tilde {s}\right) + \lambda \left(d \left(\tilde {s}, s ^ {\prime}\right) ^ {\sigma} - \delta\right) \right]. \tag {11}
$$
The Wasserstein RMDP has been previously studied [1, 26]. Kuang et al. [26] gives a state disturbance view of Wasserstein RMDP, where the robust Bellman operator requires searching the worst-case state in the vicinity of the next state sample, i.e.,  $\inf_{\tilde{s}:\| \tilde{s} -s^{\prime}\| \leq \delta}V(\tilde{s})$ . Although this approach can be applied to RMDP with large state space, the theoretical guarantee provided in [26] is only for policy iteration (planning problem) in the tabular setting (i.e., the robust Bellman operator is a contraction mapping under  $\| \cdot \|_{\infty}$ ).
B.1 Double-Sampling Uncertainty Sets
When action  $a$  is taken at state  $s$ , the nominal model transits to the next state  $s' \sim p_{s,a}^{0}$ . It can be viewed as a double-sampling process that the next state  $s'$  is generated by uniformly sampling one from  $m$  states  $s_1', s_2', \ldots, s_m'$  sampled i.i.d. according to  $p_{s,a}^{0}$ . The transition kernel in the DS uncertainty set at  $(s,a)$ -pair can be viewed as selecting the next state  $s'$  in  $\{s_1', \ldots, s_m'\}$  according to a distribution  $\alpha \in \Delta(m)$  that is perturbed from a uniform distribution  $\mathrm{Unif}(m)$  and potentially depends on the samples  $\{s_1', \ldots, s_m'\}$ . The DS uncertainty set is implicitly defined by a choice of  $m$ , divergence measure  $\mathrm{d}(\cdot, \cdot)$  and radius  $\delta > 0$ .
Bellman completeness for robust linear MDP under DS uncertainty set: A Q function class  $\mathcal{G} \subset \mathbb{R}^{S \times \mathcal{A}}$  is said to be Bellman complete if it is closed, and for all  $g \in \mathcal{G}$ ,  $\mathcal{T}^* g$  also lies in  $\mathcal{G}$  where
$$
\mathcal {T} ^ {*} g (s, a) := r (s, a) + \gamma \sum_ {s ^ {\prime}} p _ {s, a} \left(s ^ {\prime}\right) \max  _ {a} g \left(s ^ {\prime}, a\right).
$$
Bellman completeness is critical for reinforcement learning. For RMDP with the Double-Sampling (DS) uncertainty set, the linear function approximation can satisfy Bellman completeness if the nominal model is a linear MDP. In linear MDP [22], we have
$$
p _ {s, a} ^ {\circ} \left(s ^ {\prime}\right) = \phi (s, a) ^ {\top} \boldsymbol {\mu} \left(s ^ {\prime}\right), \quad r (s, a) = \phi (s, a) ^ {\top} \theta ,
$$
where  $\pmb{\mu}$  is a vector of (signed) measures over  $S$ . Then for any  $V \in \mathbb{R}^S$
$$
\begin{array}{l} r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim p _ {s, a} ^ {\circ}} [ V (s ^ {\prime}) ] = r (s, a) + \gamma \int_ {s ^ {\prime}} \phi (s, a) ^ {\top} \boldsymbol {\mu} (s ^ {\prime}) V (s ^ {\prime}) d s ^ {\prime} \\ = \phi (s, a) ^ {\top} \left(\theta + \gamma \int_ {s ^ {\prime}} \boldsymbol {\mu} \left(s ^ {\prime}\right) V \left(s ^ {\prime}\right) d s ^ {\prime}\right) \in s p a n (\Phi). \\ \end{array}
$$
For the DS uncertainty set  $\mathcal{P} = \otimes_{s,a}\mathcal{P}_{s,a}$  determined by  $m, \mathrm{d}(\cdot, \cdot), \delta$ , let  $f_V(s_{1:m}') := \inf_{\alpha \in \Delta_{[m]}\colon d(\alpha, \mathrm{Unif}([m])) \leq \delta} \sum_{i=1}^{m} \alpha_i V(s_i')$ , and  $\bar{f}_V(s_1') := \mathbb{E}_{s_{2:m}'} \underset{\sim}{\overset{i.i.d.}{\sim}} p_{s,a}^{\circ}[f_V(s_{1:m}')]$ . It then follows that for any  $V$
$$
\begin{array}{l} r(s,a) + \gamma \inf_{p\in \mathcal{P}_{s,a}}p^{\top}V = r(s,a) + \gamma \mathbb{E}_{s^{\prime}_{1:m}}{}_{i,i,d.}{}_{\sim}{}_{p^{\circ}_{s,a}}[f_{V}(s^{\prime}_{1:m})] \\ = \phi (s, a) ^ {\top} \theta + \gamma \mathbb {E} _ {s _ {1} ^ {\prime} \sim p _ {s, a} ^ {\circ}} [ \bar {f} _ {V} (s _ {1} ^ {\prime}) ] = \phi (s, a) ^ {\top} \left(\theta + \gamma \int_ {s ^ {\prime}} \boldsymbol {\mu} (s ^ {\prime}) \bar {f} _ {V} (s ^ {\prime}) d s ^ {\prime}\right) \in s p a n (\Phi). \\ \end{array}
$$
The Bellman completeness for DS-based RMDP with linear MDP nominal model is thus proved.
B.2 Integral Probability Metric Uncertainty Sets
Under the linear value function approximation,  $V_{w}(s) = \psi (s)^{\top}w$ . Denote  $\Psi \in \mathbb{R}^{S\times d}$  as the feature matrix by stacking up  $\psi (s)^{\top}$ , and the value function approximation is a linear regressor  $V_{w} = \Psi w\in \mathbb{R}^{S}$ . Let  $\Psi = [\psi_1,\dots \psi_d]$  with  $\psi_i\in \mathbb{R}^S$ , without loss of generality, assume  $\psi_{1}$  is an all-one vector, which corresponds to the bias term of the linear regressor, and  $\Psi$  is full column rank.
We propose the IPM  $\mathrm{d}_{\mathcal{F}}(p,q) = \sup_{f\in \mathcal{F}}\{p^{\top}f - q^{\top}f\}$  determined by function class
$$
\mathcal {F} = \left\{\Psi \xi : \xi \in \mathbb {R} ^ {d}, \| \xi \| \leq 1 \right\} \quad \text {r e s t a t e m e n t o f E q (5) i n m a t r i x f o r m}.
$$
The robust Bellman operator  $\mathcal{T}_{\mathcal{P}}^{\pi}V$  (2) requires solving the optimization  $\inf_{q\in \mathcal{P}_{s,a}}q^{\top}V$ , which is equivalent to
$$
\min  _ {q \in \Delta_ {S}} q ^ {\top} V \quad s. t. \sup  _ {f \in \mathcal {F}} q ^ {\top} f - (p _ {s, a} ^ {\circ}) ^ {\top} f \leq \delta .
$$
We relax the domain constraint  $q \in \Delta_{\mathcal{S}}$  to  $\sum_{s} q(s) = 1$  and define  $\mathcal{P}_{s,a} = \{q : \mathrm{d}_{\mathcal{F}}(q, p_{s,a}^{\circ}) \leq \delta, \sum_{s} q(s) = 1\}$ . This relaxation omits the boundary effect of  $\Delta_{\mathcal{S}}$ , which facilitates the following analysis. The relaxation is also made for  $\ell_p$  norm-based uncertainty set [28], and one can argue that it does not introduce any relaxation error for  $p_{s,a}^{\circ}(s') > 0, \forall s, a, s'$  and small  $\delta$ .
Restatement of Proposition 1. For the IPM with  $\mathcal{F}$  in (5), we have  $\inf_{q\in \mathcal{P}_{s,a}}q^{\top}V_w = (p_{s,a}^{\circ})^{\top}V_w - \delta \| w_{2:d}\|$ .
Proof of Proposition 1. Denote  $u = q - p_{s,a}^{\circ}$ . The inf  $q \in \mathcal{P}_{s,a}$  under the relaxation is
$$
\min  _ {u} \left(p _ {s, a} ^ {\circ}\right) ^ {\top} \Psi w + u ^ {\top} \Psi w, \quad s. t. \sup  _ {\| \xi \| \leq 1} u ^ {\top} \Psi \xi \leq \delta , u ^ {\top} \mathbf {1} = 0.
$$
Since  $\Psi$  is full column rank and  $\psi_{1}$  is an all 1 vector, we know  $\{\Psi^{\top}u: u^{\top}\mathbf{1} = 0, u \in \mathbb{R}^{S}\} = \{y \in \mathbb{R}^{d}: y_{1} = 0\}$ . The optimization problem can then be written as
$$
\min  _ {y} \left(p _ {s, a} ^ {\circ}\right) ^ {\top} \Psi w + y _ {2: d} ^ {\top} w _ {2: d}, \quad s. t. \sup  _ {\| \xi_ {2: d} \| \leq 1} y _ {2: d} ^ {\top} \xi_ {2: d} \leq \delta .
$$
The constraint is equivalent to  $\| y_{2:d}\|_*\leq \delta$  and the optimal value can then be written as  $(p_{s,a}^{\circ})^{\top}V_{w} - \delta \| w_{2:d}\|$ , which concludes the proof of Proposition 1.
The IPM has many merits due to its capability to take advantage of the geometry of the domain (state space) through the function class.
C Robust Critic Analysis
We analyze the robust critic component in the RNAC algorithm (Algorithm 1). This section would also be of independent interest for the robust policy evaluation problem, where one aims to estimate the robust value of some policy given Markovian data.
The first two subsections focus on the linear function approximation and prove the main theorems as stated in Section 5 of the main paper. The last subsection focuses on the general function approximation.
C.1 Linear Robust Value Function Approximation
Setting: We aim to approximate the robust value function  $V^{\pi}$  thorough linear function class
$$
V _ {w} (s) = \psi (s) ^ {\top} w, \quad \forall s \in \mathcal {S}. \tag {12}
$$
The "optimal" linear value approximation  $V_{w^{\pi}}$  is the solution of the projected robust Bellman equation i.e.,
$$
\Pi^ {\pi} \mathcal {T} _ {\mathcal {P}} ^ {\pi} V _ {w ^ {\pi}} = V _ {w ^ {\pi}},
$$
where the projection matrix is  $\Pi^{\pi} = \Psi (\Psi^{\top}D^{\pi}\Psi)^{-1}\Psi^{\top}D^{\pi}$  with  $D^{\pi} = \mathrm{diag}(\nu^{\pi})$  and  $\nu^{\pi}$  is the state stationary distribution of executing policy  $\pi$  on the nominal model  $p^\circ$ .
Definition 3 (Linear value approximation error).  $\epsilon_{V,bias} := \sup_{\pi} \| \Pi^{\pi} V^{\pi} - V^{\pi} \|_{\nu^{\pi}}$ .
$w^{\pi}$  is the solution of the projected robust Bellman equation  $\Pi^{\pi} \mathcal{T}_{\mathcal{P}}^{\pi} V_{w} = V_{w}$ . When  $\Pi^{\pi} \mathcal{T}_{\mathcal{P}}^{\pi}$  is  $\beta$ -contraction w.r.t.  $\| \cdot \|_{\nu^{\pi}}$ , we have (according to Corollary 4 in [53])
$$
\left\| V _ {w ^ {\pi}} - V ^ {\pi} \right\| _ {\nu^ {\pi}} \leq \frac {1}{1 - \beta} \left\| \Pi^ {\pi} V ^ {\pi} - V ^ {\pi} \right\| _ {\nu^ {\pi}} \leq \frac {\epsilon_ {V , b i a s}}{1 - \beta}. \tag {13}
$$
C.1.1 Contraction of Projected Robust Bellman Operator
The contraction property of the projected robust Bellman operator is the key to guaranteeing the convergence of linear TD algorithms. The contraction property can be guaranteed when Assumption 2 is satisfied, and we show that the DS uncertainty set with a small  $\delta$  indeed satisfies this assumption.
We refer to a divergence  $\mathrm{d}(\cdot, \cdot)$  as  $C$ -continuous at uniform distribution  $\mathrm{Unif}([m])$ , if  $\{\alpha \in \Delta_{[m]} : \mathrm{d}(\alpha, \mathrm{Unif}([m])) \leq \epsilon\} \subseteq \{\alpha \in \Delta_{[m]} : \| \alpha - \mathrm{Unif}([m]) \|_{\infty} \leq C\epsilon\}$ ,  $\forall \epsilon > 0$ .
Proposition 4. Uncertainty set implicitly defined by double sampling with divergence  $\mathrm{d}(\cdot, \cdot)$  that is  $C$ -continuous at uniform distribution  $\text{Unif}([m])$  and  $\delta < \frac{1 - \gamma}{\gamma mC}$  satisfies Assumption 2.
Proof. For the uncertainty set  $\mathcal{P}$  implicitly defined by double-sampling with robust Bellman operator 3
$$
(\mathcal {T} _ {\mathcal {P}} ^ {\pi} V) (s) := \mathbb {E} _ {a \sim \pi (\cdot | s), s _ {1: m} ^ {\prime} \sim p _ {s, a} ^ {\circ}} \left[ r (s, a) + \gamma \inf _ {\alpha \in \Delta_ {[ m ]}: \mathrm {d} (\alpha , \mathrm {U n i f} ([ m ])) \leq \delta} \sum_ {i = 1} ^ {m} \alpha_ {i} V (s _ {i} ^ {\prime}) \right].
$$
Since  $\mathrm{d}(\cdot, \cdot)$  is  $C$ -continuous at uniform distribution, take arbitrary  $(s, a, s')$ , we know
$$
\begin{array}{l} \sup  _ {q \in \mathcal {P} _ {s, a}} q (s ^ {\prime}) = \mathbb {E} _ {s _ {1: m} ^ {\prime} \sim p _ {s, a} ^ {\circ}} \left[ \sup  _ {\alpha \in \Delta_ {[ m ]}: \mathrm {d} (\alpha , \operatorname {U n i f} (m)) \leq \delta} \sum_ {i = 1} ^ {m} \alpha_ {i} \mathbb {1} (s _ {i} ^ {\prime} = s ^ {\prime}) \right] \\ \leq \mathbb {E} _ {s _ {1: m} ^ {\prime} \sim p _ {s, a} ^ {\circ}} \left[ \sup  _ {\alpha \in \Delta_ {[ m ]}: \| \alpha - \operatorname {U n i f} (m) \| _ {\infty} \leq C \delta} \sum_ {i = 1} ^ {m} \alpha_ {i} \mathbb {1} \left(s _ {i} ^ {\prime} = s ^ {\prime}\right) \right] \\ \leq \mathbb {E} _ {s _ {1: m} ^ {\prime} \sim p _ {s, a} ^ {\circ}} \left[ \sum_ {i = 1} ^ {m} (\frac {1}{m} + C \delta) \mathbb {1} (s _ {i} ^ {\prime} = s ^ {\prime}) \right] = p _ {s, a} ^ {\circ} (s ^ {\prime}) (1 + m C \delta). \\ \end{array}
$$
To guarantee  $\gamma \sup_{q\in \mathcal{P}_{s,a}}q(s^{\prime})\leq \beta p_{s,a}^{\circ}$  with some  $\beta < 1$ , we know need  $(1 + mC\delta) < \frac{1}{\gamma}$ , which can be achieved by  $\delta < \frac{1 - \gamma}{\gamma mC}$ .