SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wenhao Ding $^{1*}$  Laixi Shi $^{2*}$  Yuejie Chi $^{1}$  Ding Zhao $^{1}$ \
 $^{1}$ Carnegie Mellon University\
 $^{2}$ California Institute of Technology\
{wenhaod,laixis,yuejie}@andrew.cmu.edu, dingzhao@cmu.edu
Robustness has been extensively studied in reinforcement learning (RL) to handle various forms of uncertainty such as random perturbations, rare events, and malicious attacks. In this work, we consider one critical type of robustness against spurious correlation, where different portions of the state do not have correlations induced by unobserved confounders. These spurious correlations are ubiquitous in real-world tasks, for instance, a self-driving car usually observes heavy traffic in the daytime and light traffic at night due to unobservable human activity. A model that learns such useless or even harmful correlation could catastrophically fail when the confounder in the test case deviates from the training one. Although motivated, enabling robustness against spurious correlation poses significant challenges since the uncertainty set, shaped by the unobserved confounder and causal structure, is difficult to characterize and identify. Existing robust algorithms that assume simple and unstructured uncertainty sets are therefore inadequate to address this challenge. To solve this issue, we propose Robust State-Confounded Markov Decision Processes (RSC-MDPs) and theoretically demonstrate its superiority in avoiding learning spurious correlations compared with other robust RL counterparts. We also design an empirical algorithm to learn the robust optimal policy for RSC-MDPs, which outperforms all baselines in eight realistic self-driving and manipulation tasks. Please refer to the website for more details.
Reinforcement learning (RL), aiming to learn a policy to maximize cumulative reward through interactions, has been successfully applied to a wide range of tasks such as language generation [1], game playing [2], autonomous driving [3], etc. While standard RL has achieved remarkable success in simulated environments, a growing trend in RL is to address another critical concern - robustness - with the hope that the learned policy still performs well when the deployed (test) environment deviates from the nominal one used for training [4]. Robustness is highly desirable since the performance of the learned policy could significantly deteriorate due to the uncertainty and variations of the test environment induced by random perturbation, rare events, or even malicious attacks [5, 6].
Despite various types of uncertainty that have been investigated in RL, this work focuses on the uncertainty of the environment with semantic meanings resulting from some unobserved underlying variables. Such environment uncertainty, denoted as structured uncertainty, is motivated by innumerable real-world applications but still receives little attention in sequential decision-making tasks [7]. To specify the phenomenon of structured uncertainty, let us consider a concrete example (illustrated in Figure 1) in a driving scenario, where a shift between training and test environments caused by an unobserved confounder can potentially lead to a severe safety issue. Specifically, the observations brightness and traffic density do not have cause and effect on each other but are
controlled by a confounder (i.e. sun and human activity) that is usually unobserved  $^2$  to the agent. During training, the agent could memorize the spurious state correlation between brightness and traffic density, i.e., the traffic is heavy during the daytime but light at night. However, such correlation could be problematic during testing when the value of the confounder deviates from the training one, e.g., the traffic becomes heavy at night due to special events (human activity changes), as shown at the bottom of Figure 1. Consequently, the policy dominated by the spurious correlation in training fails on out-of-distribution samples (observations of heavy traffic at night) in the test scenarios.
The failure of the driving example in Figure 1 is attributed to the widespread and harmful spurious correlation, namely, the learned policy is not robust to the structured uncertainty of the test environment caused by the unobserved confounder. However, ensuring robustness to structured uncertainty is challenging since the targeted uncertain region – the structured uncertainty set of the environment – is carved by the unknown causal effect of the unobserved confounder, and thus hard to characterize. In contrast, prior works concerning robustness in RL [8] usually consider a homogeneous and structure-agnostic uncertainty set around the state [9, 6, 10], action [11, 12], or the training environment [13–15] measured by some heuristic functions [9, 15, 8] to account for unstructured random noise or small perturbations. Consequently, these prior works could not cope with the structured uncertainty since their uncertainty set is different from and cannot tightly cover the desired structured uncertainty set, which could be heterogeneous and allow for potentially large deviations between the training and test
Figure 1: A model trained only with heavy traffic in the daytime learns the spurious correlation between brightness and traffic density and could fail to drive in light traffic in the daytime.
In this work, to address the structured uncertainty, we first propose a general RL formulation called State-confounded Markov decision processes (SC-MDPs), which model the possible causal effect of the unobserved confounder in an RL task from a causal perspective. SC-MDPs better explain the reason for semantic shifts in the state space than traditional MDPs. Then, we formulate the problem of seeking robustness to structured uncertainty as solving Robust SC-MDPs (RSC-MDPs), which optimizes the worst performance when the distribution of the unobserved confounder lies in some uncertainty set. The key contributions of this work are summarized as follows.
- We propose a new type of robustness with respect to structured uncertainty to address spurious correlation in RL and provide a formal mathematical formulation called RSC-MDPs, which are well-motivated by ubiquitous real-world applications.
- We theoretically justify the advantage of the proposed RSC-MDP framework against structured uncertainty over the prior formulation in robust RL without semantic information.
- We implement an empirical algorithm to find the optimal policy of RSC-MDPs and show that it outperforms the baselines on eight real-world tasks in manipulation and self-driving.
2 Preliminary and Limitations of Robust RL
In this section, we first introduce the preliminary formulation of standard RL and then discuss a natural type of robustness that is widely considered in the RL literature and most related to this work - robust RL.
Standard Markov decision processes (MDPs). An episodic finite-horizon standard MDP is represented by  $\mathcal{M} = \{S, A, T, r, P\}$ , where  $S \subseteq \mathbb{R}^n$  and  $A \subseteq \mathbb{R}^{d_A}$  are the state and action spaces, respectively, with  $n / d_A$  being the dimension of state/action. Here,  $T$  is the length of the horizon;  $P = \{P_t\}_{1 \leq t \leq T}$ , where  $P_t : S \times A \to \Delta(S)$  denotes the probability transition kernel at time step  $t$ , for all  $1 \leq t \leq T$ ; and  $r = \{r_t\}_{1 \leq t \leq T}$  denotes the reward function, where  $r_t : S \times A \to [0, 1]$  represents the deterministic immediate reward function. A policy (action selection rule) is denoted by  $\pi = \{\pi_t\}_{1 \leq t \leq T}$ , namely, the policy at time step  $t$  is  $\pi_t : S \to \Delta(A)$  based on the current state  $s_t$  as  $\pi_t(\cdot | s_t)$ . To represent the long-term cumulative
${}^{2}$  sometimes they are observed but ignored given so many variables to be considered in neural networks.
reward, the value function  $V_{t}^{\pi ,P}:S\to \mathbb{R}$  and Q-value function  $Q_{t}^{\pi ,P}:S\times A\to \mathbb{R}$  associated with policy  $\pi$  at step  $t$  are defined as  $V_{t}^{\pi ,P}(s) = \mathbb{E}_{\pi ,P}\big[\sum_{k = t}^{T}r_{k}(s_{k},a_{k})\mid s_{k} = s\big]$  and  $Q_{t}^{\pi ,P}(s,a) = \mathbb{E}_{\pi ,P}\big[\sum_{k = t}^{T}r_{k}(s_{k},a_{k})\mid s_{t} = s,a_{t} = a\big],$  where the expectation is taken over the sample trajectory  $\{(s_t,a_t)\}_{1\leq t\leq T}$  generated following  $a_{t}\sim \pi_{t}(\cdot |s_{t})$  and  $s_{t + 1}\sim P_t(\cdot |s_t,a_t)$ .
Robust Markov decision processes (RMDPs). As a robust variant of standard MDPs motivated by distributionally robust optimization, RMDP is a natural formulation to promote robustness to the uncertainty of the transition probability kernel [13, 15], represented as  $\mathcal{M}_{\mathrm{rob}} = \{\mathcal{S},\mathcal{A},T,r,\mathcal{U}^{\sigma}(P^{0})\}$ . Here, we reuse the definitions of  $S, A, T, r$  in standard MDPs, and denote  $\mathcal{U}^{\sigma}(P^{0})$  as an uncertainty set of probability transition kernels centered around a nominal transition kernel  $P^{0} = \{P_{t}^{0}\}_{1\leq t\leq T}$  measured by some 'distance' function  $\rho$  with radius  $\sigma$ . In particular, the uncertainty set obeying the  $(s,a)$ -rectangularity [16] can be defined over all  $(s,a)$  state-action pairs at each time step  $t$  as
$$
\mathcal {U} ^ {\sigma} \left(P ^ {0}\right) := \otimes \mathcal {U} ^ {\sigma} \left(P _ {t, s, a} ^ {0}\right), \quad \mathcal {U} ^ {\sigma} \left(P _ {t, s, a} ^ {0}\right) := \left\{P _ {t, s, a} \in \Delta (\mathcal {S}): \rho \left(P _ {t, s, a}, P _ {t, s, a} ^ {0}\right) \leq \sigma \right\}, \tag {1}
$$
where  $\otimes$  denotes the Cartesian product. Here,  $P_{t,s,a} \coloneqq P_t(\cdot | s,a) \in \Delta(S)$  and  $P_{t,s,a}^0 \coloneqq P_t^0(\cdot | s,a) \in \Delta(S)$  denote the transition kernel  $P_t$  or  $P_t^0$  at each state-action pair  $(s,a)$  respectively. Consequently, the next state  $s_{t+1}$  follows  $s_{t+1} \sim P_t(\cdot | s_t, a_t)$  for any  $P_t \in \mathcal{U}^\sigma(P_{t,s_t,a_t}^0)$ , namely,  $s_{t+1}$  can be generated from any transition kernel belonging to the uncertainty set  $\mathcal{U}^\sigma(P_{t,s_t,a_t}^0)$  rather than a fixed one in standard MDPs. As a result, for any policy  $\pi$ , the corresponding robust value function and robust  $Q$  function are defined as
$$
V _ {t} ^ {\pi , \sigma} (s) := \inf  _ {P \in \mathcal {U} ^ {\sigma} \left(P ^ {0}\right)} V _ {t} ^ {\pi , P} (s), \quad Q _ {t} ^ {\pi , \sigma} (s, a) := \inf  _ {P \in \mathcal {U} ^ {\sigma} \left(P ^ {0}\right)} Q _ {t} ^ {\pi , P} (s, a), \tag {2}
$$
which characterize the cumulative reward in the worst case when the transition kernel is within the uncertainty set  $\mathcal{U}^{\sigma}(P^{0})$ . Using samples generated from the nominal transition kernel  $P^{0}$ , the goal of RMDPs is to find an optimal robust policy that maximizes  $V_{1}^{\pi, \sigma}$  when  $t = 1$ , i.e., perform optimally in the worst case when the transition kernel of the test environment lies in a prescribed uncertainty set  $\mathcal{U}^{\sigma}(P^{0})$ .
Lack of semantic information in RMDPs. In spite of the rich literature on robustness in RL, prior works usually hedge against the uncertainty induced by unstructured random noise or small perturbations, specified as a small and homogeneous uncertainty set around the nominal one. For instance, in RMDPs, people usually prescribe the uncertainty set of the transition kernel using a heuristic and simple function  $\rho$  with a relatively small  $\sigma$ . However, the unknown uncertainty in the real world could have a complicated and semantic structure that cannot be well-covered by a homogeneous ball regardless of the choice of the uncertainty radius  $\sigma$ , leading to either over conservative policy (when  $\sigma$  is large) or insufficient robustness (when  $\sigma$  is small). Altogether, we obtain the natural motivation of this work: How to formulate such structured uncertainty and ensure robustness against it?
3 Robust RL against Structured Uncertainty from a Causal Perspective
To describe structured uncertainty, we choose to study MDPs from a causal perspective with a basic concept called the structural causal model (SCM). Armed with the concept, we formulate State-confounded MDPs – a broader set of MDPs in the face of the unobserved confounder in the state space. Next, we provide the main formulation considered in this work – robust state-confounded MDPs, which promote robustness to structured uncertainty.
Structural causal model. We denote a structural causal model (SCM) [17] by a tuple  $\{X, Y, F, P^x\}$ , where  $X$  is the set of exogenous (unobserved) variables,  $Y$  is the set of endogenous (observed) variables, and  $P^x$  is the distribution of all the exogenous variables. Here,  $F$  is the set of structural functions capturing the causal relations between  $X$  and  $Y$  such that for each variable  $y_i \in Y$ ,  $f_i \in F$  is defined as  $y_i \gets f_i(\mathsf{PA}(y_i), x_i)$ , where  $x_i \subseteq X$  and  $\mathsf{PA}(y_i) \subseteq Y \setminus y_i$  denotes the parents of the node  $y_i$ . We say that a pair of variables  $y_i$  and  $y_j$  are confounded by a variable  $C$  (confounder) if they are both caused by  $C$ , i.e.,  $C \in \mathsf{PA}(y_i)$  and  $C \in \mathsf{PA}(y_j)$ . When two variables  $y_i$  and  $y_j$  do not have direct causality, they are still correlated if they are confounded, in which case this correlation is called spurious correlation.
Figure 2: The probabilistic graphs of our formulation (SC-MDP) and other related formulations (specified in Appendix B.1 due to the limited space).  $s_t^1$  means the first dimension of  $s_t$ .  $s_{t'}$  is a shorthand for  $s_{t+1}$ . In SC-MDP, the orange line represents the backdoor path from state  $s_{t'}^1$  to action  $a_{t'}$  opened by the confounder  $c_t$ , which makes the learned policy  $\pi$  rely on the value of  $c_t$ .
3.1 State-confounded MDPs (SC-MDPs)
We now present state-confounded MDPs (SC-MDPs), whose probabilistic graph is illustrated in Figure 2(a) with a comparison to standard MDPs in Figure 2(b). Besides the components in standard MDPs  $\mathcal{M} = \{S, A, T, r\}$ , we introduce a set of unobserved confounder  $C_s = \{c_t\}_{1 \leq t \leq T}$ , where  $c_t \in \mathcal{C}$  denotes the confounder that is generated from some unknown but fixed distribution  $P_t^c$  at time step  $t$ , i.e.,  $c_t \sim P_t^c \in \Delta(\mathcal{C})$ .
To characterize the causal effect of the confounder  $C_{\mathfrak{s}}$  on the state dynamic, we resort to an SCM, where  $C_{\mathfrak{s}}$  is the set of exogenous (unobserved) confounder and endogenous variables include all dimensions of states  $\{s_t^i\}_{1\leq i\leq n,1\leq t\leq T}$ , and actions  $\{a_t\}_{1\leq t\leq T}$ . Specifically, the structural function  $F$  is considered as  $\{\mathcal{P}_t^i\}_{1\leq i\leq n,1\leq t\leq T}$  - the transition from the current state  $s_t$ , action  $a_t$  and the confounder  $c_t$  to each dimension of the next state  $s_{t + 1}^i$  for all time steps, i.e.,  $s_{t + 1}^i\sim \mathcal{P}_t^i (\cdot |s_t,a_t,c_t)$ . Notably, the specified SCM does not confound the reward, i.e.,  $r_t(s_t,a_t)$  does not depend on the confounder  $c_t$ .
Armed with the above SCM, denoting  $P^c \coloneqq \{P_t^c\}$ , we can introduce state-confounded MDPs (SC-MDPs) represented by  $\mathcal{M}_{\mathrm{sc}} = \{\mathcal{S}, \mathcal{A}, T, r, \mathcal{C}, \{\mathcal{P}_t^i\}, P^c\}$  (Figure 2(a)). A policy is denoted as  $\pi = \{\pi_t\}$ , where each  $\pi_t$  results in an intervention (possibly stochastic) that sets  $a_t \sim \pi_t(\cdot | s_t)$  at time step  $t$  regardless of the value of the confounder.
State-confounded value function and optimal policy. Given  $s_t$ , the causal effect of  $a_t$  on the next state  $s_{t+1}$  plays an important role in characterizing value function/Q-function. To ensure the identifiability of the causal effect, the confounder  $c_t$  are assumed to obey the backdoor criterion [17, 18], leading to the following state-confounded value function (SC-value function) and state-confounded Q-function (SC-Q function) [19]:
$$
\widetilde {V} _ {t} ^ {\pi , P ^ {c}} (s) = \mathbb {E} _ {\pi , P ^ {c}} \left[ \sum_ {k = t} ^ {T} r _ {k} \left(s _ {k}, a _ {k}\right) \mid s _ {t} = s; c _ {k} \sim P _ {k} ^ {c}, s _ {k + 1} ^ {i} \sim \mathcal {P} _ {k} ^ {i} (\cdot \mid s _ {k}, a _ {k}, c _ {k}) \right],
$$
$$
\widetilde {Q} _ {t} ^ {\pi , P ^ {c}} (s, a) = \mathbb {E} _ {\pi , P ^ {c}} \left[ \sum_ {k = t} ^ {T} r _ {k} \left(s _ {k}, a _ {k}\right) \mid s _ {t} = s, a _ {t} = a; c _ {k} \sim P _ {k} ^ {c}, s _ {k + 1} ^ {i} \sim \mathcal {P} _ {k} ^ {i} (\cdot \mid s _ {k}, a _ {k}, c _ {k}) \right]. \tag {3}
$$
Remark 1. Note that the proposed SC-MDPs serve as a general formulation for a broad family of RL problems that include standard MDPs as a special case. Specifically, any standard MDP  $\mathcal{M} = \{S, A, P, T, r\}$  can be equivalently represented by at least one SC-MDP  $\mathcal{M}_{\mathrm{sc}} = \{S, A, T, r, C, \{\mathcal{P}_t^i\}, P^c\}$  as long as  $\mathbb{E}_{c_t \sim P_t^c} \left[ \mathcal{P}_t^i(\cdot | s_t, a_t, c_t) \right] = \left[ P(\cdot | s_t, a_t) \right]_i$  for all  $1 \leq i \leq n, 1 \leq t \leq T$ .
3.2 Robust state-confounded MDPs (RSC-MDPs)
In this work, we consider robust state-confounded MDPs (RSC-MDPs) – a variant of SC-MDPs promoting the robustness to the uncertainty of the unobserved confounder distribution  $P^c$ , denoted by  $\mathcal{M}_{\mathrm{sc - rob}} = \{S, A, T, r, C, \{\mathcal{P}_t^i\}, \mathcal{U}^\sigma(P^c)\}$ . Here, the perturbed distribution of the unobserved confounder is assumed in an uncertainty set  $\mathcal{U}^\sigma(P^c)$  centered around the nominal distribution  $P^c$
with radius  $\sigma$  measured by some 'distance' function  $\rho : \Delta(\mathcal{C}) \times \Delta(\mathcal{C}) \to \mathbb{R}^+$ , i.e.,
$$
\mathcal {U} ^ {\sigma} \left(P ^ {c}\right) := \otimes \mathcal {U} ^ {\sigma} \left(P _ {t} ^ {c}\right), \quad \mathcal {U} ^ {\sigma} \left(P _ {t} ^ {c}\right) := \left\{P \in \Delta (\mathcal {C}): \rho \left(P, P _ {t} ^ {c}\right) \leq \sigma \right\}. \tag {4}
$$
Consequently, the corresponding robust SC-value function and robust SC-Q function are defined as
$$
\widetilde {V} _ {t} ^ {\pi , \sigma} (s) := \inf  _ {P \in \mathcal {U} ^ {\sigma} \left(P ^ {c}\right)} \widetilde {V} _ {t} ^ {\pi , P} (s), \quad \widetilde {Q} _ {t} ^ {\pi , \sigma} (s, a) := \inf  _ {P \in \mathcal {U} ^ {\sigma} \left(P ^ {c}\right)} \widetilde {Q} _ {t} ^ {\pi , P} (s, a), \tag {5}
$$
representing the worst-case cumulative rewards when the confounder distribution lies in the uncertainty set  $\mathcal{U}^{\sigma}(P^{c})$ .
Then a natural question is: does there exist an optimal policy that maximizes the robust SC-value function  $\widetilde{V}_t^{\pi, \sigma}$  for any RSC-MDP so that we can target to learn? To answer this, we introduce the following theorem that ensures the existence of the optimal policy for all RSC-MDPs. The proof can be found in Appendix C.1.
Theorem 1 (Existence of an optimal policy). Let  $\Pi$  be the set of all non-stationary and stochastic policies. Consider any RSC-MDP, there exists at least one optimal policy  $\pi^{\mathrm{SC},\star} = \{\pi_t^{\mathrm{SC},\star}\}_{1\leq t\leq T}$  such that for all  $(s,a)\in S\times \mathcal{A}$  and  $1\leq t\leq T$ , one has
$$
\widetilde {V} _ {t} ^ {\pi^ {\mathrm {s c}}, \star , \sigma} (s) = \widetilde {V} _ {t} ^ {\star , \sigma} (s) := \sup  _ {\pi \in \Pi} \widetilde {V} _ {t} ^ {\pi , \sigma} (s) \quad a n d \quad \widetilde {Q} _ {t} ^ {\pi^ {\mathrm {s c}}, \star , \sigma} (s, a) = \widetilde {Q} _ {t} ^ {\star , \sigma} (s, a) := \sup  _ {\pi \in \Pi} \widetilde {Q} _ {t} ^ {\pi , \sigma} (s, a).
$$
In addition, RSC-MDPs also possess benign properties similar to RMDPs such that for any policy  $\pi$  and the robust optimal policy  $\pi^{\mathrm{sc},\star}$ , the corresponding robust SC Bellman consistency equation and robust SC Bellman optimality equation are also satisfied (specified in Appendix C.3.3).
Goal. Based on all the definitions and analysis above, this work aims to find an optimal policy for RSC-MDPs that maximizes the robust SC-value function in (5), yielding optimal performance in the worst case when the unobserved confounder distribution falls into an uncertainty set  $\mathcal{U}^{\sigma}(P^{c})$ .
3.3 Advantages of RSC-MDPs over traditional RMDPs
The most relevant robust RL formulation to ours is RMDPs, which has been introduced in Section 2. Here, we provide a thorough comparison between RMDPs and our RSC-MDPs with theoretical justifications, and leave the comparisons and connections to other related formulations in Figure 2 and Appendix B.1 due to space limits.
To begin, at each time step  $t$ , RMDPs explicitly introduce uncertainty to the transition probability kernels, while our RSC-MDPs add uncertainty to the transition kernels in a latent (and hence more structured) manner via perturbing the unobserved confounder that partly determines the transition kernels. As an example, imagining the true uncertainty set encountered in the real world is illustrated as the blue region in Figure 3, which could have a complicated structure. Since the uncertainty set in RMDPs is homogeneous (illustrated by the green circles), one often faces the dilemma of being either too conservative (when  $\sigma$  is large) or too reckless (when  $\sigma$  is small). In contrast, the proposed
(a) RMDPs
(b) RSC-MDPs
Figure 3: (a) RMDPs add homogeneous noise to states, while (b) RSC-MDPs perturb the confounder to influence states, resulting in a subset of the valid space.
RSC-MDPs - shown in Figure 3(b) - take advantage of the structured uncertainty set (illustrated by the orange region) enabled by the underlying SCM, which can potentially lead to much better estimation of the true uncertainty set. Specifically, the varying unobserved confounder induces diverse perturbation to different portions of the state through the structural causal function, enabling heterogeneous and structural uncertainty sets over the state space.
Theoretical guarantees of RSC-MDPs: advantages of structured uncertainty. To theoretically understand the advantages of the proposed robust formulation of RSC-MDPs with comparison to prior works, especially RMDPs, the following theorem verifies that RSC-MDPs enable additional robustness against semantic attack besides small model perturbation or noise considered in RMDPs. The proof is postponed to Appendix C.2.
Theorem 2. Consider any  $T \geq 2$ . Consider some standard MDPs  $\mathcal{M} = \{S, \mathcal{A}, P^0, T, r\}$ , equivalently represented as an SC-MDP  $\mathcal{M}_{\mathrm{sc}} = \{S, \mathcal{A}, T, r, \mathcal{C}, \{\mathcal{P}_t^i\}, P^c\}$  with  $\mathcal{C} := \{0,1\}$ , and total variation as the 'distance' function  $\rho$  to measure the uncertainty set (the admissible uncertainty level obeys  $\sigma \in [0,1]$ ). For the corresponding RMDP  $\mathcal{M}_{\mathrm{rob}}$  with the uncertainty set  $\mathcal{U}^{\sigma_1}(P^0)$  and the proposed RSC-MDP  $\mathcal{M}_{\mathrm{sc-rob}} = \{S, \mathcal{A}, T, r, \mathcal{C}, \{\mathcal{P}_t^i\}, \mathcal{U}^{\sigma_2}(P^c)\}$ , the optimal robust policy  $\pi_{\mathrm{RMDP}}^{\star,\sigma_1}$  associated with  $\mathcal{M}_{\mathrm{rob}}$  and  $\pi_{\mathrm{RSC}}^{\star,\sigma_2}$  associated with  $\mathcal{M}_{\mathrm{sc-rob}}$  obey: given  $\sigma_2 \in \left(\frac{1}{2}, 1\right]$ , there exist RSC-MDPs with some initial state distribution  $\phi$  such that
$$
\widetilde {V} _ {1} ^ {\pi_ {\mathrm {R S C}} ^ {*}, \sigma_ {2}, \sigma_ {2}} (\phi) - \widetilde {V} _ {1} ^ {\pi_ {\mathrm {R M D P}} ^ {*}, \sigma_ {1}} (\phi) \geq \frac {T}{8}, \quad \forall \sigma_ {1} \in [ 0, 1 ]. \tag {6}
$$