SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Robustness to noisy state: POMDPs and SA-MDPs. State-noisy MDPs refer to the RL problem that the agent can only access and choose the action based on a noisy observation rather than the true state at each step, including two existing types of problems: Partially observable MDPs (POMDPs) and state-adversarial MDPs (SA-MDPs), shown in Figure 3(b). In particular, at each step  $t$ , in POMDPs, the observation  $o_t$  is generated by a fixed probability transition  $\mathcal{O}(\cdot | s_t)$  (we refer to the case that  $o_t$  only depends on the state  $s_t$  but not action); for state-adversarial MDPs, the observation is an adversary  $\nu(s_t)$  against and thus determined by the conducted policy, leading to the worst performance by perturbing the state in a small set around itself. To defend the state perturbation, both POMDPs, and SA-MDPs are indeed robust to the noisy observation, or called agent-observed state, but not the real state that transitions to the environment and next steps. In contrast, our RSC-MDPs
propose the robustness to the real state shift that will directly transition to the next state in the environment, involving additional challenges induced by the appearance of out-of-distribution states.
Robustness to unobserved confounder: MDPUC and confounded MDPs. To address the misleading spurious correlations hidden in components of RL, people formulate RL problems as MDPs with some additional components – unobserved confounders. In particular, the Markov decision process with unobserved confounders (MDPUC) [37] serves as a general framework to concern all types of possible spurious correlations in RL problems – at each step, the state, action, and reward are all possibly influenced by some unobserved confounder, shown in Figure 2(d); confounded MDPs [19] mainly concerns the misleading correlation between the current action and the next state, illustrated in Figure 3(e). The proposed state-confounded MDPs (SC-MDPs) can be seen as a specified type of MDPUC that focuses on breaking the spurious correlation between different parts of the state space itself (different from confounded MDPs which consider the correlation between action and next state), motivated by various real-world applications in self-driving and control tasks. In addition, the proposed formulation is more flexible and can work in both online and offline RL settings.
Contextual MDPs (CDMPs). A contextual MDP (CMDP) [38] is basically a set of standard MDPs sharing the same state and action space but specified by different contexts within a context space. In particular, the transition kernel, reward, and action of a CMDP are all determined by a (possibly unknown) fixed context. The proposed robust state-confounded MDPs (RSC-MDPs) are similar to CMDPs if we cast the unobserved confounder as the context in CMDPs, while different in two aspects: (1) In a CMDP, the context is fixed throughout an episode, while the unobserved confounder in RSC-MDPs can vary as  $\{c_t\}_{1 \leq t \leq T}$ ; (2) In the online setting, the goal of CMDP is to beat the optimal policy depending on the context, while RSC-MDPs seek to learn the optimal policy that does not depend on the confounder  $\{c_t\}_{1 \leq t \leq T}$ .
B.2 Related literature of robustness in RL
Robust RL (robust MDPs). Concerning the robust issues in RL, a large portion of works focus on robust RL with explicit uncertainty of the transition kernel, which is well-posed and a natural way to consider the uncertainty of the environment [13, 39–48]. However, to define the uncertainty set for the environment, most existing works use task structure-agnostic and heuristic 'distance' such as R-contamination, KL divergence,  $\chi^2$ , and total variation [49, 50, 14, 51, 52, 15, 49, 53–58, 49, 59, 48, 60–64] to measure the shift between the training and test transition kernel, leading to a homogeneous (almost structure-free) uncertainty set around the state space. In contrast, we consider a more general uncertainty set that enables the robustness to a task-dependent heterogeneous uncertainty set shaped by unobserved confounder and causal structure, in order to break the spurious correlation hidden in different parts of the state space.
Robustness in RL. Despite the remarkable success that standard RL has achieved, current RL algorithms are still limited since the agent is vulnerable if the deployed environment is subject to uncertainty and even structural changes. To address these challenges, a recent line of RL works begins to concern robustness to the uncertainty or changes over different components of MDPs – state, action, reward, and transition kernel, where a review [8] can be referred to. Besides robust RL framework concerning the shift of the transition kernel and reward, to promote robustness in RL, there exist various works [11, 12] that consider the robustness to action uncertainty, i.e., the deployed action in the environment is distorted by an adversarial agent smoothly or circumstantially; some works [9, 6, 10, 65–67] investigate the robustness to the state uncertainty including but not limited to the introduced POMDPs and SA-MDPs in Appendix B.1, where the agent chooses the action based on observation – the perturbed state determined by some restricted noise or adversarial attack. The proposed RSC-MDPs can be regarded as addressing the state uncertainty since the shift of the unobserved confounder leads to state perturbation. In contrast, RSC-MDPs consider the out-of-distribution of the real state that will directly influence the subsequent transition in the environment, but not the observation in POMDPs and SA-MDPs that will not directly influence the environment.
B.3 Related literature of spurious correlation in RL
Confounder in RL. These works mainly focus on the confounder between action (treatment) and state (effect), which is a long-standing problem that exists in the causal inference area. However,
we find that the confounder may cause problems from another perspective, where the confounder is built upon different dimensions of the state variable. Some people focus on the confounder between action and state, which is common in offline settings since the dataset is fixed and intervention is not allowed. But in the online setting, actions are controlled by an agent and intervention is available to eliminate spurious correlation. [68] reduces the spurious correlation between action and state in the offline setting. [69] deal with environment-irrelevant white noise; possible shift + causal [70]. The confounder problem is usually easy to solve since agents can interact with the environment to do interventions. However, different from most existing settings, we find that even with the capability of intervention, the confounding between dimensions in states cannot be fully eliminated. Then the learned policy is heavily influenced if these confounders change during testing.
Invariant Feature learning. The problem of spurious correlation has attracted attention in the supervised learning area for a long time and many solutions are proposed to learn invariant features to eliminate spurious correlations. A general framework to remedy the ignorance of spurious correlation in empirical risk minimization (ERM) is invariant risk minimization (IRM) [71]. Other works tackle this problem with group distributional robustness [72], adversarial robustness [73], and contrastive learning [74]. These methods are also adapted to sequential settings. The idea of increasing the robustness of RL agents by training agents on multiple environments has been shown in previous works [75, 30, 30]. However, a shared assumption among these methods is that multiple environments with different values of confounder are accessible, which is not always true in the real world.
Counterfactual Data Augmentation in RL. One way to simulate multiple environments is data augmentation. However, most data augmentation works [24, 76, 25, 77-80] apply image transformation to raw inputs, which requires strong domain knowledge for image manipulation and cannot be applied to other types of inputs. In RL, the dynamic model and reward model follow certain causal structures, which allow counterfactual generation of new transitions based on the collected samples. This line of work, named counterfactual data augmentation, is very close to this work. Deep generative models [81] and adversarial examples [82] are considered for the generation to improve sample efficiency in model-based RL. CoDA [83] and MocoDA [33] leverage the concept of locally factored dynamics to randomly stitch components from different trajectories. However, the assumption of local causality may be limited.
Domain Randomization. If we are allowed to control the data generation process, e.g., the underlying mechanism of the simulator, we can apply the golden rule in causality - Randomized Controlled Trial (RCT). The well-known technique, domain randomization [84], exactly follows the idea of RCT, which randomly perturbs the internal state of the experiment in simulators. Later literature follows this direction and develops variants including randomization guided by downstream tasks in the target domain [85, 86], randomization to match real-world distributions [87, 88], and randomization to minimize data divergence [89]. However, it is usually impossible to randomly manipulate internal states in most situations in the real world. In addition, determining which variables to randomize is even harder given so many factors in complex systems.
Discovering Spurious Correlations Detecting spurious correlations helps models remove features that are harmful to generalization. Usually, domain knowledge is required to find such correlations [90-92]. However, when prior knowledge is accessible, techniques such as clustering can also be used to reveal spurious attributes [37, 93, 94]. When human inspection is available, recent works [95-97] also use explainability techniques to find spurious correlations. Another area for discovery is concept-level and interactive debugging [98, 99], which leverage concepts or human feedback to perform debugging.
C Theoretical Analyses
In this section, we verify the existence of an optimal policy of the proposed RSC-MDPs, involving additional components — confounder  $C_{\mathrm{s}}$  and the infimum optimization problems with comparisons to standard MDPs [100].
To begin with, we recall that the goal is to find a policy  $\widetilde{\pi} = \{\widetilde{\pi}_t\}_{1\leq t\leq T}\in \Pi$  such that for all  $(s,a)\in \mathcal{S}\times \mathcal{A}$
$$
\widetilde {V} _ {t} ^ {\widetilde {\pi}, \sigma} (s) = \widetilde {V} _ {t} ^ {\star , \sigma} (s) := \sup  _ {\pi \in \Pi} \widetilde {V} _ {t} ^ {\pi , \sigma} (s) \quad \text {a n d} \quad \widetilde {Q} _ {t} ^ {\widetilde {\pi}, \sigma} (s, a) = \widetilde {Q} _ {t} ^ {\star , \sigma} (s, a) := \sup  _ {\pi \in \Pi} \widetilde {Q} _ {t} ^ {\pi , \sigma} (s, a), \tag {8}
$$
which we called an optimal policy. Towards this, we start from the first claim in (8).
Step 1: Introducing additional notations. Before proceeding, we let  $\{S_t, A_t, R_t, C_t\}$  denote the random variables — state, action, reward, and confounder, at time step  $t$  for all  $1 \leq t \leq T$ . Then invoking the Markov properties, we know that conditioned on current state  $s_t$ , the future state, action, and reward are all independent from the previous  $s_1, a_1, r_1, c_1, \dots, s_{t-1}, a_{t-1}, r_{t-1}, c_{t-1}$ . In addition, we represent  $P_t \in \Delta(\mathcal{C})$  as some distribution of confounder at time step  $t$ , for all  $1 \leq t \leq T$ . For convenience, we introduce the following notation that is defined over time step  $t \leq k \leq T$ :
$$
\forall 1 \leq t \leq T: \quad P _ {+ t} := \otimes_ {t \leq k \leq T} P _ {k} \quad \text {a n d} \quad \mathcal {U} ^ {\sigma} \left(P _ {+ t} ^ {c}\right) := \otimes_ {t \leq k \leq T} \mathcal {U} ^ {\sigma} \left(P _ {k} ^ {c}\right), \tag {9}
$$
which represent some collections of variables from time step  $t$  to the end of the episode. In addition, recall that the transition kernel from time step  $t$  to  $t + 1$  is denoted as  $s_{t + 1}^i\sim \mathcal{P}_t^i (\cdot |s_t,a_t,c_t)$  for  $i\in 1,2\dots ,n$ . With slight abuse of notation, we denote  $s_{t + 1}\sim \mathcal{P}_t(\cdot |s_t,a_t,c_t)$  and abbreviate  $\mathbb{E}_{s_{t + 1}}\sim \mathcal{P}_t(\cdot |s_t,a_t,c_t)[\cdot ]$  as  $\mathbb{E}_{s_{t + 1}}[\cdot ]$  whenever it is clear.
Step 2: Establishing recursive relationship. Recall that the nominal distribution of the confounder is  $c_{t} \in P_{t}^{c}$  at time step  $t$ . We choose  $\widetilde{\pi} = \{\widetilde{\pi}_t\}$  which obeys: for all  $1 \leq t \leq T$ ,
$$
\widetilde {\pi} _ {t} (s) := \underset {\pi_ {t} \in \Delta (\mathcal {A})} {\arg \max } \left\{\mathbb {E} _ {\pi_ {t}} \left[ r _ {t} \left(s, a _ {t}\right) \right] + \inf  _ {P _ {t} \in \mathcal {U} ^ {\sigma} \left(P _ {t} ^ {c}\right)} \mathbb {E} _ {\pi_ {t}} \left[ \mathbb {E} _ {c _ {t} \sim P _ {t}} \left[ \mathbb {E} _ {s _ {t + 1}} \left[ \widetilde {V} _ {t + 1} ^ {\star , \sigma} \left(s _ {t + 1}\right) \right] \right] \right] \right\}. \tag {10}
$$
Armed with these definitions and notations, for any  $(t,s)\in \{1,2,\dots ,T\} \times S$  , one has
$$
\begin{array}{l} \widetilde {V} _ {t} ^ {\star , \sigma} (s) \\ \stackrel {\text {(i)}} {=} \sup  _ {\pi \in \Pi} \inf  _ {P \in \mathcal {U} ^ {\sigma} (P ^ {c})} \widetilde {V} _ {t} ^ {\pi , P} (s) \stackrel {\text {(i i)}} {=} \sup  _ {\pi \in \Pi} \inf  _ {P _ {+ t} \in \mathcal {U} ^ {\sigma} \left(P _ {+ t} ^ {c}\right)} \mathbb {E} _ {\pi , P _ {+ t}} \left[ \sum_ {k = t} ^ {T} r _ {k} \left(s _ {k}, a _ {k}\right) \right] \\ \stackrel {\text {(i i i)}} {=} \sup  _ {\pi \in \Pi} \inf  _ {P _ {+ t} \in \mathcal {U} ^ {\sigma} \left(P _ {+ t} ^ {c}\right)} \mathbb {E} _ {\pi_ {t}} \left[ r _ {t} (s, a _ {t}) \right. \\ \left. + \mathbb {E} _ {c _ {t} \sim P _ {t}} \left[ \mathbb {E} _ {s _ {t + 1}} \left[ \sum_ {k = t + 1} ^ {T} r _ {k} \left(s _ {k}, a _ {k}\right) \mid \pi , P _ {+ (t + 1)}, \left(S _ {t}, A _ {t}, R _ {t}, C _ {t}, S _ {t + 1}\right) = \left(s, a _ {t}, r _ {t}, c _ {t}, s _ {t + 1}\right) \right] \right] \right] \\ = \sup  _ {\pi \in \Pi} \mathbb {E} _ {\pi_ {t}} \left[ r _ {t} (s, a _ {t}) \right] + \inf  _ {P _ {+ t} \in \mathcal {U} ^ {\sigma} \left(P _ {+ t} ^ {c}\right)} \mathbb {E} _ {\pi_ {t}} \left[ \mathbb {E} _ {c _ {t} \sim P _ {t}} \right. \Bigg [ \\ \left. \mathbb {E} _ {s _ {t + 1}} \left[ \sum_ {k = t + 1} ^ {T} r _ {k} \left(s _ {k}, a _ {k}\right) \mid \pi , P _ {+ (t + 1)}, \left(S _ {t}, A _ {t}, R _ {t}, C _ {t}, S _ {t + 1}\right) = \left(s, a _ {t}, r _ {t}, c _ {t}, s _ {t + 1}\right) \right] \right] \\ \end{array}
$$
where (i) holds by the definitions in (5), (ii) is due to (3) and that  $\widetilde{V}_t^{\pi ,P}(s)$  only depends on  $P_{+t}$  by the Markov property, (iii) follows from expressing the term of interest by moving one step ahead and  $\mathbb{E}_{\pi_t}$  is taken with respect to  $a_{t}\sim \pi_{t}(\cdot |S_{t} = s)$ .
To continue, we observe that the  $\widetilde{V}_t^{\star,\sigma}(s)$  can be further controlled as follows:
$$
\begin{array}{l} \widetilde {V} _ {t} ^ {\star , \sigma} (s) \\ = \sup  _ {\pi \in \Pi} \mathbb {E} _ {\pi_ {t}} \left[ r _ {t} \left(s, a _ {t}\right) \right] + \inf  _ {P _ {+ t} \in \mathcal {U} ^ {\sigma} \left(P _ {+ t} ^ {c}\right)} \mathbb {E} _ {\pi_ {t}} \left[ \mathbb {E} _ {c _ {t} \sim P _ {t}} \right. \Bigg [ \\ \left. \mathbb {E} _ {s _ {t + 1}} \left[ \sum_ {k = t + 1} ^ {T} r _ {k} \left(s _ {k}, a _ {k}\right) \mid \pi , P _ {+ (t + 1)}, \left(S _ {t}, A _ {t}, R _ {t}, C _ {t}, S _ {t + 1}\right) = \left(s, a _ {t}, r _ {t}, c _ {t}, s _ {t + 1}\right) \right] \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {\text {(i)}} {=} \sup  _ {\pi \in \Pi} \mathbb {E} _ {\pi_ {t}} \left[ r _ {t} (s, a _ {t}) \right] + \inf  _ {P _ {t} \in \mathcal {U} ^ {\sigma} \left(P _ {t} ^ {c}\right)} \mathbb {E} _ {\pi_ {t}} \left[ \mathbb {E} _ {c _ {t} \sim P _ {t}} \left[ \mathbb {E} _ {s _ {t + 1}} \right. \right. \\ \inf  _ {P _ {+ (t + 1)} \in \mathcal {U} ^ {\sigma} \left(P _ {+ (t + 1)} ^ {c}\right)} \sum_ {k = t + 1} ^ {T} r _ {k} \left(s _ {k}, a _ {k}\right) \mid \pi , P _ {+ (t + 1)}, \left(S _ {t}, A _ {t}, R _ {t}, C _ {t}, S _ {t + 1}\right) = \left. \left(s, a _ {t}, r _ {t}, c _ {t}, s _ {t + 1}\right) \right] \Bigg ] \Bigg ] \Bigg ] \\ \leq \sup  _ {\pi \in \Pi} \mathbb {E} _ {\pi_ {t}} \left[ r _ {t} (s, a _ {t}) \right] + \inf  _ {P _ {t} \in \mathcal {U} ^ {\sigma} \left(P _ {t} ^ {c}\right)} \mathbb {E} _ {\pi_ {t}} \left[ \mathbb {E} _ {c _ {t} \sim P _ {t}} \mathbb {E} _ {s _ {t + 1}} \left[ \sup  _ {\pi^ {\prime} \in \Pi} \inf  _ {P _ {+ (t + 1)} \in \mathcal {U} ^ {\sigma} \left(P _ {+ (t + 1)} ^ {c}\right)} \right. \right. \\ \left. \sum_ {k = t + 1} ^ {T} r _ {k} \left(s _ {k}, a _ {k}\right) \mid \pi^ {\prime}, P _ {+ (t + 1)}, \left(S _ {t}, A _ {t}, R _ {t}, C _ {t}, S _ {t + 1}\right) = \left(s, a _ {t}, r _ {t}, c _ {t}, s _ {t + 1}\right) \right] \Bigg ] \\ \stackrel {\text {(i i)}} {=} \sup  _ {\pi \in \Pi} \mathbb {E} _ {\pi_ {t}} \left[ r _ {t} (s, a _ {t}) \right] \\ \left. \right. + \inf  _ {P _ {t} \in \mathcal {U} ^ {\sigma} \left(P _ {t} ^ {c}\right)} \mathbb {E} _ {\pi_ {t}} \left[ \mathbb {E} _ {c _ {t} \sim P _ {t}} \left[ \mathbb {E} _ {s _ {t + 1}} \left[ \sup  _ {\pi^ {\prime} \in \Pi} \inf  _ {P _ {+ (t + 1)} \in \mathcal {U} ^ {\sigma} \left(P _ {+ (t + 1)} ^ {c}\right)} \mathbb {E} _ {\pi^ {\prime}, P _ {+ (t + 1)}} \left[ \sum_ {k = t + 1} ^ {T} r _ {k} \left(s _ {k}, a _ {k}\right)\right]\right]\right]\right] \\ = \sup  _ {\pi \in \Pi} \left\{\mathbb {E} _ {\pi_ {t}} \left[ r _ {t} \left(s, a _ {t}\right) \right] + \inf  _ {P _ {t} \in \mathcal {U} ^ {\sigma} \left(P _ {t} ^ {c}\right)} \mathbb {E} _ {\pi_ {t}} \left[ \mathbb {E} _ {c _ {t} \sim P _ {t}} \left[ \mathbb {E} _ {s _ {t + 1}} \left[ \widetilde {V} _ {t + 1} ^ {\star , \sigma} \left(s _ {t + 1}\right) \right] \right] \right] \right\} \\ = \sup  _ {\pi_ {t} \in \Delta (\mathcal {A})} \left\{\mathbb {E} _ {\pi_ {t}} \left[ r _ {t} (s, a _ {t}) \right] + \inf  _ {P _ {t} \in \mathcal {U} ^ {\sigma} \left(P _ {t} ^ {c}\right)} \mathbb {E} _ {\pi_ {t}} \left[ \mathbb {E} _ {c _ {t} \sim P _ {t}} \left[ \mathbb {E} _ {s _ {t + 1}} \left[ \widetilde {V} _ {t + 1} ^ {\star , \sigma} \left(s _ {t + 1}\right) \right] \right] \right] \right\} \\ = \inf  _ {P _ {t} \in \mathcal {U} ^ {\sigma} \left(P _ {t} ^ {c}\right)} \mathbb {E} \left[ r _ {t} (s, a _ {t}) + \mathbb {E} _ {c _ {t} \sim P _ {t}} \mathbb {E} _ {s _ {t + 1}} \left[ \left[ \widetilde {V} _ {t + 1} ^ {\star , \sigma} \left(s _ {t + 1}\right) \right] \mid a _ {t} = \widetilde {\pi} _ {t} (s) \right] \right], \tag {11} \\ \end{array}
$$
where (i) holds by the operator  $\inf_{P_{+(t+1)} \in \mathcal{U}^{\sigma}\left(P_{+(t+1)}^{c}\right)}$  is independent from  $\pi_t$  conditioned on a fixed distribution of  $s_{t+1}$ , (ii) arises from the Markov property such that the rewards  $\{r_k(s_k, a_k)\}_{t+1 \leq k \leq T}$  conditioned on  $(S_t, A_t, R_t, C_t, S_{t+1})$  or  $S_{t+1}$  are the same, and the last equality follows from the definition of  $\widetilde{\pi}$  in (10).
Step 3: Completing the proof by applying recursion.
Applying (11) recursively for  $t + 1,\dots T$  , we arrive at
$$
\begin{array}{l} \widetilde {V} _ {t} ^ {\star , \sigma} (s) \leq \inf  _ {P _ {t} \in \mathcal {U} ^ {\sigma} \left(P _ {t} ^ {\mathrm {c}}\right)} \mathbb {E} \left[ r _ {t} (s, a _ {t}) + \mathbb {E} _ {c _ {t} \sim P _ {t}} \mathbb {E} _ {s _ {t + 1}} \left[ \left[ \widetilde {V} _ {t + 1} ^ {\star , \sigma} (s _ {t + 1}) \right] \mid a _ {t} = \widetilde {\pi} _ {t} (s) \right] \right] \\ \leq \inf  _ {P _ {t} \in \mathcal {U} ^ {\sigma} \left(P _ {t} ^ {c}\right)} \inf  _ {P _ {t + 1} \in \mathcal {U} ^ {\sigma} \left(P _ {t + 1} ^ {c}\right)} \mathbb {E} \left[ r _ {t} (s, a _ {t}) + \mathbb {E} _ {c _ {t} \sim P _ {t}} \left[ \mathbb {E} _ {s _ {t + 1}} \right. \right. \\ \left. \left. r _ {t + 1} \left(s _ {t + 1}, a _ {t + 1}\right) + \mathbb {E} _ {c _ {t + 1} \sim P _ {t + 1}} \left[ \mathbb {E} _ {s _ {t + 2}} \left[ \widetilde {V} _ {t + 2} ^ {\star , \sigma} \left(s _ {t + 2}\right) \right] \right] \right] \right] \left(a _ {t}, a _ {t + 1}\right) = \left(\widetilde {\pi} _ {t} (s), \widetilde {\pi} _ {t + 1} \left(s _ {t + 1}\right)\right) \Bigg ] \\ \leq \dots \leq \inf  _ {P _ {+ t} \in \mathcal {U} ^ {\sigma} \left(P _ {+ t} ^ {c}\right)} \mathbb {E} _ {\pi , P _ {+ t}} \left[ \sum_ {k = t} ^ {T} r _ {k} \left(s _ {k}, a _ {k}\right) \right] = \widetilde {V} _ {t} ^ {\widetilde {\pi}, \sigma} (s). \tag {12} \\ \end{array}
$$
Observing from (12) that
$$
\forall s \in \mathcal {S}: \quad \widetilde {V} _ {t} ^ {\star , \sigma} (s) \leq \widetilde {V} _ {t} ^ {\pi , \sigma} (s) \leq \sup  _ {\pi \in \Pi} \widetilde {V} _ {t} ^ {\pi , \sigma} (s) = \widetilde {V} _ {t} ^ {\star , \sigma} (s), \tag {13}
$$
which directly verifies the first assertion in (8)  $\widetilde{V}_t^{\widetilde{\pi},\sigma}(s) = \widetilde{V}_t^{\star,\sigma}(s)$  for all  $s \in S$ . The second assertion in (8) can be achieved analogously. Until now, we verify that there exists at least a policy  $\widetilde{\pi}$  that obeys (8), which we refer to as an optimal policy since its value is equal to or larger than any other non-stationary and stochastic policies over all states  $s \in S$ .
We establish the proof by separating it into several key steps.
Figure 8: The illustration of the transition kernels of the standard MDP  $\mathcal{M}$  and the proposed SC-MDP  $\mathcal{M}_{\mathrm{sc}}$  at the first time step  $t = 1$ , i.e.,  $P_{t}^{0}$  and  $\mathcal{P}_1$  respectively.
Step 1: Constructing a hard instance  $\mathcal{M}$  of standard MDP. In this section, we consider the following standard MDP instance  $\mathcal{M} = \{\mathcal{S},\mathcal{A},P^0,T,r\}$  where  $\mathcal{S} = \{[0,0],[0,1],[1,0],[1,1]\}$  is the state space consisting of four elements in dimension  $n = 2$ , and  $\mathcal{A} = \{0,1\}$  is the action space with only two options. The transition kernel  $P^0 = \{P_t^0\}_{1\leq t\leq T}$  at different time steps  $1\leq t\leq T$  is defined as
$$
P _ {1} ^ {0} \left(s ^ {\prime} \mid s, a\right) = \left\{ \begin{array}{l l} \mathbb {1} \left(s ^ {\prime} = [ 0, 0 ]\right) \mathbb {1} (a = 0) + \mathbb {1} \left(s ^ {\prime} = [ 0, 1 ]\right) \mathbb {1} (a = 1) & \text {i f} s = [ 0, 0 ] \\ \mathbb {1} \left(s ^ {\prime} = s\right) & \text {o t h e r w i s e} \end{array} , \right. \tag {14}
$$
which is illustrated in Fig. 8(a), and
$$
P _ {t} ^ {0} \left(s ^ {\prime} \mid s, a\right) = \mathbb {1} \left(s ^ {\prime} = s\right), \quad \forall (t, s, a) \in \{2, 3, \dots , T \} \times \mathcal {S} \times \mathcal {A}. \tag {15}
$$
Note that this transition kernel  $P^0$  ensures that the next state transitioned from the state  $[0, 0]$  is either  $[0, 0]$  or  $[0, 1]$ . The reward function is specified as follows: for all time steps  $1 \leq t \leq T$ ,
$$
r _ {t} (s, a) = \left\{ \begin{array}{l l} 1 & \text {i f} s = [ 0, 0 ] \text {o r} s = [ 1, 1 ] \\ 0 & \text {o t h e r w i s e} \end{array} \right.. \tag {16}
$$