SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Step 2: The equivalence between  $\mathcal{M}$  and one SC-MDP. Then, we shall show that the constructed standard MDP  $\mathcal{M}$  can be equivalently represented by one SC-MDP  $\mathcal{M}_{\mathrm{sc}} = \{\mathcal{S},\mathcal{A},T,r,\mathcal{C},\{\mathcal{P}_t^i\} ,P^c\}$  with  $\mathcal{C}\coloneqq \{0,1\}$ . The equivalence is defined as the sequential observations  $\{s_t,a_t,r_t\}_{1\leq t\leq T}$  induced by any policy and any initial state distribution in two Markov processes are identical. To specify,  $\mathcal{S},\mathcal{A},T,r$  are kept the same as  $\mathcal{M}$ . Here,  $\{\mathcal{P}_t^i\}$  shall be specified in a while, which determines the transition to each dimension of the next state conditioned on the current state, action, and confounder distribution for all time steps, i.e.,  $s_{t + 1}^{i}\sim \mathbb{E}_{c_{t}\sim P_{t}^{c}}\left[\mathcal{P}_{t}^{i}(\cdot \mid s_{t},a_{t},c_{t})\right]$  for any  $i$ -th dimension of the state ( $i\in \{1,2\}$ ) and all time step  $1\leq t\leq T$ . For convenience, we denote  $\mathcal{P}_t\coloneqq [\mathcal{P}_t^1,\mathcal{P}_t^2 ]\in \Delta (\mathcal{S})$  as the transition kernel towards the next state, namely,  $s_{t + 1}\sim \mathbb{E}_{c_t\sim P_t^c}\left[\mathcal{P}_t(\cdot \mid s_t,a_t,c_t)\right]$ .
Then we simply set the nominal distribution of the confounder as follows:
$$
P _ {t} ^ {c} \left(c _ {t}\right) = \mathbb {1} \left(c _ {t} = 0\right), \quad \forall 1 \leq t \leq T, c _ {t} \in \mathcal {C}. \tag {17}
$$
In addition, before introducing the transition kernel  $\{\mathcal{P}_t^i\}$  of the SC-MDP  $\mathcal{M}_{\mathrm{sc}}$ , we introduce an auxiliary transition kernel  $P^{\mathrm{sc}} = \{P_t^{\mathrm{sc}}\}$  as follows:
$$
P _ {1} ^ {\mathrm {s c}} \left(s ^ {\prime} \mid s, a\right) = \left\{ \begin{array}{l l} \mathbb {1} \left(s ^ {\prime} = [ 1, 0 ]\right) \mathbb {1} (a = 0) + \mathbb {1} \left(s ^ {\prime} = [ 1, 1 ]\right) \mathbb {1} (a = 1) & \text {i f} (s, a) = ([ 0, 0 ], 0) \\ \mathbb {1} \left(s ^ {\prime} = s\right) & \text {o t h e r w i s e} \end{array} , \right. \tag {18}
$$
$$
P _ {t} ^ {\mathrm {s c}} \left(s ^ {\prime} \mid s, a\right) = \mathbb {1} \left(s ^ {\prime} = s\right), \quad \forall (t, s, a) \in \{2, 3, \dots , T \} \times \mathcal {S} \times \mathcal {A}. \tag {19}
$$
It can be observed that  $P^{\mathrm{sc}}$  is similar to  $P^0$  except for the transition in the state  $[0, 0]$ .
Armed with this transition kernel  $P^{\mathrm{sc}}$ , the  $\{\mathcal{P}_t^i\}$  of the SC-MDP  $\mathcal{M}_{\mathrm{sc}}$  is set to obey
$$
\mathcal {P} _ {1} \left(s ^ {\prime} \mid s, a, c _ {1}\right) = \left\{ \begin{array}{l l} \left(1 - c _ {1}\right) P _ {1} ^ {0} \left(s ^ {\prime} \mid s, a\right) + c _ {1} P _ {1} ^ {\mathrm {s c}} \left(s ^ {\prime} \mid s, a\right) & \text {i f} s = [ 0, 0 ] \\ \mathbb {1} \left(s ^ {\prime} = s\right) & \text {o t h e r w i s e} \end{array} , \right. \tag {20}
$$
which is illustrated in Fig. 8(b), and
$$
\mathcal {P} _ {t} \left(s ^ {\prime} \mid s, a, c _ {t}\right) = \mathbb {1} \left(s ^ {\prime} = s\right), \quad \forall (t, s, a, c _ {t}) \in \{2, 3, \dots , T \} \times \mathcal {S} \times \mathcal {A} \times \mathcal {C}. \tag {21}
$$
With them in mind, we are ready to verify that the marginalized transition from the current state and action to the next state in the SC-MDP  $\mathcal{M}_{\mathrm{sc}}$  is identical to the one in MDP  $\mathcal{M}$ : for all  $(t, s_t, a_t, s_{t+1}) \in \{1, 2, \dots, T\} \times \mathcal{S} \times \mathcal{A} \times \mathcal{S}$ :
$$
\mathbb {P} \left(s _ {t + 1} \mid s _ {t}, a _ {t}\right) = \mathbb {E} _ {c _ {t} \sim P _ {c} ^ {c}} \left[ \mathcal {P} _ {t} \left(s _ {t + 1} \mid s _ {t}, a _ {t}, c _ {t}\right) \right] = \mathcal {P} _ {t} \left(s _ {t + 1} \mid s _ {t}, a _ {t}, 0\right) = P ^ {0} \left(s _ {t + 1} \mid s _ {t}, a _ {t}\right) \tag {22}
$$
where the second equality holds by the definition of  $P^c$  in (17), and the last equality holds by the definitions of  $\mathcal{P}$  (see (20) and (21)).
In summary, we verified that the standard MDP  $\mathcal{M} = \{S, \mathcal{A}, P^0, T, r\}$  is equal to the above specified SC-MDP  $\mathcal{M}_{\mathrm{sc}}$ .
Step 3: Defining corresponding RMDP and RSC-MDP. Equipped with the equivalent standard MDP  $\mathcal{M}$  and SC-MDP  $\mathcal{M}_{\mathrm{sc}}$ , we consider the robust variants of them respectively — a RMDP  $\mathcal{M}_{\mathrm{rob}} = \{S, \mathcal{A}, \mathcal{U}^{\sigma_1}(P^0), T, r\}$  with some uncertainty level  $\sigma_1$ , and the proposed RSC-MDP  $\mathcal{M}_{\mathrm{sc - rob}} = \{S, \mathcal{A}, T, r, \mathcal{C}, \{\mathcal{P}_t^i\}, \mathcal{U}^{\sigma_2}(P^c)\}$  with some uncertainty level  $\sigma_2$ .
In this section, without loss of generality, we consider total variation as the 'distance' function  $\rho$  for the uncertainty sets of both RMDP  $\mathcal{M}_{\mathrm{rob}}$  and RSC-MDP  $\mathcal{M}_{\mathrm{sc - rob}}$ , i.e., for any probability vectors  $P^{\prime}, P \in \Delta(\mathcal{C})$  (or  $P^{\prime}, P \in \Delta(S)$ ),  $\rho(P^{\prime}, P) \coloneqq \frac{1}{2} \|P^{\prime} - P\|_{1}$ . Consequently, for any uncertainty level  $\sigma \in [0,1]$ , the uncertainty set  $\mathcal{U}^{\sigma_1}(P^0)$  of the RMDP (see (1)) and  $\mathcal{U}^{\sigma_2}(P^c)$  of the RSC-MDP  $\mathcal{M}_{\mathrm{sc - rob}}$  (see (4)) are defined as follow, respectively:
$$
\mathcal {U} ^ {\sigma} (P ^ {0}) := \otimes \mathcal {U} ^ {\sigma} (P _ {t, s, a} ^ {0}), \qquad \mathcal {U} ^ {\sigma} (P _ {t, s, a} ^ {0}) := \left\{P _ {t, s, a} \in \Delta (\mathcal {S}): \frac {1}{2} \left\| P _ {t, s, a} - P _ {t, s, a} ^ {0} \right\| _ {1} \leq \sigma \right\},
$$
$$
\mathcal {U} ^ {\sigma} \left(P ^ {c}\right) := \otimes \mathcal {U} ^ {\sigma} \left(P _ {t} ^ {c}\right), \quad \mathcal {U} ^ {\sigma} \left(P _ {t} ^ {c}\right) := \left\{P \in \Delta (\mathcal {C}): \frac {1}{2} \| P - P _ {t} ^ {c} \| _ {1} \leq \sigma \right\}. \tag {23}
$$
Step 4: Comparing between the performance of the optimal policy of RMDP  $\mathcal{M}_{\mathrm{rob}}$  ( $\pi_{\mathrm{RMDP}}^{\star, \sigma_1}$ ) and that of RSC-MDP  $\mathcal{M}_{\mathrm{sc-rob}}$  ( $\pi_{\mathrm{RSC}}^{\star, \sigma_2}$ ). To continue, we specify the robust optimal policy  $\pi_{\mathrm{RMDP}}^{\star, \sigma_1}$  associated with  $\mathcal{M}_{\mathrm{rob}}$  and  $\pi_{\mathrm{RSC}}^{\star, \sigma_2}$  associated with  $\mathcal{M}_{\mathrm{sc-rob}}$  and then compare their performance on RSC-MDP with some initial state distribution.
To begin, we introduce the following lemma about the robust optimal policy  $\pi_{\mathrm{RMDP}}^{\star,\sigma_1}$  associated with the RMDP  $\mathcal{M}_{\mathrm{rob}}$ .
Lemma 1. For any  $\sigma_1\in (0,1]$ , the robust optimal policy of  $\mathcal{M}_{\mathrm{rob}}$  obeys
$$
\forall s \in \mathcal {S}: \quad \left[ \pi_ {\mathrm {R M D P}} ^ {\star , \sigma_ {1}} \right] _ {1} (0 | s) = 1. \tag {24a}
$$
In addition, we characterize the robust SC-value functions of the RSC-MDP  $\mathcal{M}_{\mathrm{sc - rob}}$  associated with any policy, combined with the optimal policy and its optimal robust SC-value functions, shown in the following lemma.
Lemma 2. Consider any  $\sigma_{2} \in \left(\frac{1}{2}, 1\right]$  and the RSC-MDP  $\mathcal{M}_{\mathrm{sc - rob}} = \{\mathcal{S}, \mathcal{A}, T, r, \mathcal{C}, \{\mathcal{P}_t^i\}, \mathcal{U}^{\sigma_2}(P^c)\}$ . For any policy  $\pi$ , the corresponding robust SC-value functions satisfy
$$
\widetilde {V} _ {1} ^ {\pi , \sigma_ {2}} ([ 0, 0 ]) = 1 + (T - 1) \inf  _ {P \in \mathcal {U} ^ {\sigma} \left(P _ {1} ^ {c}\right)} \mathbb {E} _ {c _ {1} \sim P} \left[ \pi_ {1} (0 | [ 0, 0 ]) \left(1 - c _ {1}\right) + \pi_ {1} (1 | [ 0, 0 ]) c _ {1} \right]. \tag {25a}
$$
In addition, the optimal robust SC-value function and the robust optimal policy  $\pi_{\mathsf{RSC}}^{\star,\sigma_2}$  of the RMDP  $\mathcal{M}_{\mathrm{sc - rob}}$  obeys:
$$
\widetilde {V} _ {1} ^ {\pi_ {\mathrm {R S C}} ^ {\star , \sigma^ {2}}, \sigma_ {2}} ([ 0, 0 ]) = \widetilde {V} _ {1} ^ {\star , \sigma_ {2}} ([ 0, 0 ]) = 1 + \frac {T - 1}{2}. \tag {26}
$$
Armed with above lemmas, applying Lemma 2 with policy  $\pi = \pi_{\mathsf{RMDP}}^{\star ,\sigma_1}$  obeying  $\left[\pi_{\mathsf{RMDP}}^{\star ,\sigma_1}\right]_1(0|s) = 1$  in Lemma 1, one has
$$
\begin{array}{l} \widetilde {V} _ {1} ^ {\pi_ {\mathrm {R M D P}} ^ {\star , \sigma_ {1}}, \sigma_ {2}} ([ 0, 0 ]) = 1 + (T - 1) \inf  _ {P \in \mathcal {U} ^ {\sigma} \left(P _ {1} ^ {\mathrm {c}}\right)} \mathbb {E} _ {c _ {1} \sim P} \left[ 1 - c _ {1} \right] \\ \leq 1 + (T - 1) \left[ \frac {1}{4} \cdot 1 + \frac {3}{4} \cdot 0 \right] = 1 + \frac {T - 1}{4}, \tag {27} \\ \end{array}
$$
where the inequality holds by the fact that the probability distribution  $P$  obeying  $P_{1}(0) = \frac{1}{4}$  and  $P_{1}(1) = \frac{3}{4}$  is inside the uncertainty set  $\mathcal{U}^{\sigma_2}(P_1^c)$  (recall that  $\sigma_{2} \in (\frac{1}{2}, 1]$  and  $P_{1}^{c}(0) = 1$ ).
Finally, combining (27) and (26) together, we complete the proof by showing that with the initial state distribution  $\phi$  defined as  $\phi ([0,0]) = 1$ , we arrive at
$$
\widetilde {V} _ {1} ^ {\pi_ {\mathrm {R S C}} ^ {\star , \sigma_ {2}}, \sigma_ {2}} (\phi) - \widetilde {V} _ {1} ^ {\pi_ {\mathrm {R M D P}} ^ {\star , \sigma_ {1}}, \sigma_ {2}} ([ 0, 0 ]) = \widetilde {V} _ {1} ^ {\star , \sigma_ {2}} (\phi) - \widetilde {V} _ {1} ^ {\pi_ {\mathrm {R M D P}} ^ {\star , \sigma_ {1}}, \sigma_ {2}} ([ 0, 0 ]) \geq \frac {T - 1}{4} \geq \frac {T}{8}, \tag {28}
$$
where the last inequality holds by  $T \geq 2$ .
C.3 Proof of auxiliary results
C.3.1 Proof of Lemma 1
Step 1: specifying the minimum of the robust value functions over states. For any uncertainty set  $\sigma_{1} \in (0,1]$ , we first characterize the robust value function of any policy  $\pi$  over different states. To start, we denote the minimum of the robust value function over states at each time step  $t$  as below:
$$
V _ {\min , t} ^ {\pi , \sigma_ {1}} := \min  _ {s \in \mathcal {S}} V _ {t} ^ {\pi , \sigma_ {1}} (s) \geq 0, \tag {29}
$$
where the last inequality holds that the reward function defined in (16) is always non-negative. Obviously, there exists at least one state  $s_{\min,t}^{\pi}$  that satisfies  $V_t^{\pi,\sigma_1}(s_{\min,t}^{\pi}) = V_{\min,t}^{\pi,\sigma_1}$ .
With this in mind, we shall verify that for any policy  $\pi$
$$
\forall 1 \leq t \leq T: \quad V _ {t} ^ {\pi , \sigma_ {1}} ([ 0, 1 ]) = V _ {t} ^ {\pi , \sigma_ {1}} ([ 1, 0 ]) = 0. \tag {30}
$$
To achieve this, we use a recursive argument. First, the base case can be verified since when  $t + 1 = T + 1$ , the value functions are all zeros at  $T + 1$  step, i.e.,  $V_{T + 1}^{\pi ,\sigma_1}(s) = 0$  for all  $s\in S$ . Then, the goal is to verify the following fact
$$
V _ {t} ^ {\pi , \sigma_ {1}} ([ 0, 1 ]) = V _ {t} ^ {\pi , \sigma_ {1}} ([ 1, 0 ]) = 0 \tag {31}
$$
with the assumption that  $V_{t+1}^{\pi, \sigma_1}(s) = 0$  for any state  $s = \{[0,1], [1,0]\}$ . It is easily observed that for any policy  $\pi$ , the robust value function when state  $s = \{[0,1], [1,0]\}$  at any time step  $t$  obeys
$$
\begin{array}{l} 0 \leq V _ {t} ^ {\pi , \sigma_ {1}} (s) = \mathbb {E} _ {a \sim \pi_ {t} (\cdot | s)} \left[ r _ {t} (s, a) + \inf  _ {P \in \mathcal {U} ^ {\sigma_ {1}} \left(P _ {t, s, a} ^ {0}\right)} P V _ {t + 1} ^ {\pi , \sigma_ {1}} \right] \\ \stackrel {\text {(i)}} {=} 0 + (1 - \sigma_ {1}) V _ {t + 1} ^ {\pi , \sigma_ {1}} (s) + \sigma_ {1} V _ {\min , t + 1} ^ {\pi , \sigma_ {1}} \stackrel {\text {(i i)}} {=} 0 + \sigma_ {1} V _ {\min , t + 1} ^ {\pi , \sigma_ {1}} \\ \leq 0 + \sigma_ {1} V _ {t + 1} ^ {\pi , \sigma_ {1}} (s) = 0 \tag {32} \\ \end{array}
$$
where (i) holds by  $r_t(s, a) = 0$  for all  $s = \{[0,1], [1,0]\}$ , the fact  $P_t^0(s \mid s, a) = 1$  for  $s \in S$  (see (14) and (15)), and the definition of the uncertainty set  $\mathcal{U}^{\sigma_1}(P^0)$  in (23). Here (ii) follows from the recursive assumption  $V_{t+1}^{\pi, \sigma_1}(s) = 0$  for any state  $s = \{[0,1], [1,0]\}$ , and the last equality holds by  $V_{\min, t+1}^{\pi, \sigma_1} \leq V_{t+1}^{\pi, \sigma_1}([0,1])$  (see (29)). Until now, we complete the proof for (31) and then verify (30).
Note that (30) directly leads to
$$
\forall 1 \leq t \leq T: \quad V _ {\min , t} ^ {\pi , \sigma_ {1}} = 0. \tag {33}
$$
Step 2: Considering the robust value function at state  $[0,0]$ . Armed with the above facts, we are now ready to derive the robust value function for the state  $[0,0]$ .
When  $2 \leq t \leq T$ , one has
$$
\begin{array}{l} V _ {t} ^ {\pi , \sigma_ {1}} ([ 0, 0 ]) = \mathbb {E} _ {a \sim \pi_ {t} (\cdot | [ 0, 0 ])} \left[ r _ {t} ([ 0, 0 ], a) + \inf  _ {P \in \mathcal {U} ^ {\sigma_ {1}} (P _ {t, [ 0, 0 ], a})} P V _ {t + 1} ^ {\pi , \sigma_ {1}} \right] \\ \stackrel {\text {(i)}} {=} 1 + \left[ (1 - \sigma_ {1}) V _ {t + 1} ^ {\pi , \sigma_ {1}} ([ 0, 0 ]) + \sigma_ {1} V _ {\min , t + 1} ^ {\pi , \sigma_ {1}} \right] \\ = 1 + \left(1 - \sigma_ {1}\right) V _ {t + 1} ^ {\pi , \sigma_ {1}} ([ 0, 0 ]) \tag {34} \\ \end{array}
$$
where (i) holds by  $r_t([0,0],a) = 1$  for all  $a\in \{0,1\}$  and the definition of  $P^0$  (see (15)), and the last equality arises from (33).
Applying (34) recursively for  $t, t + 1, \dots, T$  yields that
$$
V _ {t} ^ {\pi , \sigma_ {1}} ([ 0, 0 ]) = \sum_ {k = t} ^ {T} (1 - \sigma_ {1}) ^ {k - t} \geq 1. \tag {35}
$$
When  $t = 1$ , the robust value function obeys:
$$
\begin{array}{l} V _ {1} ^ {\pi , \sigma_ {1}} ([ 0, 0 ]) = \mathbb {E} _ {a \sim \pi_ {1} (\cdot | [ 0, 0 ])} \left[ r _ {1} ([ 0, 0 ], a) + \inf  _ {P \in \mathcal {U} ^ {\sigma_ {1}} (P _ {1, [ 0, 0 ], a})} P V _ {2} ^ {\pi , \sigma_ {1}} \right] \\ \stackrel {\text {(i)}} {=} 1 + \pi_ {1} (0 | [ 0, 0 ]) \inf  _ {P \in \mathcal {U} ^ {\sigma_ {1}} (P _ {1, [ 0, 0 ], 0})} P V _ {2} ^ {\pi , \sigma_ {1}} + \pi_ {1} (1 | [ 0, 0 ]) \inf  _ {P \in \mathcal {U} ^ {\sigma_ {1}} (P _ {1, [ 0, 0 ], 1})} P V _ {2} ^ {\pi , \sigma_ {1}} \\ \stackrel {\text {(i i)}} {=} 1 + \pi_ {1} (0 | [ 0, 0 ]) \left[ (1 - \sigma_ {1}) V _ {2} ^ {\pi , \sigma_ {1}} ([ 0, 0 ]) + \sigma_ {1} V _ {\min , 2} ^ {\pi , \sigma_ {1}} \right] \\ + \pi_ {1} (1 | [ 0, 0 ]) \left[ (1 - \sigma_ {1}) V _ {2} ^ {\pi , \sigma_ {1}} ([ 0, 1 ]) + \sigma_ {1} V _ {\min , 2} ^ {\pi , \sigma_ {1}} \right] \\ = 1 + \pi_ {1} (0 \mid [ 0, 0 ]) (1 - \sigma_ {1}) V _ {2} ^ {\pi , \sigma_ {1}} ([ 0, 0 ]) \tag {36} \\ \end{array}
$$
where (i) holds by  $r_1([0,0],a) = 1$  for all  $a\in \{0,1\}$ , (ii) follows from the definition of  $P^0$  (see (14)), and the last equality arises from (30) and (33).
Step 3: the optimal policy  $\pi_{\mathsf{RMDP}}^{\star, \sigma_1}$ . Observing that  $V_1^{\pi, \sigma_1}([0,0])$  is increasing monotonically as  $\pi_1(0 \mid [0,0])$  is larger, we directly have that  $\pi_{\mathsf{RMDP}}^{\star, \sigma_1}(0 \mid [0,0]) = 1$ .
Considering that the action does not influence the state transition for  $t = 2,3,\dots ,T$  and all other states  $s\neq [0,0]$ , without loss of generality, we choose the robust optimal policy as
$$
\forall s \in \mathcal {S}: \left[ \pi_ {\mathrm {R M D P}} ^ {\star , \sigma_ {1}} \right] _ {1} (0 | s) = 1. \tag {37}
$$
C.3.2 Proof of Lemma 2
To begin with, for any uncertainty level  $\sigma_{2} \in \left(\frac{1}{2}, 1\right]$  and any policy  $\pi = \{\pi_t\}$ , we consider the robust SC-value function  $\widetilde{V}_t^{\pi, \sigma_2}$  of the RSC-MDP  $\mathcal{M}_{\mathrm{sc - rob}}$ .
Step 1: deriving  $\widetilde{V}_t^{\pi, \sigma_2}$  for  $2 \leq t \leq T$ . Towards this, for any  $2 \leq t \leq T$  and  $s \in S$ , one has
$$
\begin{array}{l} \widetilde {V} _ {t} ^ {\pi , \sigma_ {2}} (s) = \mathbb {E} _ {a \sim \pi_ {t} (s)} \left[ \widetilde {Q} _ {t} ^ {\pi , \sigma_ {2}} (s, a) \right] \\ \stackrel {\mathrm {(i)}} {=} \mathbb {E} _ {a \sim \pi_ {t} (s)} \left[ r _ {t} (s, a) + \inf  _ {P \in \mathcal {U} ^ {\sigma} \left(P _ {t} ^ {c}\right)} \mathbb {E} _ {c _ {t} \sim P} \left[ \mathcal {P} _ {t, s, a, c _ {t}} \widetilde {V} _ {t + 1} ^ {\pi , \sigma_ {2}} \right] \right] \\ \stackrel {\mathrm {(i i)}} {=} r _ {t} (s, a) + \inf  _ {P \in \mathcal {U} ^ {\sigma} \left(P _ {t} ^ {c}\right)} \mathbb {E} _ {c _ {t} \sim P} \left[ \mathcal {P} _ {t, s, a, c _ {t}} \widetilde {V} _ {t + 1} ^ {\pi , \sigma_ {2}} \right] \\ = r _ {t} (s, a) + \widetilde {V} _ {t + 1} ^ {\pi , \sigma} (s), \tag {38} \\ \end{array}
$$
where (i) follows from the state-confounded Bellman consistency equation in (47), (ii) holds by that the reward function  $r_t$  and  $\mathcal{P}_t$  are all independent from the action (see (16) and (21)), and the last inequality holds by  $\mathcal{P}_t(s' | s, a, c_t) = \mathbb{1}(s' = s)$  is independent from  $c_t$  (see (21)).
Applying the above fact recursively for  $t, t + 1, \dots, T$  leads to that for any  $s \in S$ ,
$$
\widetilde {V} _ {t} ^ {\pi , \sigma_ {2}} (s) = r _ {t} (s, a _ {t}) + \widetilde {V} _ {t + 1} ^ {\pi , \sigma} (s) = r _ {t} (s, a) + r _ {t + 1} (s, a _ {t + 1}) + \widetilde {V} _ {t + 2} ^ {\pi , \sigma} (s)
$$
$$
= \dots = r _ {t} \left(s, a _ {t}\right) + \sum_ {k = t + 1} ^ {T} r _ {k} \left(s _ {k}, a _ {k}\right), \tag {39}
$$
which directly yields (see reward  $r$  in (16))
$$
\widetilde {V} _ {2} ^ {\pi , \sigma_ {2}} ([ 0, 0 ]) = \widetilde {V} _ {2} ^ {\pi , \sigma_ {2}} ([ 1, 1 ]) = T - 1 \quad \text {a n d} \quad \widetilde {V} _ {2} ^ {\pi , \sigma_ {2}} ([ 0, 1 ]) = \widetilde {V} _ {2} ^ {\pi , \sigma_ {2}} ([ 1, 0 ]) = 0. \tag {40}
$$
Step 2: characterizing  $\widetilde{V}_1^{\pi, \sigma_2}([0,0])$  for any policy  $\pi$ . In this section, we consider the value of  $\widetilde{V}_1^{\pi, \sigma_2}$  on the state  $[0,0]$ . To proceed, one has
$$
\begin{array}{l} \widetilde {V} _ {1} ^ {\pi , \sigma_ {2}} ([ 0, 0 ]) = \mathbb {E} _ {a \sim \pi_ {1} ([ 0, 0 ])} \left[ \widetilde {Q} _ {1} ^ {\pi , \sigma_ {2}} ([ 0, 0 ], a) \right] \\ \stackrel {\mathrm {(i)}} {=} \mathbb {E} _ {a \sim \pi_ {1} ([ 0, 0 ])} \left[ r _ {1} ([ 0, 0 ], a) + \inf  _ {P \in \mathcal {U} ^ {\sigma} \left(P _ {1} ^ {c}\right)} \mathbb {E} _ {c _ {1} \sim P} \left[ \mathcal {P} _ {1, [ 0, 0 ], a, c _ {1}} \widetilde {V} _ {2} ^ {\pi , \sigma_ {2}} \right] \right] \\ \stackrel {\mathrm {(i i)}} {=} 1 + \inf  _ {P \in \mathcal {U} ^ {\sigma} \left(P _ {1} ^ {c}\right)} \mathbb {E} _ {c _ {1} \sim P} \left[ \left(\pi_ {1} (0 | [ 0, 0 ]) \mathcal {P} _ {1, [ 0, 0 ], 0, c _ {1}} + \pi_ {t} (1 | [ 0, 0 ]) \mathcal {P} _ {1, [ 0, 0 ], 1, c _ {1}}\right) \widetilde {V} _ {2} ^ {\pi , \sigma} \right] \\ \stackrel {\mathrm {(i i i)}} {=} 1 + \inf  _ {P \in \mathcal {U} ^ {\sigma} (P _ {1} ^ {c})} \mathbb {E} _ {c _ {1} \sim P} \left[ \pi_ {1} (0 | [ 0, 0 ]) \left((1 - c _ {1}) P _ {1, [ 0, 0 ], 0} ^ {0} + c _ {1} P _ {1, [ 0, 0 ], 0} ^ {\mathrm {s c}}\right) \widetilde {V} _ {2} ^ {\pi , \sigma} \right. \\ \left. + \pi_ {1} (1 \mid [ 0, 0 ]) \left((1 - c _ {1}) P _ {1, [ 0, 0 ], 1} ^ {0} + c _ {1} P _ {1, [ 0, 0 ], 1} ^ {\mathrm {s c}}\right) \widetilde {V} _ {2} ^ {\pi , \sigma} \right] \\ \stackrel {\text {(i v)}} {=} 1 + \inf  _ {P \in \mathcal {U} ^ {\sigma} \left(P _ {1} ^ {\mathrm {c}}\right)} \mathbb {E} _ {c _ {1} \sim P} \left[ \pi_ {1} (0 \mid [ 0, 0 ]) \left((1 - c _ {1}) \widetilde {V} _ {2} ^ {\pi , \sigma} ([ 0, 0 ]) + c _ {1} \widetilde {V} _ {2} ^ {\pi , \sigma} ([ 1, 0 ])\right) \right. \\ \left. + \pi_ {1} (1 \mid [ 0, 0 ]) \left((1 - c _ {1}) \widetilde {V} _ {2} ^ {\pi , \sigma} ([ 0, 1 ]) + c _ {1} \widetilde {V} _ {2} ^ {\pi , \sigma} ([ 1, 1 ])\right) \right] \\ = 1 + (T - 1) \inf  _ {P \in \mathcal {U} ^ {\sigma} \left(P _ {1} ^ {c}\right)} \mathbb {E} _ {c _ {1} \sim P} \left[ \pi_ {1} (0 | [ 0, 0 ]) (1 - c _ {1}) + \pi_ {1} (1 | [ 0, 0 ]) c _ {1} \right] \\ = 1 + (T - 1) \pi_ {1} (0 | [ 0, 0 ]) + (T - 1) \inf  _ {P \in \mathcal {U} ^ {\sigma} \left(P _ {1} ^ {c}\right)} \mathbb {E} _ {c _ {1} \sim P} \left[ c _ {1} \left(1 - 2 \pi_ {1} (0 | [ 0, 0 ])\right) \right], \tag {41} \\ \end{array}
$$
where (i) holds by robust state-confounded Bellman consistency equation in (47), (ii) follows from  $r_1([0,0],a) = 1$  for all  $a\in \{0,1\}$  which is independent from  $c_{t}$ . (iii) arises from the definition of  $\mathcal{P}$  in (20), (iv) can be verified by plugging in the definitions from (14) and (18), and the penultimate equality holds by (40).
Step 3: characterizing the optimal robust SC-value functions. Before proceeding, we recall the fact that  $\mathcal{U}^{\sigma}(P_1^c) = \left\{P\in \Delta (\mathcal{C}):\frac{1}{2}\| P - P_1^c\| _1\leq \sigma_2\right\}$ .
Observing from (41) that for any fixed  $\pi_1(0\mid [0,0])$ ,  $c_{1}\left(1 - 2\pi_{1}(0\mid [0,0])\right)$  is monotonously increasing with  $c_{1}$  when  $1 - 2\pi_{1}(0\mid [0,0])\geq 0$  and decreasing with  $c_{1}$  otherwise, it is easily verified that the maximum of the following function
$$
f \left(\pi_ {1} (0 \mid [ 0, 0 ])\right) := (T - 1) \inf  _ {P \in \mathcal {U} ^ {\sigma} \left(P _ {1} ^ {c}\right)} \mathbb {E} _ {c _ {1} \sim P} \left[ c _ {1} \left(1 - 2 \pi_ {1} (0 \mid [ 0, 0 ])\right) \right] \tag {42}
$$
obeys
$$
\max  f \left(\pi_ {1} (0 \mid [ 0, 0 ])\right) = \left\{ \begin{array}{l l} 0 & \text {i f} \pi_ {1} (0 \mid [ 0, 0 ]) \geq \frac {1}{2} \\ (T - 1) \sigma_ {2} \left(1 - 2 \pi_ {1} (0 \mid [ 0, 0 ])\right) & \text {o t h e r w i s e} \end{array} . \right. \tag {43}
$$
Then, note that the value of  $\widetilde{V}_1^{\pi,\sigma_2}([0,0])$  only depends on  $\pi_1(\cdot|[0,0])$  which can be represented by  $\pi_1(0|[0,0])$ . Plugging in (43) into (41) arrives at when  $\pi_1(0|[0,0]) \geq \frac{1}{2}$ ,
$$
\max  _ {\pi_ {1} (0 \mid [ 0, 0 ]) \geq \frac {1}{2}} \widetilde {V} _ {1} ^ {\pi , \sigma_ {2}} ([ 0, 0 ])
$$
$$
\begin{array}{l} = \max  _ {\pi_ {1} (0 | [ 0, 0 ]) \geq \frac {1}{2}} 1 + (T - 1) \pi_ {1} (0 | [ 0, 0 ]) + (T - 1) \sigma_ {2} \left(1 - 2 \pi_ {1} (0 | [ 0, 0 ])\right) \\ = 1 + (T - 1)\sigma_{2} + (T - 1)\max_{\pi_{1}(0\mid [0,0])\geq \frac{1}{2}}(1 - 2\sigma_{2})\pi_{1}(0\mid [0,0]) \\ = 1 + (T - 1) \sigma_ {2} + \frac {(T - 1) (1 - 2 \sigma_ {2})}{2} = 1 + \frac {T - 1}{2}, \tag {44} \\ \end{array}
$$
where the penultimate equality holds by  $\sigma_2 > \frac{1}{2}$  and letting  $\pi_1(0\mid [0,0]) = \frac{1}{2}$ . Similarly, when  $\pi_1(0\mid [0,0]) < \frac{1}{2}$ ,
$$
\max  _ {\pi_ {1} (0 \mid [ 0, 0 ]) <   \frac {1}{2}} \widetilde {V} _ {1} ^ {\pi , \sigma_ {2}} ([ 0, 0 ]) = \max  _ {\pi_ {1} (0 \mid [ 0, 0 ]) <   \frac {1}{2}} 1 + (T - 1) \pi_ {1} (0 \mid [ 0, 0 ]) <   1 + \frac {T - 1}{2}. \tag {45}
$$
Consequently, combining (44) and (45), we conclude that
$$
\widetilde {V} _ {1} ^ {\pi_ {\mathrm {R S C}} ^ {\star , \sigma_ {2}}, \sigma_ {2}} ([ 0, 0 ]) = \widetilde {V} _ {1} ^ {\star , \sigma_ {2}} ([ 0, 0 ]) = \max  _ {\pi} \widetilde {V} _ {1} ^ {\pi , \sigma_ {2}} ([ 0, 0 ]) = 1 + \frac {T - 1}{2}. \tag {46}
$$
C.3.3 Auxiliary results of RSC-MDPs
It is easily verified that for any RSC-MDP  $\mathcal{M}_{\mathrm{sc - rob}} = \{\mathcal{S},\mathcal{A},T,r,\mathcal{C},\{\mathcal{P}_t^i\} ,\mathcal{U}^{\sigma_2}(P^c)\}$ , any policy  $\pi$  and optimal policy  $\pi^{\star}$  satisfy the corresponding robust state-confounded Bellman consistency equation and Bellman optimality equation shown below, respectively:
$$
\widetilde {Q} _ {t} ^ {\pi , \sigma} (s, a) = r _ {t} (s, a) + \inf  _ {P \in \mathcal {U} ^ {\sigma} (P _ {t} ^ {c})} \mathbb {E} _ {c _ {t} \sim P} \left[ \mathcal {P} _ {t, s, a, c _ {t}} \widetilde {V} _ {t + 1} ^ {\pi , \sigma} \right],
$$
$$
\widetilde {Q} _ {t} ^ {\star , \sigma} (s, a) = r _ {t} (s, a) + \inf  _ {P \in \mathcal {U} ^ {\sigma} \left(P _ {t} ^ {c}\right)} \mathbb {E} _ {c _ {t} \sim P} \left[ \mathcal {P} _ {t, s, a, c _ {t}} \widetilde {V} _ {t + 1} ^ {\star , \sigma} \right], \tag {47}
$$