SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Step 1: We first show that if a state dimension  $s_{i,t}$  is in  $s_t^r$ , then  $s_{i,t} \nsubseteq R_t | a_{t-1:t}, \mathbf{s}_{t-1}$ .
We prove it by contradiction. Suppose that  $s_{i,t}$  is independent of  $R_{t}$  given  $a_{t-1:t}$  and  $\mathbf{s}_{t-1}$ . Then according to the faithfulness assumption, we can see from the graph that  $s_{i,t}$  does not have a directed path to  $r_{t+\tau}$ , which contradicts the assumption, because, otherwise,  $a_{t-1:t}$  and  $\mathbf{s}_{t-1}$  cannot break the paths between  $s_{i,t}$  and  $R_{t}$  which leads to the dependence.
We next show that if  $s_{i,t} \not\perp R_t | a_{t-1:t}, \mathbf{s}_{t-1}$ , then  $s_{i,t} \in s_t^r$ .
Similarly, by contradiction suppose that  $s_{i,t}$  does not have a directed path to  $r_{t + \tau}$ . From the graph, it is easy to see that  $a_{t - 1:t}$  and  $\mathbf{s}_{t - 1}$  must d-separate the path between  $s_{i,t}$  and  $R_{t}$ . According to the Markov assumption,  $s_{i,t}$  is independent of  $R_{t}$  given  $a_{t - 1:t}$  and  $\mathbf{s}_{t - 1}$ , which contradicts to the assumption. Since we have a contradiction, it must be that  $s_{i,t}$  has a directed path to  $r_{t + \tau}$ , i.e.  $s_{i,t} \in s_t^r$ .
Step 2: In step 1, we have shown that  $s_{i,t} \not\perp R_t | a_{t-1:t}, \mathbf{s}_{t-1}$ , if and only if it has a directed path to  $r_{t+\tau}$ . From the graph, it is easy to see that for those state dimensions which have a directed path to  $r_{t+\tau}$ ,  $a_{t-1:t}$  and  $\mathbf{s}_{t-1}$  cannot break the path between  $s_{i,t}$  and  $R_t$ . Moreover, for those state dimensions which do not have a directed path to  $r_{t+\tau}$ ,  $a_{t-1:t}$  and  $s_{t-1}^r$  are enough to break the path between  $s_{i,t}$  and  $R_t$ .
Therefore, for  $s_{i,t}, s_{i,t} \not\downarrow R_t | a_{t-1:t}, s_{t-1}$ , if and only if  $s_{i,t} \not\downarrow R_{t+1} | a_{t-1:t}, s_{t-1}^r$ .
Step 3: In the previous steps, it has been shown that if a state dimension  $s_{i,t}$  is in  $s_t^r$ , then  $s_{i,t} \not\perp R_t|a_{t-1:t}, s_{t-1}^r$ , and if a state dimension  $s_{i,t}$  is not in  $s_t^r$ , then  $s_{i,t} \not\perp R_t|a_{t-1:t}, s_{t-1}^r$ . This implies that  $s_t^r$  are minimally sufficient for policy learning to maximize the future reward.
Moreover, the proposition below shows that  $s_t^a$ , which receives an edge from  $a_{t-1}$ , can be directly controlled by actions and can be characterized by conditional dependence with the action variable.
Proposition 2. Under the assumption that the graphical representation, corresponding to the environment model, is Markov and faithful to the measured data,  $s_t^a \subseteq \mathbf{s_t}$  is a minimal subset of state dimensions that are sufficient for direct control, and  $s_{i,t} \in s_t^a$  if and only if  $s_{i,t} \not\perp a_{t-1} | \mathbf{s}_{t-1}$ .
Below, we give the proof of Proposition 2.
Proof. The proof contains the following two steps.
- In step 1, we show that a state dimension  $s_{i,t}$  is in  $s_t^a$ , that is, it receives an edge from  $a_{t - 1}$ , if and only if  $s_{i,t} \not\perp a_{t - 1}|s_{t - 1}$ .
- In step 2, we show that  $s_t^a$  contains a minimally sufficient subset of state dimensions that can be directly controlled by actions.
Step 1: We first show that if a state dimension  $s_{i,t}$  is in  $s_t^a$ , then  $s_{i,t} \nleq a_{t-1}|s_{t-1}$ .
We prove it by contradiction. Suppose that  $s_{i,t}$  is independent of  $a_{t-1}$  given  $\mathbf{s}_{t-1}$ . Then according to the faithfulness assumption, we can see from the graph that  $s_{i,t}$  does not receive an edge from  $a_{t-1}$ , which contradicts the assumption, because, otherwise,  $\mathbf{s}_{t-1}$  cannot break the paths between  $s_{i,t}$  and  $a_{t-1}$  which leads to the dependence.
We next show that if  $s_{i,t} \not\downarrow a_{t-1} | \mathbf{s}_{t-1}$ , then  $s_{i,t} \in s_t^a$ .
Similarly, by contradiction suppose that  $s_{i,t}$  does not receive an edge from  $a_{t-1}$ . From the graph, it is easy to see that  $\mathbf{s}_{t-1}$  must break the path between  $s_{i,t}$  and  $a_{t-1}$ . According to the Markov assumption,  $s_{i,t}$  is independent of  $a_{t-1}$  given  $\mathbf{s}_{t-1}$ , which contradicts the assumption. Since we have a contradiction, it must be that  $s_{i,t}$  has an edge from  $a_{t-1}$ .
Step 2: In the previous steps, it has been shown that if a state dimension  $s_{i,t}$  is in  $s_t^a$ , then  $s_{i,t} \not\perp a_{t-1} | \mathbf{s}_{\mathbf{t}-1}$ , and if a state dimension  $s_{i,t}$  is not in  $s_t^a$ , then  $s_{i,t} \not\perp a_{t-1} | \mathbf{s}_{\mathbf{t}-1}$ . This implies that  $s_t^a$  is minimally sufficient for one-step direct control.
Proposition 3. Under the assumption that the graphical representation, corresponding to the environment model, is Markov and faithful to the measured data, we can build a connection between the graph structure and statistical independence of causal variables in the RL system, with (1)  $s_{i,t} \in s_t^{ar}$  if and only if  $s_{i,t} \nsubseteq R_t | a_{t-1}, s_{t-1}^r$  and  $s_{i,t} \nsubseteq a_{t-1} | \mathbf{s}_{\mathbf{t}-1}$ , (2)  $s_{i,t} \in s_t^{\bar{a}r}$  if and only if  $s_{i,t} \nsubseteq R_t | a_{t-1}, s_{t-1}^r$  and  $s_{i,t} \nsubseteq a_{t-1} | \mathbf{s}_{\mathbf{t}-1}$ , (3)  $s_{i,t} \in s_t^{a\bar{r}}$  if and only if  $s_{i,t} \nsubseteq R_t | a_{t-1}, s_{t-1}^r$  and  $s_{i,t} \nsubseteq a_{t-1} | \mathbf{s}_{\mathbf{t}-1}$ , and (4)  $s_{i,t} \in s_t^{\bar{a}\bar{r}}$  if and only if  $s_{i,t} \nsubseteq R_t | a_{t-1}, s_{t-1}^r$  and  $s_{i,t} \nsubseteq a_{t-1} | \mathbf{s}_{\mathbf{t}-1}$ .
Proof. This proposition can be easily proved by leveraging the results from Propositions 1 and 2.
$$
\left[ o _ {t}, R _ {t} \right] = f \left(s _ {t} ^ {r}, s _ {t} ^ {\bar {r}}, \eta_ {t}\right), \tag {12}
$$
$$
\begin{array}{r c l} o _ {t} & = & f _ {1} \left(s _ {t} ^ {r}, s _ {t} ^ {\bar {r}}\right), \\ R _ {t} & = & f _ {2} \left(s _ {t} ^ {r}, \eta_ {t}\right). \end{array} \tag {13}
$$
The following theorem shows that the different types of states  $s_t^{ar}$ ,  $s_t^{\bar{a}r}$ ,  $s_t^{a\bar{r}}$ , and  $s_t^{\bar{a}\bar{r}}$  are blockwise identifiable from observed image variable  $o_t$ , reward variable  $r_t$ , and action variable  $a_t$ , under reasonable and weak assumptions.
A1. The mapping  $f$  in Eq. 12 is smooth and invertible with smooth inverse.
A2. For all  $i \in \{1, \ldots, d_o + d_R\}$  and  $j \in \mathcal{F}_{i,:}$ , there exist  $\{\tilde{\mathbf{s}}_t^{(l)}\}_{l=1}^{|F_{i,:}|}$  so that  $\text{span}\{\mathbf{J}_f(\tilde{\mathbf{s}}_t^{(l)})_{i,:}\}_{l=1}^{|F_{i,:}|} = \mathbb{R}_{\tilde{\mathcal{F}}_{i,:}}^{d_s}$ , and there exists a matrix  $T$  with its support identical to that of  $\mathbf{J}_{\hat{f}}^{-1}(\hat{\mathbf{S}}_t)\mathbf{J}_f(\tilde{\mathbf{s}}_t)$ , so that  $[\mathbf{J}_f(\tilde{\mathbf{s}}_t^{(l)})T]_{j,:} \in \mathbb{R}_{\tilde{\mathcal{F}}_{i,:}}^{d_s}$ .
Then, reward-relevant and controllable states  $s_t^{ar}$ , reward-relevant but not controllable states  $s_t^{\tilde{a}\tilde{r}}$ , reward-irrelevant but controllable states  $s_t^{\tilde{a}\tilde{r}}$ , and noise  $s_t^{\tilde{a}\tilde{r}}$ , are blockwise identifiable.
In the theorem presented above, Assumption A1 only assumes the invertibility of function  $f$ , while functions  $f_{1}$  and  $f_{2}$  are considered general and not necessarily invertible, as that in [23]. Since the function  $f$  is the mapping from all (latent) variables, including noise factors, that influence the observed variables, the invertibility assumption holds reasonably. However, note that it is not reasonable to assume the invertibility of the function  $f_{2}$  since usually, the reward function is not invertible. Assumption A2, which is also given in [20, 22, 23], aims to establish a more generic condition that rules out certain sets of parameters to prevent ill-posed conditions. Specifically, it ensures that the Jacobian is not partially constant. This condition is typically satisfied asymptotically, and it is necessary to avoid undesirable situations where the problem becomes ill-posed.
Proof. The proof consists of four steps.
1. In step 1, we show that  $s_t^a = s_t^{ar} \cup s_t^{a\bar{r}}$  is blockwise identifiable, by using the characterization that the action variable  $a_{t}$  only directly influences  $s_t^{ar}$  and  $s_t^{a\bar{r}}$ .
2. In step 2, we show that  $s_t^r = s_t^{ar} \cup s_t^{\bar{a} r}$  is blockwise identifiable, by using the characterization that the future cumulative reward  $R_{t}$  is only influenced by  $s_t^{ar}$  and  $s_t^{\bar{a} r}$ .
3. In step 3, we show that  $s_t^{ar}$  is blockwise identifiable, by using the identifiability of  $s_t^{ar} \cup s_t^{a\bar{r}}$  and  $s_t^{ar} \cup s_t^{\bar{a}\bar{r}}$ .
4. In step 4, we further show the blockwise identifiability of  $s_t^{\bar{a}\bar{r}}$ ,  $s_t^{a\bar{r}}$ , and  $s_t^{\bar{a}\bar{r}}$ .
Step 1: prove the block identifiability of  $s_t^a$ .
For simplicity of notation, below, we omit the subscript  $t$ .
Let  $h \coloneqq f^{-1} \circ \hat{f}$ . We have
$$
\hat {\mathbf {s}} = h (\mathbf {s}), \tag {14}
$$
where  $h = f^{-1} \circ \hat{f}$  is the transformation between the true latent variable and the estimated one, and  $\hat{f}: S \to X$  denotes the estimated invertible generating function. Note that as both  $f^{-1}$  and  $\hat{f}$  are smooth and invertible,  $h$  and  $h^{-1}$  is smooth and invertible.
Since  $h(\cdot)$  is smooth over  $\mathcal{S}$ , its Jacobian can be written as follows:
$$
J _ {h ^ {- 1}} = \left[ \begin{array}{l l} A := \frac {\partial s ^ {\tilde {a}}}{\partial \hat {s} ^ {\tilde {a}}} & B := \frac {\partial s ^ {\tilde {a}}}{\partial \hat {s} ^ {\tilde {a}}} \\ C := \frac {\partial s ^ {\tilde {a}}}{\partial \hat {s} ^ {\tilde {a}}} & D := \frac {\partial s ^ {\tilde {a}}}{\partial \hat {s} ^ {\tilde {a}}} \end{array} \right] \tag {15}
$$
The invertibility of  $h^{-1}$  implies that  $J_{h^{-1}}$  is full rank. Since  $s^a$  has changing distributions over the action variable  $a$  while  $s^{\bar{a}}$  has invariant distributions over different values of  $a$ , we can derive that  $C = 0$ . Furthermore, because  $J_{h^{-1}}$  is full rank and  $C$  is a zero matrix,  $D$  must be of full rank, which implies  $h_a'^{-1}$  is invertible, where  $h_a'^{-1}$  denotes the first derivative of  $h_a^{-1}$ . Therefore,  $s^a$  is blockwise identifiable up to invertible transformations.
Step 2: prove the blockwise identifiability of  $s_t^p$ .
Recall that we have the following mapping:
$$
\left[ o _ {t}, R _ {t} \right] = f \left(s _ {t} ^ {r}, s _ {t} ^ {\bar {r}}, \eta_ {t}\right),
$$
$$
\begin{array}{r c l} o _ {t} & = & f _ {1} (s _ {t} ^ {r}, s _ {t} ^ {\bar {r}}), \\ R _ {t} & = & f _ {2} (s _ {t} ^ {r}, \eta_ {t}). \end{array}
$$
Note that here to recover  $s_t^r$ , we need to take into account all future rewards, because  $s_t^r$  contains all those state dimensions that have a directed path to future rewards  $r_{t+1:T}$ .  $\eta_t$  represents all other factors, except  $s_t^r$ , that influence  $R_t$  at time instance  $t$ . Further note here we assume the invertibility of  $f$ , while  $f_1$  and  $f_2$  are general functions not necessarily invertible.
We denote by  $\tilde{\mathbf{s}} = (s^r, s^{\bar{r}}, \eta)$ . We further denote by the dimension of  $s^r$  by  $d_{s^r}$ , the dimension of  $s^{\bar{r}}$  by  $d_{s^{\bar{r}}}$ , the dimension of  $\tilde{\mathbf{s}}$  by  $d_{\tilde{s}}$ , the dimension of  $o$  by  $d_o$ , and the dimension of  $R_t$  by  $d_R$ .
We denote by  $\mathcal{F}$  the support of  $\mathbf{J}_f(\mathbf{s})$ , by  $\hat{\mathcal{F}}$  the support of  $\mathbf{J}_{\hat{f}}(\hat{\mathbf{s}})$ , and by  $\mathcal{T}$  the support of  $\mathbf{T}(\mathbf{s})$ . We also denote  $T$  as a matrix with the same support as  $\mathcal{T}$ . The proof technique mainly follows [23], as well as the required assumptions, and is also related to [20, 22].
Since  $h \coloneqq \hat{f}^{-1} \circ f$ , we have  $\hat{f} = f \circ h^{-1}(\mathfrak{s})$ . By applying the chain rule repeatedly, we have
$$
\mathbf {J} _ {\hat {f}} (\hat {\mathbf {s}}) = \mathbf {J} _ {f} (\tilde {\mathbf {s}}) \cdot \mathbf {J} _ {h ^ {- 1}} (h (\tilde {\mathbf {s}})). \tag {16}
$$
With Assumption A2, for any  $i\in \{1,\ldots ,d_o + d_R\}$ , there exists  $\{\tilde{\mathbf{s}}^{(l)}\}_{l = 1}^{|F_{i,:}|}$ , s.t.  $\mathrm{span}(\{\mathbf{J}_f(\tilde{\mathbf{s}}^{(l)})_{i,:}\}_{l = 1}^{|F_{i,:}|}) = \mathbb{R}_{\mathcal{F}_{i,:}}^{d_{\tilde{s}}}$ .
Since  $\{\mathbf{J}_f(\tilde{\mathbf{s}}^{(l)})_{i,:}\}_{l = 1}^{\left|\mathcal{F}_{i,:}\right|}$  forms a basis of  $\mathsf{R}_{\mathcal{F}_{i,:}}^{d_{\bar{s}}}$ , for any  $j_0\in \mathcal{F}_{i,:}$ , we can write canonical basis vectors  $e_{j_0}\in \mathbb{R}_{\mathcal{F}_{i,:}}^{d_{\bar{s}}}$  as:
$$
e _ {j _ {0}} = \sum_ {l \in \mathcal {F} _ {i,:}} \alpha_ {l} \cdot \mathbf {J} _ {g} (\tilde {\mathbf {s}} ^ {(l)}) _ {i,:}, \tag {17}
$$
where  $\alpha_{l}\in \mathbb{R}$  is a coefficient.
Then, following Assumption A2, there exists a deterministic matrix  $T$  such that
$$
T _ {j _ {0,:}} = e _ {j _ {0}} ^ {\top} T = \sum_ {l \in \mathcal {F} _ {i,:}} \alpha_ {l} \cdot \mathbf {J} _ {g} (\tilde {\mathbf {s}} ^ {(l)}) _ {i,:} T \in \mathbb {R} _ {\hat {\mathcal {F}} _ {i,:}} ^ {d _ {\tilde {s}}}, \tag {18}
$$
where  $\in$  is due to that each element in the summation belongs to  $\mathbb{R}_{\hat{\mathcal{F}}_{i,:}}^{d_s}$ .
$$
\forall j \in \mathcal {F} _ {i,:}, T _ {j,:} \in \mathrm {R} _ {\hat {\mathcal {F}} _ {i,:}} ^ {d _ {\bar {s}}}
$$
$$
\forall (i, j) \in \mathcal {F}, \quad \{i \} \times T _ {j:} \subset \hat {\mathcal {F}}. \tag {19}
$$
We would like to show that  $\hat{s}^r$  does not depend on  $s^{\bar{r}}$  and  $\eta$ , that is,  $T_{i,j} = 0$  for  $i \in \{1, \dots, d_{s^r}\}$  and  $j \in \{d_{s^r} + 1, \dots, d_{\bar{s}}\}$ .
We prove it by contradiction. Suppose that  $\hat{s}^r$  had dependence on  $s^{\bar{r}}$ , that is,  $\exists (j_{s^r}, j_{s^{\bar{r}}}) \in \mathcal{T}$  with  $j_{s^r} \in \{1, \ldots, d_{s^r}\}$  and  $j_{s^{\bar{r}}} \in \{d_{s^r} + 1, \ldots, d_{s^r} + d_{s^{\bar{r}}}\}$ .
Hence, there must exist  $i_r \in \{d_o + 1, \dots, d_o + d_R\}$ , such that,  $(i_r, j_{s^r}) \in \mathcal{F}$ .
It follows from Equation 19 that:
$$
\{i _ {r} \} \times \mathcal {T} _ {j _ {s ^ {\tilde {r}}};} \in \hat {\mathcal {F}} \Rightarrow (i _ {r}, j _ {s ^ {\tilde {r}}}) \in \hat {\mathcal {F}}. \tag {20}
$$
However, due to the structure of  $\hat{f}_2$ ,  $[\mathbf{J}_{\hat{f}_2}]_{i_r,j_{s^{\bar{r}}}} = 0$ , which results in a contradiction. Therefore, such  $(i_r,j_{s^{\bar{r}}})$  does not exist and  $\hat{s}^r$  does not depend on  $s^{\bar{r}}$ . The same reasoning implies that  $\hat{s}^r$  does not depend on  $\eta$ . Thus,  $\hat{s}_r$  does not depend on  $(s^{\bar{r}},\eta)$ . In conclusion,  $\hat{s}^r$  does not contain extra information beyond  $s^r$ .
Similarly, we can show that  $(\hat{s}^{\bar{r}},\hat{\eta})$  does not contain information of  $s_r$ .
Therefore, there is a one-to-one mapping between  $s^r$  and  $\hat{s}^r$ .
Step 3: prove the blockwise identifiability of  $s_t^{ar}$ .
In Step 1 and Step 2, we have shown that both  $s^a$  and  $s^r$  are blockwise identifiable. That is,
$$
\begin{array}{r c l} \hat {s} ^ {r} & = & h _ {r} (s ^ {r}), \\ \hat {s} ^ {a} & = & h _ {a} (s ^ {a}), \end{array} \tag {21}
$$
where  $h_a$  and  $h_r$  are invertible functions.
According to the invariance relation of  $s^{ar}$ , We have the following relations:
$$
\hat {s} ^ {a r} = h _ {r} \left(s ^ {r}\right) _ {1: d _ {s ^ {a r}}} = h _ {a} \left(s ^ {a}\right) _ {1: d _ {s ^ {a r}}}. \tag {22}
$$
It remains to show that both  $\tilde{h}_r\coloneqq h_r(\cdot)_{1:d_{s^{ar}}}$  and  $\tilde{h}_a\coloneqq h_a(\cdot)_{1:d_{s^{ar}}}$  do not depend on  $s^{\bar{a} r}$  and  $s^{a\bar{r}}$  in their arguments.
We will prove this by contradiction, following the proof technique in [58]. Without loss of generality, we suppose  $\exists l\in \{1,\dots ,d_{s^{\bar{a} r}}\}$ ,  $s^{r*}\in S^r$ , s.t.,  $\frac{\partial\tilde{h}_r}{\partial s_l^{\bar{a} r}} (s^{r*})\neq 0$ . As  $h$  is smooth, it has continuous partial derivatives. Thus,  $\frac{\partial\tilde{h}_r}{\partial s_l^{\bar{a} r}}\neq 0$  holds true in a neighbourhood of  $s^{r*}$ , i.e.,
$$
\exists \eta > 0, s. t., s _ {l} ^ {\bar {a} r} \rightarrow \tilde {h} _ {r} \left(s ^ {a r *}, \left(s _ {- l} ^ {\bar {a} r *}, s _ {l} ^ {\bar {a} r *}\right)\right) \text {i s s t r i c t l y m o n o t o n i c o n} \left(s _ {l} ^ {\bar {a} r *} - \eta , s _ {l} ^ {\bar {a} r *} + \eta\right), \tag {23}
$$
where  $s_{-l}^{\bar{a}r}$  denotes variable  $s^{\bar{a}r}$  excluding the dimension  $l$ .
We further define an auxiliary function  $\psi : S^{ar} \times S^{\bar{a}r} \times S^{a\bar{r}} \to \mathbb{R}_{\geq 0}$  as follows:
$$
\psi \left(s ^ {a r}, s ^ {\tilde {a} r}, s ^ {a \tilde {r}}\right) := \left| \tilde {h} _ {r} \left(s ^ {r}\right) - \tilde {h} _ {a} \left(s ^ {a}\right) \right|. \tag {24}
$$
To obtain the contradiction to the invariance, it remains to show that  $\psi > 0$  with a probability greater than zero w.r.t. the true generating process.
There are two situations at  $(s^{a r*}, s^{\bar{a}r*}, s^{a\bar{r} *})$  where  $s^{\bar{a}r*}$  is an arbitrary point in  $S^{\bar{a}r}$ :
- situation 1:  $\psi(s^{a r*}, s^{\bar{a} r*}, s^{a\bar{r} *}) > 0$ ;
- situation 2:  $\psi(s^{a r*}, s^{\bar{a} r*}, s^{a\bar{r} *}) = 0$ .
In situation 1, we have identified a specific point  $\psi(s^{a r^*}, s^{\bar{a} r^*}, s^{a\bar{r}^*})$  that makes  $\psi > 0$ .
In situation 2, Eq. 23 implies that  $\forall s_{l}^{\bar{a}r} \in (s_{l}^{ar*}, s_{l}^{ar*} + \eta)$ ,  $\psi(s^{ar*}, (s_{-l}^{\bar{a}r*}, s_{l}^{\bar{a}r})$ ,  $s^{a\bar{r}^*}) > 0$ .
Thus, in both situations, we can locate a point  $(s^{a r*}, s^{\bar{a}r^{*'}}$ $s^{\bar{a}\bar{r}^{*}})$  such that  $\psi(s^{ar*}, s^{\bar{a}r^{*'}}$ $s^{\bar{a}\bar{r}^{*}}) > 0$ , where  $s^{\bar{a}r^{*'}} = s^{\bar{a}r^{*}}$  in situation 1 and  $s_{l}^{\bar{a}r^{*'}} \in (s_{l}^{ar*}, s_{l}^{ar*} + \eta)$ ,  $s_{-l}^{ar'} = s_{-l}^{ar*}$  in situation 2.
Since  $\psi$  is a composition of continuous functions, it is continuous. As pre-image of open sets are always open for continuous functions, the open set  $\mathsf{R}_{>0}$  has an open set  $\mathcal{U} \in S^{ar} \times S^{\bar{a}r} \times S^{a\bar{r}}$  as its preimage. Due to  $(s^{ar*}, s^{\bar{a}r*'}, s^{a\bar{r}*}) \in \mathcal{U}$ ,  $\mathcal{U}$  is nonempty. Further, as  $\mathcal{U}$  is nonempty and open,  $\mathcal{U}$  has a Lebesgue measure of greater than zero.
As we assume that  $p_{s^{ar}, s^{\bar{a}r}, s^{a\bar{r}}}$  is fully supported over the entire domain  $S^{ar} \times S^{\bar{a}r} \times S^{a\bar{r}}$ , we can deduce that  $\mathrm{P}_p[\mathcal{U}] > 0$ . That is,  $\psi > 0$  with a probability greater than zero, which contradicts the invariance condition, Therefore, we can show that  $\hat{h}_r(s^r)$  does not depend on  $s^{\bar{a}r}$ .
Similarly, we can show that  $\hat{h}_a(s^a)$  does not depend on  $s^{a\tilde{r}}$ .
Finally, the smoothness and invertibility of  $\hat{h}_r$  and  $\hat{h}_a$  follow from the smoothness and invertibility of  $h_r$  and  $h_a$  over the entire domain.
Therefore,  $h_r(h_a)$  is a smooth invertible mapping between  $s^{ar}$  and  $\hat{s}^{ar}$ . That is,  $s^{ar}$  is blockwise invertible.
Step 4: prove the blockwise identifiability of  $s_t^{\bar{a} r}$ ,  $s_t^{\bar{a}\bar{r}}$ , and  $s_t^{\bar{a}\bar{r}}$ .
We can use the same technique in Step 3 to show the identifiability of  $s^{\bar{a}r}$  and  $s^{a\bar{r}}$ . Specifically, since  $s_r$  and  $s^{ar}$  are identifiable, we can show that  $s^{\bar{a}r}$  is identifiable. Similarly, since  $s^a$  and  $s^{ar}$  are identifiable, we can show that  $s^{a\bar{r}}$  is identifiable. Furthermore, since  $s^{ar}$ ,  $s^{\bar{a}r}$ , and  $s^{a\bar{r}}$  are identifiable, we can show that  $s^{\bar{a}\bar{r}}$  is identifiable
B Derivation of the Objective Function
We start by defining the components of the world mode as follows:
$$
\left\{ \begin{array}{l l} \text {O b s e r v a t i o n M o d e l :} & p _ {\theta} \left(o _ {t} \mid \mathbf {s} _ {\mathbf {t}}\right) \\ \text {R e w a r d M o d e l :} & p _ {\theta} \left(r _ {t} \mid s _ {t} ^ {r}\right) \\ \text {T r a n s i t i o n M o d e l :} & p _ {\gamma} \left(\mathbf {s} _ {\mathbf {t}} \mid \mathbf {s} _ {\mathbf {t} - 1}, a _ {t - 1}\right) \\ \text {R e p r e s e n t a t i o n M o d e l :} & q _ {\phi} \left(\mathbf {s} _ {\mathbf {t}} \mid o _ {t}, \mathbf {s} _ {\mathbf {t} - 1}, a _ {t - 1}\right) \end{array} \right. \tag {25}
$$
The latent dynamics can be disentangled into four categories:
$$
\left\{ \begin{array}{l} p _ {\gamma_ {1}} \left(s _ {t} ^ {a r} \mid s _ {t - 1} ^ {r}, a _ {t - 1}\right) \\ p _ {\gamma_ {2}} \left(s _ {t} ^ {a r} \mid s _ {t - 1} ^ {r}\right) \\ p _ {\gamma_ {3}} \left(s _ {t} ^ {a \bar {r}} \mid \mathbf {s} _ {\mathbf {t} - 1}, a _ {t - 1}\right) \\ p _ {\gamma_ {4}} \left(s _ {t} ^ {a \bar {r}} \mid \mathbf {s} _ {\mathbf {t} - 1}\right) \end{array} \right. \quad \left\{ \begin{array}{l} q _ {\phi_ {1}} \left(s _ {t} ^ {a r} \mid o _ {t}, s _ {t - 1} ^ {r}, a _ {t - 1}\right) \\ q _ {\phi_ {2}} \left(s _ {t} ^ {a r} \mid o _ {t}, s _ {t - 1} ^ {r}\right) \\ q _ {\phi_ {3}} \left(s _ {t} ^ {a \bar {r}} \mid o _ {t}, \mathbf {s} _ {\mathbf {t} - 1}, a _ {t - 1}\right) \\ q _ {\phi_ {4}} \left(s _ {t} ^ {a \bar {r}} \mid o _ {t}, \mathbf {s} _ {\mathbf {t} - 1}\right) \end{array} \right. \tag {26}
$$
We follow the derivation framework in Dreamer [6] and define the information bottleneck objective for latent dynamics models [25]
$$
\max  I \left(\mathbf {s} _ {\mathbf {1}: \mathbf {T}}; \left(o _ {1: T}, r _ {1: T}\right) \mid a _ {1: T}\right) - \beta \cdot I \left(\mathbf {s} _ {\mathbf {1}: \mathbf {T}}, i _ {1: T} \mid a _ {1: T}\right), \tag {27}
$$
where  $\beta$  is scalar and  $i_t$  are dataset indices that determine the observations  $p(o_t \mid i_t) = \delta(o_t - \bar{o}_t)$  as in [26]. Maximizing the objective leads to model states that can predict the sequence of observations and rewards while limiting the amount of information extracted at each time step. We derive the lower bound of the first term in Equation 27:
$$
\begin{array}{l} I \left(\mathbf {s} _ {\mathbf {1}: \mathbf {T}}; \left(o _ {1: T}, r _ {1: T}\right) \mid a _ {1: T}\right) \\ = \mathrm {E} _ {q \left(o _ {1: T}, r _ {1: T}, \mathbf {s} _ {\mathbf {I}: \mathbf {T}}, a _ {1: T}\right)} \left(\sum_ {t} \ln p \left(o _ {1: T}, r _ {1: T} \mid \mathbf {s} _ {\mathbf {I}: \mathbf {T}}, a _ {1: T}\right) - \underbrace {\ln p \left(o _ {1 : T} , r _ {1 : T} \mid a _ {1 : T}\right)} _ {\text {c o n s t}}\right) \\ \stackrel {+} {=} \mathrm {E} \left(\sum_ {t} \ln p \left(o _ {1: T}, r _ {1: T} \mid \mathbf {s} _ {\mathbf {l}: \mathbf {T}}, a _ {1: T}\right)\right) \tag {28} \\ \geq \mathrm {E} \left(\sum_ {t} \ln p \left(o _ {1: T}, r _ {1: T} \mid \mathbf {s} _ {\mathbf {1}: \mathbf {T}}, a _ {1: T}\right)\right) - \mathrm {K L} \left(p \left(o _ {1: T}, r _ {1: T} \mid \mathbf {s} _ {\mathbf {1}: \mathbf {T}}, a _ {1: T}\right) \| \prod_ {t} p _ {\theta} \left(o _ {t} \mid s _ {t}\right) p _ {\theta} \left(r _ {t} \mid s _ {t} ^ {r}\right)\right) \\ = \mathrm {E} \left(\sum_ {t} \ln p _ {\theta} \left(o _ {t} \mid \mathbf {s} _ {\mathbf {t}}\right) + \ln p _ {\theta} \left(r _ {t} \mid s _ {t} ^ {r}\right)\right). \\ \end{array}
$$
Thus, we obtain the objective function:
$$
\mathcal {J} _ {\mathrm {O}} ^ {t} = \ln p _ {\theta} \left(o _ {t} \mid \mathbf {s} _ {\mathbf {t}}\right) \quad \mathcal {J} _ {\mathrm {R}} ^ {t} = \ln p _ {\theta} \left(r _ {t} \mid s _ {t} ^ {r}\right) \tag {29}
$$
For the second term in Equation 27, we use the non-negativity of the KL divergence to obtain an upper bound,