SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Section 3.4, we perform sampling and replacing as an implementation of causal interventions. Denote  $|\mathcal{E}|_p$  as the number of edges to predict and  $|S|$  as the size of the intervention set, which is usually set as a small constant. The spatio-temporal causal intervention mechanism owns a computation complexity compounding of sampling and replacing as  $\mathcal{O}(|S|d) + \mathcal{O}(|\mathcal{E}_p||S|d)$  in training, and no extra computation complexity in the inference stage.
Therefore, the overall computation complexity of EAGLE is:
$$
\mathcal {O} \left(\left| \mathcal {E} \right| d + \left| \mathcal {V} \right| d ^ {2}\right) + \mathcal {O} \left(\left| \mathbf {z} \right| d\right) + \mathcal {O} (K \log | \mathcal {V} |) + \mathcal {O} (\left| \mathcal {S} \right| d) + \mathcal {O} \left(\left| \mathcal {E} _ {p} \right| \left| \mathcal {S} \right| d\right). \tag {B.2}
$$
In summary, EAGLE has a linear computation complexity with respect to the number of nodes and edges, which is on par with DIDA [94] and other existing dynamic GNNs. We believe that the computational complexity bottleneck of EAGLE lies in the spatio-temporal causal intervention mechanism. We further analyze the intervention efficiency in Appendix D.5.
Space Complexity Analysis. We analyze the space complexity of each part in EAGLE as follows. Denote  $|\mathcal{V}|$  and  $|\mathcal{E}|$  as the number of nodes and edges, respectively,  $K$  as the number of environments,  $T$  as the number of time slices,  $L$  as the number of layers in EA-DGNN,  $L'$  as the number of layers in ECVAE,  $d$  as the dimension of input node features,  $d' = Kd$  as the hidden dimension of EACConv layers in EA-DGNN,  $d''$  as the hidden dimension of the encoder and decoder networks layer of ECVAE,  $\sum_{v\in \mathcal{V}}\mathrm{Var}(\mathbf{z}_v^{\mathbf{e}'})$  as the variance of  $K$  environment-aware representations.
Here we provide a rough analysis of EAGLE's space complexity. Note that, as the space complexity analysis of deep learning models is complicated, we omit some less important terms, such as intermediate activations, etc.
- storing the dynamic graph:  $\mathcal{O}(KT(|\mathcal{V}| + |\mathcal{E}|))$
- storing the node input features:  $\mathcal{O}(|\mathcal{V}|KTd)$ .
the EAConv layer:  $\mathcal{O}(Ld^{\prime 2})$
- the encoder and decoder networks of ECVAE:  $\mathcal{O}(L'd'')$
- storing generated environment samples (the number set to be the same with the observed ones):  $\mathcal{O}(|\mathcal{V}|KTd^{\prime \prime})$
- storing the states for function  $\mathbb{I}(\cdot, \cdot) \colon \mathcal{O}(K \sum_{v \in \mathcal{V}} \operatorname{Var}(\mathbf{z}_v^{\epsilon}))$ .
Then the overall space complexity of EAGLE can be roughly calculated as:
$$
\mathcal {O} (K T (| \mathcal {V} | + | \mathcal {E} |)) + \mathcal {O} (| \mathcal {V} | K T d) + \mathcal {O} \left(L d ^ {\prime 2}\right) + \mathcal {O} \left(L ^ {\prime} d ^ {\prime \prime}\right) + \mathcal {O} (| \mathcal {V} | K T d ^ {\prime \prime}) + \mathcal {O} (K \sum_ {v \in \mathcal {V}} \operatorname {V a r} \left(\mathbf {z} _ {v} ^ {\mathbf {e} ^ {\prime}}\right)). \tag {B.3}
$$
However, it is hard to intuitively draw conclusions about memory requirements from the space complexity analysis. Based on our experiments experience, EAGLE can be trained and tested under the hardware configurations (including memory requirements) listed in Appendix E.3, which is on par with the related works' requirements.
$$
\mathbf {z} = \bigcup_ {v \in \mathcal {V}} \bigcup_ {k = 1} ^ {K} \bigcup_ {t = 1} ^ {T} \left\{\mathbf {z} _ {v, k} ^ {t} \right\} \in \mathbb {R} ^ {\left(| \mathcal {V} | \times K \times T\right) \times d ^ {\prime}} \stackrel {\text {d e f}} {=} \mathcal {S} _ {\mathrm {o b}} \tag {C.1}
$$
$$
\mathcal {L} _ {\mathrm {E C V A E}} = \mathrm {K L} [ q _ {\phi} (\mathbf {e} \mid \mathbf {z}, \mathbf {y}) \| p _ {\omega} (\mathbf {e} \mid \mathbf {y}) ] - \frac {1}{| \mathbf {z} |} \sum_ {i = 1} ^ {| \mathbf {z} |} \log p _ {\omega} (\mathbf {z} \mid \mathbf {y}, \mathbf {e} ^ {(i)}), \tag {C.2}
$$
Proof. The distribution distance between  $q_{\phi}(\mathbf{e} \mid \mathbf{z}, \mathbf{y})$  and  $p_{\omega}(\mathbf{e} \mid \mathbf{z}, \mathbf{y})$  can be calculated by the KL-divergence:
$$
\begin{array}{l} \mathrm {K L} [ q _ {\phi} (\mathbf {e} \mid \mathbf {z}, \mathbf {y}) \| p _ {\omega} (\mathbf {e} \mid \mathbf {z}, \mathbf {y}) ] = \int q _ {\phi} (\mathbf {e} \mid \mathbf {z}, \mathbf {y}) \log \frac {q _ {\phi} (\mathbf {e} \mid \mathbf {z} , \mathbf {y})}{p _ {\omega} (\mathbf {e} \mid \mathbf {z} , \mathbf {y})} \mathrm {d} \phi \\ = \int q _ {\phi} (\mathbf {e} \mid \mathbf {z}, \mathbf {y}) \log \frac {q _ {\phi} (\mathbf {e} \mid \mathbf {z} , \mathbf {y}) p _ {\omega} (\mathbf {z} \mid \mathbf {y}) p _ {\omega} (\mathbf {y})}{p _ {\omega} (\mathbf {e} , \mathbf {z} , \mathbf {y})} d \phi \\ = \int q _ {\phi} (\mathbf {e} \mid \mathbf {z}, \mathbf {y}) \log q _ {\phi} (\mathbf {e} \mid \mathbf {z}, \mathbf {y}) \mathrm {d} \phi + \underbrace {\int q _ {\phi} (\mathbf {e} \mid \mathbf {z} , \mathbf {y}) \log p _ {\omega} (\mathbf {z} \mid \mathbf {y}) \mathrm {d} \phi} _ {\log p _ {\omega} (\mathbf {z} | \mathbf {y})} \\ + \int q _ {\phi} (\mathbf {e} \mid \mathbf {z}, \mathbf {y}) \log p _ {\omega} (\mathbf {y}) \mathrm {d} \phi - \int q _ {\phi} (\mathbf {e} \mid \mathbf {z}, \mathbf {y}) \log p _ {\omega} (\mathbf {e}, \mathbf {z}, \mathbf {y}) \mathrm {d} \phi \\ = \log p _ {\omega} (\mathbf {z} \mid \mathbf {y}) + \int q _ {\phi} (\mathbf {e} \mid \mathbf {z}, \mathbf {y}) \log \frac {q _ {\phi} (\mathbf {e} \mid \mathbf {z} , \mathbf {y})}{p _ {\omega} (\mathbf {z} \mid \mathbf {y} , \mathbf {e}) p _ {\omega} (\mathbf {e} \mid \mathbf {y})} \mathrm {d} \phi \\ = \log p _ {\omega} (\mathbf {z} \mid \mathbf {y}) + \mathbb {E} _ {q _ {\phi} (\mathbf {e} | \mathbf {z}, \mathbf {y})} [ \log q _ {\phi} (\mathbf {e} \mid \mathbf {z}, \mathbf {y}) - \log p _ {\omega} (\mathbf {e}, \mathbf {z} \mid \mathbf {y}) ] \\ = \log p _ {\omega} (\mathbf {z} \mid \mathbf {y}) - \mathbb {E} _ {q _ {\phi} (\mathbf {e} | \mathbf {z}, \mathbf {y})} [ - \log q _ {\phi} (\mathbf {e} \mid \mathbf {z}, \mathbf {y}) + \log p _ {\omega} (\mathbf {e}, \mathbf {z} \mid \mathbf {y}) ]. \tag {C.3} \\ \end{array}
$$
Thus the conditional log-likelihood  $\log p_{\omega}(\mathbf{z}\mid \mathbf{y})$  can be rewritten as:
$$
\log p _ {\omega} (\mathbf {z} \mid \mathbf {y}) = \mathrm {K L} \left[ q _ {\phi} (\mathbf {e} \mid \mathbf {z}, \mathbf {y}) \| p _ {\omega} (\mathbf {e} \mid \mathbf {z}, \mathbf {y}) \right] + \mathbb {E} _ {q _ {\phi} (\mathbf {e} | \mathbf {z}, \mathbf {y})} [ - \log q _ {\phi} (\mathbf {e} \mid \mathbf {z}, \mathbf {y}) + \log p _ {\omega} (\mathbf {e}, \mathbf {z} \mid \mathbf {y}) ]. \tag {C.4}
$$
Since this KL-divergence is non-negative, we then provide an Evidence Lower Bound (ELBO) for  $\log p_{\omega}(\mathbf{y}\mid \mathbf{z})$
$$
\begin{array}{l} \log p _ {\omega} (\mathbf {z} \mid \mathbf {y}) \geq \mathbb {E} _ {q _ {\phi} (\mathbf {e} \mid \mathbf {z}, \mathbf {y})} [ - \log q _ {\phi} (\mathbf {e} \mid \mathbf {z}, \mathbf {y}) + \log p _ {\omega} (\mathbf {e}, \mathbf {z} \mid \mathbf {y}) ] \\ = \mathbb {E} _ {q _ {\phi} (\mathbf {e} | \mathbf {z}, \mathbf {y})} [ - \log q _ {\phi} (\mathbf {e} | \mathbf {z}, \mathbf {y}) + \log p _ {\omega} (\mathbf {e} | \mathbf {y}) ] + \mathbb {E} _ {q _ {\phi} (\mathbf {e} | \mathbf {z}, \mathbf {y})} [ \log p _ {\omega} (\mathbf {z} | \mathbf {y}, \mathbf {e}) ] \tag {C.5} \\ = - \mathrm {K L} [ q _ {\phi} (\mathbf {e} | \mathbf {z}, \mathbf {y}) \| p _ {\omega} (\mathbf {e} | \mathbf {y}) ] + \mathbb {E} _ {q _ {\phi} (\mathbf {e} | \mathbf {z}, \mathbf {y})} [ \log p _ {\omega} (\mathbf {z} | \mathbf {y}, \mathbf {e}) ]. \\ \end{array}
$$
We can maximize  $\log p_{\omega}(\mathbf{z}\mid \mathbf{y})$  by maximizing the ELBO, or minimizing:
$$
\mathcal {L} _ {\mathrm {E C V A E}} = - \operatorname {E L B O} = \mathrm {K L} \left[ q _ {\phi} (\mathbf {e} \mid \mathbf {z}, \mathbf {y}) \| p _ {\omega} (\mathbf {e} \mid \mathbf {y}) \right] - \mathbb {E} _ {q _ {\phi} (\mathbf {e} | \mathbf {z}, \mathbf {y})} \left[ \log p _ {\omega} (\mathbf {z} \mid \mathbf {y}, \mathbf {e}) \right]. \tag {C.6}
$$
While the second term in  $\mathcal{L}_{\mathrm{ECVAE}}$  is the maximum likelihood estimation, which is infeasible to calculate directly under the expectation of the latent environment variable  $\mathbf{e} \sim p_{\phi}(\mathbf{e} \mid \mathbf{z}, \mathbf{y})$  across  $T$  time slices. Inspired by Markov Chain Monte Carlo (MCMC) sampling [23], it can be estimated as:
$$
\mathcal {L} _ {\mathrm {E C V A E}} = \mathrm {K L} \left[ q _ {\phi} (\mathbf {e} \mid \mathbf {z}, \mathbf {y}) \| p _ {\omega} (\mathbf {e} \mid \mathbf {y}) \right] - \frac {1}{| \mathbf {z} |} \sum_ {i = 1} ^ {| \mathbf {z} |} \log p _ {\omega} (\mathbf {z} \mid \mathbf {y}, \mathbf {e} ^ {(i)}). \tag {C.7}
$$
In implementations, we assume  $q_{\phi}(\mathbf{e} \mid \mathbf{z}, \mathbf{y})$  and  $p_{\omega}(\mathbf{e} \mid \mathbf{y})$  follow the multivariate normal distribution  $\mathcal{N}(\boldsymbol{\mu}; \boldsymbol{\sigma})$  parameterized by  $\boldsymbol{\mu}$  and  $\boldsymbol{\sigma}$ , so that the KL-divergence can be easily calculated. In order to optimize the ECVAE by back-propagation, we utilize a reparameterization trick:  $\mathbf{e}^{(i)} = e_{\omega}(\mathbf{z}, \mathbf{y}, \epsilon^{(i)})$  where  $\epsilon^{(i)} \sim \mathcal{N}(\mathbf{0}; \mathbf{I})$ . Here  $e_{\omega}(\cdot, \cdot, \cdot)$  is some vector-valued functions parameterized by  $\phi$ .
Proposition 2 (A Solution for  $\mathbb{I}^{\star}(\cdot)$ ). Denote  $\mathbf{z}_v^{\mathbf{e}'} = [\mathbf{z}_{v,1}',\mathbf{z}_{v,2}',\dots ,\mathbf{z}_{v,K}']$ , where  $\mathbf{z}_{v,k}' = \bigcup_{t = 1}^{T}\mathbf{z}_{v,k}^{t}$ . Let  $\operatorname {Var}(\mathbf{z}_v^{\mathbf{e}'})\in \mathbb{R}^K$  represents the variance of  $K$  environment-aware representations. The Boolean function  $\mathbb{I}(\cdot)$  is a solution for  $\mathbb{I}^{\star}(\cdot)$  with the following state update equation:
$$
\mathbb {I} (i, j) = \left\{ \begin{array}{l l} \mathbb {I} (i - 1, j) \vee \mathbb {I} (i - 1, j - \operatorname {V a r} \left(\mathbf {z} _ {v} ^ {\mathbf {e} ^ {\prime}}\right) [ i - 1 ]), & j \geq \operatorname {V a r} \left(\mathbf {z} _ {v} ^ {\mathbf {e} ^ {\prime}}\right) [ i - 1 ] \\ \mathbb {I} (i - 1, j), & \text {o t h e r w i s e} \end{array} , \right. \tag {C.8}
$$
$$
\delta_ {v} = \sum \operatorname {V a r} \left(\mathbf {z} _ {v} ^ {\mathbf {e} /}\right) - 2 j, \tag {C.9}
$$
$$
\mathcal {P} _ {\mathbf {e}} ^ {I} (v) = \left\{\mathbf {e} _ {k} \mid \operatorname {V a r} \left(\mathbf {z} _ {v} ^ {\mathbf {e} ^ {\prime}}\right) [ k ] \leq \frac {1}{K} \sum \operatorname {V a r} \left(\mathbf {z} _ {v} ^ {\mathbf {e} ^ {\prime}}\right) - \frac {\delta_ {v}}{2} \right\}, \quad \mathcal {P} _ {\mathbf {e}} ^ {V} (v) = \overline {{\mathcal {P} _ {\mathbf {e}} ^ {I} (v)}}, \tag {C.10}
$$
Proof. In order to prove the boolean function  $\mathbb{I}(\cdot)$  is a solution for  $\mathbb{I}^{\star}(\cdot)$  in Assumption 1, the problem  $\mathbb{I}(\cdot)$  solves should satisfy: (a) Optimal substructure; (b) Non-aftereffect property; (c) Overlapping sub-problems.
Next, we prove the above three conditions are valid.
(a) Optimal substructure. It is said that the problem has the optimal substructure property when the optimal solution of the problem covers the optimal solutions of its subproblems. Now we prove the optimal solution of determining the spatio-temporal invariant patterns  $\mathcal{P}_{\mathrm{e}}^{I}(v)$  for node  $v$  is constructed from the optimal solutions of the sub-problems with the bottom-up approach by using the optimal substructure property of the problem.
As  $\mathbb{I}(i,j)$  indicates whether it is feasible to select some elements from the first  $i$  elements in  $\operatorname{Var}(\mathbf{z}_v^{\mathbf{e}'})$  so that their sum is  $j$ ,  $\sum \operatorname{Var}(\mathbf{z}_v^{\mathbf{e}'})$  means the sum of elements in  $\operatorname{Var}(\mathbf{z}_v^{\mathbf{e}'})$ , the target of the problem is to find  $\delta_v$  that denotes the optimal threshold for invariance of node  $v$ . We reduce this problem by dividing  $\operatorname{Var}(\mathbf{z}_v^{\mathbf{e}'})$  into two subsets  $\operatorname{Var}(\mathbf{z}_v^{\mathbf{e}'})_I$  and  $\operatorname{Var}(\mathbf{z}_v^{\mathbf{e}'})_V$ , where  $\delta_v$  represents the value when the difference between the sum of two subsets is the smallest, i.e.,
$$
\delta_ {v} = \min  \left(\sum \operatorname {V a r} \left(\mathbf {z} _ {v} ^ {\mathrm {e} ^ {\prime}}\right) - 2 j\right), \quad \text {w h e r e} \quad \mathbb {I} (K, j) = = \text {T r u e}. \tag {C.11}
$$
Let  $\mathbb{I}_I(i,j)$  and  $\mathbb{I}_V(i,j)$  represent whether it is feasible to select some elements from the first  $i$  elements in  $\mathrm{Var}(\mathbf{z}_v^{\mathbf{e}'})_I$  and  $\mathrm{Var}(\mathbf{z}_v^{\mathbf{e}'})_V$  respectively, so that their sum is  $j$ . Suppose the sum of  $\mathrm{Var}(\mathbf{z}_v^{\mathbf{e}'})_I$  and  $\mathrm{Var}(\mathbf{z}_v^{\mathbf{e}'})_V$  are  $\sum (\mathrm{Var}(\mathbf{z}_v^{\mathbf{e}'})_I)$  and  $\sum (\mathrm{Var}(\mathbf{z}_v^{\mathbf{e}'})_V)$ , we have:
$$
\sum \left(\operatorname {V a r} \left(\mathbf {z} _ {v} ^ {\mathbf {e} /}\right) _ {I}\right) + \sum \left(\operatorname {V a r} \left(\mathbf {z} _ {v} ^ {\mathbf {e} /}\right) _ {V}\right) = \sum \operatorname {V a r} \left(\mathbf {z} _ {v} ^ {\mathbf {e} /}\right). \tag {C.12}
$$
Then, the target of the original problem is transformed into finding an optimal solution for  $i$  and  $j$ , satisfying:
$$
\left\{ \begin{array}{l} \mathbb {I} _ {I} (i, j) = = \text {T r u e}, \\ \mathbb {I} _ {I} (U _ {i}) = \sum_ {j = 1} ^ {n} \left(\mathbb {I} _ {I} (j)\right) _ {i} \end{array} \right. \tag {C.13}
$$
$$
\left\lfloor \mathbb {I} _ {V} (K - i, \sum (\operatorname {V a r} (\mathbf {z} _ {v} ^ {\mathbf {e} ^ {\prime}}) _ {I}) - j) = = \text {T r u e}, \right.
$$
$$
s. t. \min  \left(\sum (\operatorname {V a r} \left(\mathbf {z} _ {v} ^ {\mathbf {e} /}\right) _ {I}) - j - \left(\sum (\operatorname {V a r} \left(\mathbf {z} _ {v} ^ {\mathbf {e} /}\right) _ {V}) - \left(\sum (\operatorname {V a r} \left(\mathbf {z} _ {v} ^ {\mathbf {e} /}\right) _ {I}) - j\right)\right)\right). \tag {C.14}
$$
According to the definition of  $\mathbb{I}(i,j)$ , we can draw a conclusion the  $\mathbb{I}_I(i,j)$  and  $\mathbb{I}_V(i,j)$  both are the subproblems of the original problem, which similarly target at figuring out whether it is feasible to select some elements from the first  $i$  elements that satisfy their sum is  $j$ . So, function  $\mathbb{I}_I(i,j)$  and
$\mathbb{I}_V(i,j)$  is the same with  $\mathbb{I}(i,j)$  in Eq. (C.8). Assuming that when the difference in the subsets is maximized,  $\mathbb{I}_I(i,j)$  and  $\mathbb{I}_V(i,j)$  return  $j_{I}$  and  $j_{V}$ , respectively, that satisfy:
$$
\left\{ \begin{array}{l} \mathbb {I} _ {I} (i, j _ {I}) = = \text {T r u e}, \\ \mathbb {I} _ {V} (K - i, j _ {V}) = = \text {T r u e}. \end{array} \right. \tag {C.15}
$$
$$
\left\{ \begin{array}{l l} \sum \left(\operatorname {V a r} \left(\mathbf {z} _ {v} ^ {\mathbf {e} ^ {\prime}}\right) _ {I}\right) - j _ {I} = \delta_ {I}, & \mathbb {I} _ {I} (i, j _ {I}) = = \text {T r u e}, \\ \sum \left(\operatorname {V a r} \left(\mathbf {z} _ {v} ^ {\mathbf {e} ^ {\prime}}\right) _ {V}\right) - j _ {V} = \delta_ {V}, & \mathbb {I} _ {V} (K - i, \sum \left(\operatorname {V a r} \left(\mathbf {z} _ {v} ^ {\mathbf {e} ^ {\prime}}\right) _ {I}\right) - j _ {I}) = = \text {T r u e}. \end{array} \right. \tag {C.16}
$$
As  $\mathbb{I}_I(i,j_I)$  and  $\mathbb{I}_V(K - i,\sum (\mathrm{Var}(\mathbf{z}_v^{\mathbf{e}'})_I) - j_I)$  are both the optimal solutions, so we reach:
$$
\left. \left| \delta_ {I} - \delta_ {V} \right| = \left| \sum \left(\operatorname {V a r} \left(\mathbf {z} _ {v} ^ {\mathbf {e} ^ {\prime}}\right) _ {I}\right) - j _ {I} - \left(\sum \left(\operatorname {V a r} \left(\mathbf {z} _ {v} ^ {\mathbf {e} ^ {\prime}}\right) _ {V}\right) - j _ {V}\right) \right| \leq \delta_ {v}. \right. \tag {C.17}
$$
The optimal substructure property is proven.
(b) Non-aftereffect property. This property implies that once the state of a certain stage is determined by  $\mathbb{I}(\cdot)$ , it is not affected by future decision-making. In other words, the subsequent process will not affect the previous state, but only be related to the current state. Eq. (C.8) implies that, when  $j \geq \operatorname{Var}(\mathbf{z}_v^{\mathbf{e}'})[i - 1]$ , the state is decided by previous state  $\mathbb{I}(i - 1, j)$  or  $\mathbb{I}(i - 1, j - \operatorname{Var}(\mathbf{z}_v^{\mathbf{e}'})[i - 1])$ ; accordingly, when  $j < \operatorname{Var}(\mathbf{z}_v^{\mathbf{e}'})[i - 1]$ , the state is solely decided by previous state  $\mathbb{I}(i - 1, j)$ . All state transition processes are based on historical states and executed in a one-way transition mode, meeting the requirements of the non-aftereffect property. Generally speaking, the non-aftereffect property is a relaxation condition, that is, as long as the properties of the optimal substructure property are satisfied, the non-aftereffect property will be basically satisfied.
(c) Overlapping sub-problems.  $\mathbb{I}(\cdot)$  solve the problem from top to bottom in a recursive way, each sub-problem is not always a new problem, but a large number of repeated sub-problems, that is, when different decision sequences reach a certain stage, they will generate duplicate problems. When figuring out  $\mathbb{I}(\cdot)$ , we consider two situations:
- Not select the  $i$ -th element in  $\operatorname{Var}(\mathbf{z}_v^{\mathbf{e}'})$ , i.e.,  $\mathbb{I}(i,j) = \mathbb{I}(i - 1,j)$ ;
- Select the  $i$ -th element in  $\operatorname{Var}(\mathbf{z}_v^{\mathbf{e}'})$ , i.e.,  $\mathbb{I}(i,j) = \mathbb{I}(i - 1, j - \operatorname{Var}(\mathbf{z}_v^{\mathbf{e}'})[i - 1])$ .
As we can observe, we need the solution of the sub-problem  $\mathbb{I}(i - 1,j)$  and  $\mathbb{I}(i - 1,j - \mathrm{Var}(\mathbf{z}_v^{\mathbf{e}'})[i - 1])$  when figuring out  $\mathbb{I}(i,j)$ , which are already calculated by  $\mathbb{I}(i - 1,\cdot)$ . So we can prove that the problem  $\mathbb{I}(\cdot)$  solved has overlapping sub-problems.
We then have proven the function  $\mathbb{I}(\cdot)$  is a solution for the function  $\mathbb{I}^{\star}(\cdot)$  in Assumption 1, from which we can obtain the optimal spatio-temporal invariance threshold  $\delta_v$  of node  $v$ . Then the spatio-temporal invariant/variant patterns for node  $v$  can be exploited by Eq. (C.10), and their unions constitute the overall  $\mathcal{P}_{\mathrm{e}}^{I}$  and  $\mathcal{P}_{\mathrm{e}}^{V}$ . We conclude the proof for Proposition 2.
Proof. We first propose the following lemma to rewrite the Sufficient Condition and the Invariance Property in Assumption 1 using the information theory [43].
Lemma 1 (Mutual Information Equivalence). The Invariance Property and Sufficient Condition in Assumption 1 can be equivalently represented with the Mutual Information  $I(\cdot ;\cdot)$ :
(a) Invariance Property:  $p(\mathbf{Y}^T\mid \mathcal{P}_{\mathbf{e}}^I,\mathbf{e}) = p(\mathbf{Y}^T\mid \mathcal{P}_{\mathbf{e}}^I)\Leftrightarrow I(\mathbf{Y}^T;\mathbf{e}\mid \mathcal{P}_{\mathbf{e}}^I) = 0;$
(b) Sufficient Condition:  $\mathbf{Y}^T\perp \mathcal{P}_{\mathbf{e}}^V\mid \mathcal{P}_{\mathbf{e}}^I\Leftrightarrow I(\mathbf{Y}^T;\mathcal{P}_{\mathbf{e}}^I)$  is maximized.
Proof. We prove Lemma 1 by respectively proving the above two conditions are valid.
(a) Invariance Property. According to the definition of the Mutual Information, we can easily get the following equation:
$$
I \left(\mathbf {Y} ^ {T}; \mathbf {e} \mid \mathcal {P} _ {\mathbf {e}} ^ {I}\right) = \mathrm {K L} \left[ p \left(\mathbf {Y} ^ {T} \mid \mathbf {e}, \mathcal {P} _ {\mathbf {e}} ^ {I}\right) \| p \left(\mathbf {Y} ^ {T} \mid \mathcal {P} _ {\mathbf {e}} ^ {I}\right) \right] = 0, \tag {C.18}
$$
where  $\mathrm{KL}[\cdot ||\cdot ]$  is the Kullback-Leibler (KL) divergence [37]. Thus we have proved the equivalence in Invariance Property.
(b) Sufficient Condition. We demonstrate sufficiency and necessity through the following two steps.
First, we prove that for  $\mathbf{Y}^T$ ,  $\mathcal{P}_{\mathbf{e}}^I$  and  $\epsilon$  satisfying  $\mathbf{Y}^T = g(\mathbf{Z}_I^{1:T}) + \epsilon$  would also satisfy  $\mathcal{P}_{\mathbf{e}}^I = \arg \max_{\mathcal{P}_{\mathbf{e}}^I} I(\mathbf{Y}^T; \mathcal{P}_{\mathbf{e}}^I)$ . We perform proving by contradiction. Suppose  $\mathcal{P}_{\mathbf{e}}^I \neq \arg \max_{\mathcal{P}_{\mathbf{e}}^I} I(\mathbf{Y}^T; \mathcal{P}_{\mathbf{e}}^I)$  and there exists  $\mathcal{P}_{\mathbf{e}}^{II'} = \arg \max_{\mathcal{P}_{\mathbf{e}}^I} \mathcal{P}_{\mathbf{e}}^I$  where  $\mathcal{P}_{\mathbf{e}}^{II'} \neq \mathcal{P}_{\mathbf{e}}^I$ . We can always find a mapping function  $\mathcal{M}$  so that  $\mathcal{P}_{\mathbf{e}}^{II'} = \mathcal{M}(\mathcal{P}_{\mathbf{e}}^I, \pi)$  where  $\pi$  is a random variable. Then we reach:
$$
I \left(\mathbf {Y} ^ {T}; \mathcal {P} _ {\mathbf {e}} ^ {I t}\right) = I \left(\mathbf {Y} ^ {T}; \mathcal {P} _ {\mathbf {e}} ^ {I}, \pi\right) = I \left(g \left(\mathbf {Z} _ {I} ^ {1: T}\right); \mathcal {P} _ {\mathbf {e}} ^ {I}, \pi\right) = I \left(g \left(\mathbf {Z} _ {I} ^ {1: T}\right); \mathcal {P} _ {\mathbf {e}} ^ {I}\right) = I \left(\mathbf {Y} ^ {T}; \mathcal {P} _ {\mathbf {e}} ^ {I}\right), \tag {C.19}
$$
which leads to a contradiction.
Next, we prove that for  $\mathbf{Y}^T$ ,  $\mathcal{P}_{\mathbf{e}}^{I}$  and  $\epsilon$  satisfying  $\mathcal{P}_{\mathbf{e}}^{I} = \arg \max_{\mathcal{P}_{\mathbf{e}}^{I}} I(\mathbf{Y}^{T}; \mathcal{P}_{\mathbf{e}}^{I})$  would also satisfy  $\mathbf{Y}^T = g(\mathbf{Z}_{I}^{1:T}) + \epsilon$ . We also perform proving by contradiction. Suppose that  $\mathbf{Y}^T \neq g(\mathbf{Z}_{I}^{1:T}) + \epsilon$  and  $\mathbf{Y}^T = g(\mathbf{Z}_{I'}^{1:T}) + \epsilon$  where  $\mathcal{P}_{\mathbf{e}}^{II'} \neq \mathcal{P}_{\mathbf{e}}^{I}$ . We then have the following inequality:
$$
I \left(g \left(\mathbf {Z} _ {I ^ {\prime}} ^ {1: T}\right); \mathcal {P} _ {\mathbf {e}} ^ {I}\right) \leq I \left(g \left(\mathbf {Z} _ {I ^ {\prime}} ^ {1: T}\right); \mathcal {P} _ {\mathbf {e}} ^ {I ^ {\prime}}\right), \tag {C.20}
$$
from which we can obtain that  $\mathcal{P}_{\mathbf{e}}^{I I} = \arg \max_{\mathcal{P}_{\mathbf{e}}^{I}}I(\mathbf{Y}^{T};\mathcal{P}_{\mathbf{e}}^{I})$  , leading to a contradiction.
Now, we prove Proposition 3 in the following two perspectives.
First, we prove that minimizing the expectation term  $(\mathcal{L}_{\mathrm{task}})$  in Eq. (12) can enforce the model to satisfy the Sufficient Condition in Assumption 1 (note that we omit the subscript  $\mathcal{P}_{\mathbf{e}}^{I}$  in  $\mathbf{Z}_I^{1:T}$  for brevity in the rest of this subsection if there are no special declarations).
From the SCM model in Figure 1(b), we can analyze that  $\max_{q(\mathbf{Z}^{1:T}|\mathbf{G}^{1:T})}I(\mathbf{Y}^T;\mathbf{Z}^{1:T})$  is equivalent to  $\min_{q(\mathbf{Z}^{1:T}|\mathbf{G}^{1:T})}I(\mathbf{Y}^T;\mathbf{G}^{1:T}\mid \mathbf{Z}^{1:T})$ , as we filter out the spurious correlations that contain in the dependencies between  $\mathbf{Y}^T$  and  $\mathbf{G}^{1:T}$ . Treating  $q(\mathbf{Z}^{1:T}\mid \mathbf{G}^{1:T})$  as the variational distribution, we have the following upper bound: