SlowGuess/NeurIPS_2023_OCR huggingface dataset:
When  $L = 0$ , output of  $\mathcal{A}$  is exactly  $\mathbb{H}^V$ , which has been simulated by  $\mathcal{A}'$  above.
Suppose  $L = k + 1$ , let  $\tilde{\mathcal{A}}$  denote  $\mathbf{R}^2$ -GNN extracted from  $\mathcal{A}$  but without the last layer  $k + 1$ . By induction, we can construct a  $\mathbf{R}^2$ -TGNN  $\tilde{\mathcal{A}}'$  that simulates  $\tilde{\mathcal{A}}(\mathbb{H}^V, \mathbb{H}^E)$ . Then we need to append three layers to  $\tilde{\mathcal{A}}'$  to simulate the last layer of  $\mathcal{A}$ .
Let  $u^{(L)}, g^{(L)}, r^{(L)}$  denote parameters of the last layer of  $\mathcal{A}$ . Using notations in Equation (2), let  $\{C^{(l)}, (A_{j}^{(l)})_{j=1}^{|P_2|}, A_{aux1}^{(l)}, A_{aux2}^{(l)}, R^{(l)}\}_{l=1}^3$  denote parameters of the three layers appended to  $\tilde{\mathcal{A}}_T'$ . They are defined as follows:
First, we can choose specific function in the first two added layers, such that the following holds:
1. For any added node  $e_{ij}$ , feature outputted by the new model is  $\mathbf{x}_{e_{ij}}^{(2)} = [\mathbb{H}_{ij}^{E},\mathbf{x}_{i}',\mathbf{x}_{j}']$ , where  $\mathbf{x}^{(2)}$  denotes the feature outputted by the second added layer, and  $\mathbf{x}_i',\mathbf{x}_j'$  are node features of  $i,j$  outputted by  $\tilde{\mathcal{A}}^{\prime}$ . For a feature  $\mathbf{x}$  of added node of this form, we define  $\mathbf{x}_0,\mathbf{x}_1,\mathbf{x}_2$  as corresponding feature slices where  $\mathbb{H}_{ij}^{E},\mathbf{x}_{i}',\mathbf{x}_{j}'$  have been stored.
2. For any primal node, its feature  $\mathbf{x}$  only stores  $\mathbf{x}_i'$  in  $\mathbf{x}_1$ , and  $\mathbf{x}_0, \mathbf{x}_2$  are all slices of dummy bits.
Let  $\mathbf{X}$  be a multiset of features that represents function input. For the last added layer, we can choose specific functions as follows:
$$
R ^ {(3)} (\mathbf {X}) := r ^ {(L)} \left(\left\{\left. \left\{\mathbf {x} _ {1} \mid \mathbf {x} \in \mathbf {X}, \text {p r i m a l} (\mathbf {x}) \right\} \right\}\right) \right. \tag {45}
$$
$$
A _ {a u x 1} ^ {(3)} (\mathbf {X}) := g ^ {(L)} \left(\left\{\left\{\left(\mathbf {x} _ {1}, \mathbf {x} _ {2}, \mathbf {x} _ {0}\right) \mid \mathbf {x} \in \mathbf {X} \right\} \right\}\right) \tag {46}
$$
$$
C ^ {(3)} \left(\mathbf {x} _ {a u x 1}, \mathbf {x} _ {g}\right) := u ^ {(L)} \left(\mathbf {x} _ {a u x 1}, \mathbf {x} _ {g}\right) \tag {47}
$$
where  $\mathbf{x}_{aux1},\mathbf{x}_g$  are outputs of  $R^{(3)}$  and  $A_{aux1}^{(3)}$ , and all useless inputs of  $C^{(3)}$  are omitted. Comparing this construction with Equation (41). It's easy to see that after the last layer appended, we can construct an equivalent  $\mathbb{R}^2$ -TGNN  $\mathcal{A}'$  that captures  $\mathcal{A}$  on transformed graph. By inductive argument, we prove time-then-graph  $\subseteq \mathbb{R}^2$ -TGNN  $\circ F^T$ .
Then we need to show  $\mathbf{R}^2$ -TGNN  $\circ F^T \subseteq$  time-then-graph.
In Theorem 16, we will prove  $\mathbf{R}^2$ -TGNN  $\circ F^T = \mathbf{R}^2$ -GNN  $\circ F \circ H$ . Its proof doesn't depend on Theorem 15, so let's assume it's true for now. Then, instead of proving  $\mathbf{R}^2$ -TGNN  $\circ F^T$ , it's sufficient to show  $\mathbf{R}^2$ -GNN  $\circ F \circ H \subseteq$  time-then-graph.
Let  $P_1^T, P_2^T$  denote the set of temporalized unary and binary predicate sets defined in Definition 12. Based on most expressive ability of Recurrent Neural Networks shown in Siegelmann and Sontag [1992], we can get a most expressive representation for unary and binary fact sequences through RNN. A most expressive RNN representation function is always injective, thus there exists a decoder function translating most-expressive representations back to raw sequences. Therefore, we are able to find an appropriate RNN such that its output features  $\mathbb{H}^V, \mathbb{H}^E$  in Equation (42), Equation (43) contain all information needed to reconstruct all temporalized unary and binary facts related to the corresponding nodes.
For any  $\mathbf{R}^2$ -GNN  $\mathcal{A}$  on transformed collapsed temporal knowledge graph, we want to construct an equivalent time-then-graph model  $\{\mathbf{RNN},\mathcal{A}'\}$  to capture  $\mathcal{A}$ . In order to show the existence of the time-then-graph model, we will do an inductive construction over layer number  $L$  of  $\mathcal{A}$ . Here in order to build inductive argument, we will consider a following stronger result and aim to prove it: In addition to the existence of  $\mathcal{A}'$ , we claim there also exists a function  $f_{\mathcal{A}}$  with the following property: For any two nodes  $a,b$  with at least one edge,  $f_{\mathcal{A}}(\mathbf{x}_a',\mathbf{x}_b',\mathbb{H}_{ab}^E) = \mathbf{x}_{e_{ab}}$ , where  $\mathbf{x}_a',\mathbf{x}_b',\mathbb{H}_{ab}^E$  are features of  $a,b$  and edge information between  $a,b$  outputted by  $\mathcal{A}'$ , and  $\mathbf{x}_{e_{ab}}$  is the feature of added node  $e_{ab}$  outputted by  $\mathcal{A} \circ F \circ H$ . It suffices to show that there exists such function  $f_{\mathcal{A}}$  as well as a time-then-graph model  $\{\mathbf{RNN},\mathcal{A}'\}$  such that the following conditions hold:
For any graph  $G$  and its node  $a, b \in V(G)$ ,
1.  $\mathbb{H}_a^{V,(l)} = [\mathbf{x}_a,Enc(\{\{\mathbf{x}_{e_{aj}}|j\in \mathcal{N}(a)\})]$
2. If there is at least one edge between  $a, b$  in history,  $f_{\mathcal{A}}(\mathbb{H}_a^{V,(l)}, \mathbb{H}_b^{V,(l)}, \mathbb{H}_{ab}^E) = \mathbf{x}_{e_{ab}}$ . Otherwise,  $f_{\mathcal{A}}(\mathbb{H}_a^{V,(l)}, \mathbb{H}_b^{V,(l)}, \mathbb{H}_{ab}^E) = \mathbf{0}$
where  $\mathbb{H}_a^{V,(l)},\mathbb{H}_b^{V,(l)}$  are node features outputted by  $\mathcal{A}'$ , while  $\mathbf{x}_a,\mathbf{x}_{e_{ab}}$  are node features outputted by  $\mathcal{A}$  on transformed collapsed graph.  $Enc(\mathbf{X})$  is some injective encoding that stores all information of multiset  $\mathbf{X}$ . For a node feature  $\mathbb{H}_a^{V,(l)}$  of above form, let  $\mathbb{H}_{a,0}^{V,(l)}\coloneqq \mathbf{x}_a,\mathbb{H}_{a,1}^{V,(l)} = Enc(\{\{\mathbf{x}_{e_{aj}}|j\in \mathcal{N}(a)\})$  denote two slices that store independent information in different positions.
For the base case  $L = 0$ , the node feature only depends on temporalized unary facts related to the corresponding node. Since by RNN we can use most expressiveness representation to capture all unary facts. A specific RNN already captures  $\mathcal{A}$  when  $L = 0$ . Moreover, there is no added node  $e_{ab}$  that relates to any unary fact, so a constant function already satisfies the condition of  $f_{\mathcal{A}}$  when  $L = 0$ . Therefore, our result holds for  $L = 0$
Assume  $L = k + 1$ , let  $\widehat{\mathcal{A}}$  denote the model generated by the first  $k$  layers of  $\mathcal{A}$ . By induction, there is time-then-graph model  $\widehat{\mathcal{A}}'$  and function  $f_{\widehat{\mathcal{A}}'}$  that captures output of  $\widehat{\mathcal{A}}'$  on transformed collapsed graph. We can append a layer to  $\widehat{\mathcal{A}}'$  to build  $\mathcal{A}'$  that simulates  $\mathcal{A}$ . Let  $\{C^{(L)}, (A_j^{(L)})_{j=1}^{T|P_2|}, A_{aux1}^{(L)}, A_{aux2}^{(L)}, R^{(L)}\}$  denote the building blocks of layer  $L$  of  $\mathcal{A}$ , and let  $u^*, g^*, r^*$  denote functions used in the layer that will be appended to  $\widehat{\mathcal{A}}'$ . They are defined below:
$$
g ^ {*} \left(\{\{(\mathbb {H} _ {i} ^ {V, (l - 1)}, \mathbb {H} _ {j} ^ {V, (l - 1)}, \mathbb {H} _ {i j} ^ {E} | j \in \mathcal {N} (i)) \} \}\right) := A _ {a u x 1} ^ {(L)} \left(\{\{f _ {\hat {\mathcal {A}}} (\mathbb {H} _ {i} ^ {V, (l - 1)}, \mathbb {H} _ {j} ^ {V, (l - 1)}, \mathbb {H} _ {i j} ^ {E}) | j \in \mathcal {N} (i) \} \}\right) \tag {48}
$$
$$
r ^ {*} \left(\{\{\mathbb {H} _ {j} ^ {V, (l - 1)} | j \in V (G) \} \}\right) = R ^ {(L)} \left(\{\{\mathbb {H} _ {j, 0} ^ {V, (l - 1)} | j \in V (G) \} \} \cup \left(\bigcup_ {j \in V (G)} D e c \left(\mathbb {H} _ {j, 1} ^ {V, (l - 1)}\right)\right)\right) \tag {49}
$$
$$
u ^ {*} (\mathbf {x} _ {g}, \mathbf {x} _ {r}) = C ^ {(L)} (\mathbf {x} _ {g}, \mathbf {x} _ {r}) \tag {50}
$$
where  $\mathbf{x}_g, \mathbf{x}_r$  are outputs of  $g^*$  and  $r^*$ .  $Dec(\mathbf{X})$  is a decoder function that do inverse mapping of  $Enc(\mathbf{X})$  mentioned above, so  $Dec(\mathbb{H}_{j,1}^{V,(l-1)})$  is actually  $\{\{\mathbf{x}_{e_{aj}} | j \in \mathcal{N}(a)\}\}$ . Note that primal nodes
in transformed graph only has type aux1- neighbors, so two inputs  $\mathbf{x}_g,\mathbf{x}_r$ , one for aux1 aggregation output and one for global readout are already enough for computing the value. Comparing the three rules above with Equation (2), we can see that our new model  $\mathcal{A}'$  perfectly captures  $\mathcal{A}$ .
We've captured  $\mathcal{A}$ , and the remaining work is to construct  $f_{\mathcal{A}}$  defined above to complete inductive assumption. We can just choose a function that simulates message passing between pairs of added nodes  $e_{ab}$  and  $e_{ba}$  as well as message passing between  $e_{ab}$  and  $a$ , and that function satisfies the condition for  $f_{\mathcal{A}}$ . Formally speaking,  $f_{\mathcal{A}}$  can be defined below:
$$
f _ {\mathcal {A}} \left(\mathbb {H} _ {i} ^ {V, (l)}, \mathbb {H} _ {j} ^ {V, (l)}, \mathbb {H} _ {i j} ^ {E}\right) := \operatorname {S i m} _ {\mathcal {A} _ {L}} \left(\mathbb {H} _ {i} ^ {V, (l - 1)}, \mathbb {H} _ {g} ^ {(l - 1)}, g _ {i j}, g _ {j i}, \mathbb {H} _ {i j} ^ {E}\right) \tag {51}
$$
$$
g _ {i j} := f _ {\hat {\mathcal {A}} ^ {\prime}} \left(\mathbb {H} _ {i} ^ {V, (l - 1)}, \mathbb {H} _ {j} ^ {V, (l - 1)}, \mathbb {H} _ {i j} ^ {E}\right), \mathbb {H} _ {g} ^ {(l - 1)} := \{\{\mathbb {H} _ {i} ^ {V, (l - 1)} | i \in V (G) \} \} \tag {52}
$$
Let's explain this equation,  $\mathbf{Sim}_{\mathcal{A}_L}(a,g,s,b,e)$  is a local simulation function which simulates single-iteration message passing in the following scenario:
Suppose there is a graph  $H$  with three constants  $V(H) = \{a, e_{ab}, e_{ba}\}$ . There is an aux1 edge between  $a$  and  $e_{ab}$ , an aux2 edge between  $e_{ab}$  and  $e_{ba}$ , and additional edges of different types between  $e_{ab}$  and  $e_{ba}$ . The description of additional edges can be founded in  $e$ . Initial node features of  $a, e_{ab}, e_{ba}$  are set to  $a, s, b$  respectively. and the global readout output is  $g$ . Finally, run  $L$ -th layer of  $\mathcal{A}$  on  $H$ , and  $\operatorname{Sim}_{\mathcal{A}_L}$  is node feature of  $e_{ab}$  outputted by  $\mathcal{A}_L$ .
Note that if we use appropriate injective encoding or just use concatenation technic,  $\mathbb{H}_g^{(l - 1)},\mathbb{H}_i^{V,(l - 1)},\mathbb{H}_j^{V,(l - 1)}$  can be accessed from  $\mathbb{H}_i^{V,(l)},\mathbb{H}_i^{V,(l)}$ . Therefore the above definition for  $f_{\mathcal{A}}$  is well-defined. Moreover, in the above explanation we can see that  $f_{\mathcal{A}}(\mathbb{H}_i^{V,(l - 1)},\mathbb{H}_j^{V,(l - 1)},\mathbb{H}_{ij}^E)$  is exactly node feature of  $e_{ij}$  outputted by  $\mathcal{A}$  on the transformed collapsed graph, so our proof finishes.
I Proof of Theorem 16
Theorem 16.  $R^2$ -TGNN  $\circ F^T = R^2$ -TGNN  $\circ F \circ H$ .
First, we recall the definition for  $\mathbf{R}^2$ -TGNN as in Equation (53):
$$
\mathbf {x} _ {v} ^ {t} = \mathcal {A} _ {t} \left(G _ {t}, v, \mathbf {y} ^ {t}\right) \quad \text {w h e r e} \quad \mathbf {y} _ {v} ^ {t} = \left[ I _ {G _ {t}} (v): \mathbf {x} _ {v} ^ {t - 1} \right], \forall v \in V (G _ {t}) \tag {53}
$$
We say a  $\mathbb{R}^2$ -TGNN is homogeneous if  $\mathcal{A}_1, \ldots, \mathcal{A}_T$  share the same parameters. In particular, we first prove Lemma 30, namely, homogeneous  $\mathbb{R}^2$ -TGNN and  $\mathbb{R}^2$ -TGNN (where paramters in  $\mathcal{A}_1, \ldots, \mathcal{A}_T$  may differ) have the same expressiveness.
Lemma 30. homogenous  $R^2$ -TGNN =  $R^2$ -TGNN
Proof. The forward direction homogeneous  $\mathbf{R}^2$ -TGNN  $\subseteq$ $\mathbf{R}^2$ -TGNN trivially holds. It suffices to prove the backward direction.
Let  $\mathcal{A}:\{\mathcal{A}_t\}_{t = 1}^T$  denote a  $\mathbb{R}^2$ -TGNN. Without loss of generality, we can assume all models in each timestamps have the same layer number  $L$ . Then for each  $1\leq t\leq T$ , we can assume all  $\mathcal{A}_t$  can be represented by  $\{C_t^{(l)},(A_{t,j}^{(l)})_{j = 1}^{|P_2|},R_t^{(l)}\}_{l = 1}^L$ . Furthermore, without loss of generality, we can assume all output dimensions for  $A_{t,j}^{(l)},R_{t}^{(l)}$  and  $C_t^{(l)}$  are  $d$ . As for input dimension, all of these functions also have input dimension  $d$  for  $2\leq l\leq L$ . Specially, by updating rules of  $\mathbb{R}^2$ -TGNN Equation (53), in the initialization stage of each timestamp we have to concat a feature with length  $|P_{1}|$  to output of the former timestamp, so the input dimension for  $A_{t,j}^{(1)},R_{t}^{(1)},C_{t}^{(1)}$  is  $d + |P_1|$ .
We can construct an equivalent homogeneous  $\mathbf{R}^2$ -TGNN with  $L$  layers represented by  $\{C^{*,(l)}, (A_j^{*,(l)})_{j=1}^{|P_2|}, R^{*,(l)}\}_{l=1}^L$ . For  $2 \leq l \leq L$ ,  $C^{*,(l)} A_j^{*,(l)}, R^{*,(l)}$  use output and input feature dimension  $d' = T d$ . Similar to the discussion about feature dimension above, since we need to concat the unary predicates information before each timestamp, for layer  $l = 1$ ,  $C^{*,(1)}, A_j^{*,(1)}, R^{*,(1)}$  have input dimension  $d' + |P_1|$  and output dimension  $d'$ . For dimension alignment,  $\mathbf{x}_v^0$  used in Equation (53) is defined as zero-vector with length  $d'$ .
Next let's define some symbols for notation simplicity. For a feature vector  $\mathbf{x}$ , let  $\mathbf{x}[i,j]$  denotes the slice of  $\mathbf{x}$  in dimension  $[i,j]$ . By the discussion above, in the following construction process we will only need feature  $\mathbf{x}$  with dimension  $d'$  or  $d' + |P_1|$ . When  $\mathbf{x}$  has dimension  $d'$ ,  $\mathbf{x}^{(i)}$  denotes  $\mathbf{x}[(i-1)d+1,id]$ , otherwise it denotes  $\mathbf{x}[|P_1|+(i-1)d+1,|P_1|+id]$ . Let  $[\mathbf{x}_1,\dots,\mathbf{x}_T]$  or  $[\mathbf{x}_t]_{t=1}^T$  denotes the concatenation of a sequence of feature  $\mathbf{x}_1,\dots,\mathbf{x}_T$ , and  $[\mathbf{x}]^n$  denote concatenation of  $n$  copies of  $\mathbf{x}$ ,  $\mathbf{0}^n$  denotes zero vectors of length  $n$ . Furthermore, let  $\mathbf{X}$  denotes a multiset of  $\mathbf{x}$ . Follows the updating rules defined in Equation (2), for all  $1 \leq j \leq |P_2|$ ,  $1 \leq l \leq L$ ,  $A_j^{*,(l)}, R^{*,(l)}$  should get input of form  $\mathbf{X}$ , and the combination function  $C^{*,(l)}$  should get input of form  $(\mathbf{x}_0, (\mathbf{x}_j)_{j=1}^{|P_2|}, \mathbf{x}_g)$ , where  $\mathbf{x}_0$  is from the node itself,  $(\mathbf{x}_j)_{j=1}^{|P_2|}$  are from aggregation functions  $(A_j^{*,(l)})_{j=1}^{|P_2|}$  and  $\mathbf{x}_g$  is from the global readout  $R^{*,(l)}$ . The dimension of  $\mathbf{x}$  or  $\mathbf{X}$  should match the input dimension of corresponding function. For all  $1 \leq l \leq L$ , parameters in layer  $l$  for the new model are defined below
$$
l = 1: C ^ {*}, ^ {(l)} \left(\mathbf {x} _ {0}, \left(\mathbf {x} _ {j}\right) _ {j = 1} ^ {| P _ {2} |}, \mathbf {x} _ {g}\right) := \left[ C _ {t} ^ {(l)} \left(\left[ \mathbf {x} _ {0} [ 1, | P _ {1} | ], \mathbf {x} _ {0} ^ {(t - 1)} \right], \left(\mathbf {x} _ {j} ^ {(t)}\right) _ {j = 1} ^ {| P _ {2} |}, \mathbf {x} _ {g} ^ {(t)}\right) \right] _ {t = 1} ^ {T} \tag {54}
$$
$$
2 \leq l \leq L: C ^ {*}, (l) \left(\mathbf {x} _ {0}, \left(\mathbf {x} _ {j}\right) _ {j = 1} ^ {| P _ {2} |}, \mathbf {x} _ {g}\right) := \left[ C _ {t} ^ {(l)} \left(\mathbf {x} _ {0} ^ {(t)}, \left(\mathbf {x} _ {j} ^ {(t)}\right) _ {j = 1} ^ {| P _ {2} |}, \mathbf {x} _ {g} ^ {(t)}\right) \right] _ {t = 1} ^ {T} \tag {55}
$$
$$
\forall j \in [ K ], l = 1: A _ {j} ^ {* (l)} (\mathbf {X}) = \left[ \right. A _ {t, j} ^ {(l)} \left( \right.\left\{ \right.\left[ \mathbf {x} [ 1, \mid P _ {1} \right], \mathbf {x} ^ {(t - 1)} \left. \right] \mid \mathbf {x} \in \mathbf {X} \left. \right\}\left. \right)\left. \right] _ {t = 1} ^ {T} \tag {56}
$$
$$
l = 1: R ^ {*, (l)} (\mathbf {X}) = \left. \left[ R _ {t} ^ {(l)} \left(\{\{[ \mathbf {x} [ 1, | P _ {1} | ], \mathbf {x} ^ {(t - 1)} ] \mid \mathbf {x} \in \mathbf {X} \}\right) \right\} \right] _ {t = 1} ^ {T} \tag {57}
$$
$$
\forall j \in [ K ], 2 \leq l \leq L: A _ {j} ^ {*}, ^ {(l)} (\mathbf {X}) = \left[ \right. A _ {t, j} ^ {(l)} \left( \right.\left\{ \right.\left\{\mathbf {x} ^ {(t)} \mid \mathbf {x} \in \mathbf {X} \right\}\left. \right)\left. \right\}\left. \right)\left. \right] _ {t = 1} ^ {T} \tag {58}
$$
$$
2 \leq l \leq L: R ^ {* (l)} (\mathbf {X}) = \left[ R _ {t} ^ {(l)} \left(\{\{\mathbf {x} ^ {(t)} | \mathbf {x} \in \mathbf {X} \}\right) \right] _ {t = 1} ^ {T} \tag {59}
$$
The core trick is to use  $T$  disjoint slices  $\mathbf{x}^{(1)}\ldots \ldots \mathbf{x}^{(T)}$  to simulate  $T$  different models  $\mathcal{A}_1\ldots \ldots \mathcal{A}_T$  at the same time. Since these slices are isolated from each other, a proper construction above can be found. The only speciality is that in layer  $l = 1$ , we have to incorporate the unary predicate information  $\mathbf{x}[1,|P_1|]$  into each slice. By the construction above, we can see that for any node  $v$ ,  $\mathbf{x}_v^{(T)}$  is exactly the its feature outputted by  $\mathcal{A}$ . Therefore, we finally construct an homogeneous  $\mathbb{R}^2$ -TGNN equivalent with  $\mathcal{A}$ .
Now, we start to prove Theorem 16.
Theorem 16.  $R^2$ -TGNNs  $\circ F^T = R^2$ -GNNs  $\circ F \circ H$  on any universal graph class  $\mathcal{G}_u$ .
Proof. Since  $\mathbf{R}^2$ -TGNN  $\circ F^T$  only uses a part of predicates of  $P' = F(H(P))$  in each timestamp, the forward direction  $\mathbf{R}^2$ -TGNN  $\circ F^T \subseteq \mathbf{R}^2$ -GNN  $\circ F \circ H$  trivially holds.
For any  $\mathbb{R}^2$ -GNN  $\mathcal{A}$  under  $P'$ , we want to construct an  $\mathbb{R}^2$ -TGNN  $\mathcal{A}'$  under  $F^T(P)$  such that for any temporal knowledge graph  $G$ ,  $\mathcal{A}'$  outputs the same feature vectors as  $\mathcal{A}$  on  $F^T(G)$ . We can assume  $\mathcal{A}$  is represented as  $(C^{(l)}, (A_j^{(l)})_{j=1}^K, A_{aux1}^{(l)}, A_{aux2}^{(l)}, R^{(l)})_{l=1}^L$ , where  $K = T|P_2|$ .
First, by setting feature dimension to be  $d' = T|P| + 3$ . We can construct an  $\mathbb{R}^2$ -TGNN  $\mathcal{A}'$  whose output feature stores all facts in  $F(H(G))$  for any graph  $G$ . Formally speaking,  $\mathcal{A}'$  should satisfy the following condition:
For any primal node  $a$ , its feature outputted by  $\mathcal{A}' \circ F^T$  should store all unary facts of form  $A_i(a), A_i \in T|P_1|$  or  $\text{primal}(a)$  on  $F(H(G))$ . For any non-primal node  $e_{ab}$ , its feature outputted by  $\mathcal{A}' \circ F^T$  should store all binary facts of form  $r_i(a,b), r_i \in T|P_2|$  or  $r_{aux1}(a,b), r_{aux2}(a,b)$  where  $b$  is another node on  $F(H(G))$ .
The  $\mathcal{A}'$  is easy to construct since we have enough dimension size to store different predicates independently, and these facts are completely encoded into the initial features of corresponding timestamp. Let  $(\mathcal{A}_1', \dots, \mathcal{A}_T')$  denote  $\mathcal{A}'$ .
Next, in order to simulate  $\mathcal{A}$ , we need to append some layers to  $\mathcal{A}_T'$ . Let  $L$  denote the layer number of  $\mathcal{A}$ , we need to append  $L$  layers represented as  $(C^{*,(l)}, (A_j^{*,(l)})_{j=1}^{|P_2|}, A_{aux1}^{*,(l)}, A_{aux2}^{*,(l)}, R^{*,(l)})_{l=1}^L$
Since we have enough information encoded in features, we can start to simulate  $\mathcal{A}$ . Since neighbor distribution of primal nodes don't change between  $F^T(G)_T$  and  $F(H(G))$ , it's easy to simulate all messages passed to primal nodes as destinations by  $A_{aux1}^{*,(l)}$ . For messages passed to non-primal node
Figure 6: Hierarchic expressiveness.
$e_{ab}$  as destination, it can be divided into messages from  $a$  and messages from  $e_{ba}$ . The first class of messages is easy to simulate since the aux1 edge between  $e_{ab}$  and  $a$  is the same on  $F^T(G)_T$  and  $F(H(G))$ .
For the second class of messages, since edges of type  $r_i, 1 \leq i \leq T | P_2|$  may be lost in  $F^T(G)_T$ , we have to simulate these messages only by the unchanged edge of type aux2. It can be realized by following construction:
$$
1 \leq l \leq L, A _ {a u x 2} ^ {* (l)} (\mathbf {X}) = \left[ \left[ A _ {j} ^ {\prime , (l)} (\mathbf {X})\right) \right] _ {j = 1} ^ {K}, A _ {a u x 2} ^ {(l)} (\mathbf {X}) ] \tag {60}
$$
where  $K = T|P_2|, A_j'^{(l)}(\mathbf{X}) \coloneqq A_j^{(l)}(\mathbf{X})$  if and only if  $e_{ba}$  has neighbor  $r_j$  on  $F(H(G))$ , otherwise  $A_j'^{(l)}(\mathbf{X}) \coloneqq \mathbf{0}$ . Note that  $\mathbf{X}$  is exactly the feature of  $e_{ba}$ , and we can access the information about its  $r_j$  neighbors from feature since  $\mathcal{A}'$  has stored information about these facts.
In conclusion, we've simulated all messages between neighbors. Furthermore, since node sets on  $F^T(G)_T$  and  $F(H(G))$  are the same, global readout  $R^{(l)}$  is also easy to simulate by  $R^{*,(l)}$ . Finally, using the original combination function  $C^{(l)}$ , we can construct an  $\mathbb{R}^2$ -TGNN on  $F^T$  equivalent to  $\mathcal{A}$  on  $F(H(G))$  for any temporal knowledge graph  $G$ .
J Proof of Theorem 17
Based on Theorem 15, Theorem 16 and Corollary 11.2, in order to prove Theorem 17, it suffices to show the following theorems.
Theorem 31. If time range  $T > 1$ $R^2$ -TGNN  $\subsetneq R^2$ -GNN  $\circ H$ .
Theorem 32. If time range  $T > 1$ $R^2$ -TGNN  $\nsubseteq$  time-and-graph.
Proof. Since a formal proof Theorem 32 relates to too many details in definition of time-and-graph (Please refer to Gao and Ribeiro [2022]) which is not the focus here. We will just a brief proof sketch of Theorem 32: That's because time-and-graph can not capture a chain of information that is continuously scattered in time intervals. Specifically,  $\varphi(x) \coloneqq \exists^{\geq 1}y$ ,  $(r_1^2(x,y) \wedge (\exists^{\geq 1}x,r_1^1(y,x)))$  can't be captured by time-and-graph but  $\varphi(x)$  is in  $\mathbb{R}^2$ -TGNN.
We mainly give a detailid proof of Theorem 31: Since in each timestamp  $t$ ,  $\mathbf{R}^2$ -TGNN only uses a part of predicates in temporalized predicate set  $P' = H(P)$ ,  $\mathbf{R}^2$ -TGNN  $\subseteq \mathbf{R}^2$ -GNN  $\circ H$  trivially holds. To show  $\mathbf{R}^2$ -TGNN is strictly weaker than  $\mathbf{R}^2$ -GNN  $\circ H$ . Consider the following classifier:
Let time range  $T = 2$ , and let  $r$  be a binary predicate in  $P_{2}$ . Note that there are two different predicates  $r^{1}, r^{2}$  in  $P' = H(P)$ . Consider the following temporal graph  $G$  with 5 nodes  $\{1,2,3,4,5\}$ . Its two snapshots  $G_{1}, G_{2}$  are as follows:
$$
G _ {1} = \{r (1, 2), r (4, 5) \}
$$
$$
G _ {2} = \{r (2, 3) \}.
$$
It follows that after transformation  $H$ , the static version of  $G$  is:
$$
H (G) = \{r _ {1} (1, 2), r _ {1} (4, 5), r _ {2} (2, 3) \}.
$$
Consider the logical classifier  $\exists y\Big(r_1(x,y)\land (\exists xr_2(x,y))\Big)$  under  $P^{\prime}$ . It can be captured by some  $\mathbb{R}^2$  -GNN under  $P^{\prime}$ . Therefore,  $\mathbb{R}^2$  -GNN  $\circ H$  can distinguish nodes 1,4.
datasets	φ1	φ2	φ3	φ4
Avg # Nodes	477	477	477	477
Time_range	2	2	2	10
#Unary predicate	2	2	2	3
# Binary predicate(non-temporalized)	1	1	1	3
Avg # Degree (in single timestamp)	3	3	3	5
Avg # positive percentage	50.7	52	25.3	73.3
Table 4: statistical information for synthetic datasets.
datasets	AIFB	MUTAG	Brain-10
# Nodes	8285	23644	5000
Time_Range	\	\	12
# Relation types	45	23	20
# Edges	29043	74227	1761414
# Classes	4	2	10
# Train Nodes	140	272	4500
# Test Nodes	36	68	500
Table 5: statistical information for Real datasets.
hyper-parameter	range
learning rate	0.01
combination	mean/max/add
aggregation/readout	mean/max/add
layer	1,2,3
hidden dimension	10,64,100
Table 6: Hyper-parameters.