SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
C ^ {p, (L ^ {\prime} + 3)} \left(\mathbf {x} _ {0}, \left(\mathbf {x} _ {j}\right) _ {j = 1} ^ {K}, \mathbf {x} _ {a u x 1}, \mathbf {x} _ {a u x 2}, \mathbf {x} _ {g}\right) = C ^ {(L)} \left(\mathbf {x} _ {0} ^ {(1)}, \left(\mathbf {x} _ {a u x 1} ^ {(j)}\right) _ {j = 1} ^ {K}, \mathbf {x} _ {g} ^ {(1)}\right) \tag {25}
$$
$$
R ^ {*}, \left(L ^ {\prime} + 3\right) (\mathbf {X}) = \left[ R ^ {(L)} \left(\{\{\mathbf {x} _ {v} ^ {(1)} | \mathbf {x} _ {v} \in \mathbf {X}, \mathbf {p r i m a l} (v) \}\}\right), \mathbf {0} ^ {d ^ {\prime} - d} \right] \tag {26}
$$
$$
A _ {a u x 1} ^ {*; (L ^ {\prime} + 3)} (\mathbf {X}) = \left[ A _ {1} ^ {(L)} \left(\{\{\mathbf {x} ^ {(1)} | \mathbf {x} \in \mathbf {X} \}\}\right) \dots \dots . A _ {K} ^ {(L)} \left(\{\{\mathbf {x} ^ {(K)} | \mathbf {x} \in \mathbf {X} \}\}\right) \right] \tag {27}
$$
Note that  $C^{(L)}, A_{j}^{(L)}, R^{(L)}$  are all parameters in the last layer of  $\mathcal{A}$  mentioned previously. All other parameters in this layer are set to functions that always output zero vectors. We can see that this layer simulates the work of last layer of  $\mathcal{A}$  as follows:
- For all  $1 \leq j \leq K$ , use the  $j$ -th slice of feature vector  $\mathbf{x}^{(j)}$  to simulate  $A_{j}^{(L)}$  and store results of aggregation function  $A_{j}^{(L)}$  on this slice.
- Global readout trivially emulates what  $R^{(L)}$  does, but only reads features for primal nodes. It can be done since we always have a special dimension in feature to say whether it's a primal node.
- We just simulate what  $C^{(L)}$  does on primal nodes. For  $1 \leq j \leq K$  The type  $r_j$  aggregation result (output of  $A_j^{(L)}$ ) used for input of  $C^{(L)}$  is exactly  $j$ -th slice of return value of  $A_{aux1}^{*,(L' + 3)}$ .
By construction above,  $\mathcal{A}'$  is a desired model that have the same output as  $\mathcal{A}$ .
F Proof of Theorem 10
Theorem 10.  $\mathcal{FOC}_2\subseteq R^2\text{-} GNN\circ F$  on any universal graph class  $\mathcal{G}_u$
Proof. For any  $\mathcal{FOC}_2$  classifier  $\varphi$  under predicate set  $P$ , we want to construct a 0/1-GNN  $\mathcal{A}$  on  $P' = P \cup \{\text{primal}, \text{aux1}, \text{aux2}\}$  equivalent to  $\varphi$  with graph transformation  $F$ .
Recall that  $\mathcal{FOC}_2 = \mathcal{RSFOC}_2$  shown in Lemma 22 and  $0/1$ -GNNs  $\subseteq \mathbb{R}^2$ -GNNs, it suffices to prove that  $0/1$ -GNN  $\circ F$  capture  $\mathcal{RSFOC}_2$ . By Lemma 19, since  $\wedge, \vee, \neg$  are closed under  $0/1$ -GNN it suffices to show that when  $\varphi$  is in the form  $\exists^{\geq n}\big(\varphi_S(x,y) \wedge \varphi'(y)\big), S \subseteq P_2$ , we can capture it.
We prove by induction over quantifier depth  $m$  of  $\varphi$ . Since 0-depth formulas are only about unary predicate that can be extracted from one-hot initial feature, our theorem trivially holds for  $m = 0$ . Now, we assume it also holds for  $m \leq k$ , it suffices to prove the case when  $m = k + 1$ . Then there are two possibilities:
1. When  $S \neq \emptyset$ :
Consider the following logical classifier under  $P^{\prime}$
$$
\widehat {\varphi} _ {S} (x) := \left(\bigwedge_ {r \in S} \exists x r (x, y)\right) \wedge \left(\bigwedge_ {r \notin S} \neg \exists x r (x, y)\right) \tag {28}
$$
$\widehat{\varphi}_S(x)$  restricts that for any  $r \in P'$ ,  $x$  has  $r$ -type neighbor if and only if  $r \in S$ . Review the definition of transformation  $F$ , we know that for any added node  $e_{ab}$ ,  $(F(G), e_{ab}) \models \widehat{\varphi}_S$  if and only if  $(G, a, b) \models \varphi_S(a, b)$ , where  $\varphi_S(x, y)$  is the relation-specification formula defined in Definition 21. That is to say for any  $r_i, 1 \leq i \leq K$ , there is relation  $r_i$  between  $a, b$  if and only if  $i \in S$ .
Now consider the following formula:
$$
\widehat {\varphi} := \exists^ {\geq n} y \left(a u x 1 (x, y) \wedge \widehat {\varphi} _ {S} (y) \wedge \left(\exists x \left(a u x 2 (x, y) \wedge (\exists y (a u x 1 (x, y) \wedge \varphi^ {\prime} (y)))\right)\right)\right) \tag {29}
$$
For any graph  $G$  and its node  $v$ , it's easy to see that  $(G,v) \models \varphi \Leftrightarrow (F(G),v) \models \widehat{\varphi}$ . Therefore we only need to capture  $\widehat{\varphi}$  by 0/1-GNN on every primal node of transformed graphs. By induction, since quantifier depth of  $\varphi'(y)$  is no more than  $k$ , we know  $\varphi'(y)$  is in 0/1-GNN.  $\widehat{\varphi}$  is generated from  $\varphi'(y)$  using rules  $\wedge$  and  $\exists y(r(x,y) \wedge \varphi'(y))$ . By Lemma 19,  $\wedge$  is closed under 0/1-GNN. For  $\exists y(r(x,y) \wedge \varphi'(y))$ , we find that the construction needed is the same as construction for single-element  $S$  on single-edge graph class  $\mathcal{G}_s$  used in Theorem 4. Therefore, since we can manage these two rules, we can also finish the construction for  $\widehat{\varphi}$ , which is equivalent to  $\varphi$  on primal nodes of transformed graph.
2. When  $S = \emptyset$
First, consider the following two logical classifiers:
$$
\bar {\varphi} (x) := \left(\operatorname {p r i m a l} (x) \wedge \varphi^ {\prime} (x)\right) \tag {30}
$$
$\bar{\varphi}$  says a node is primal, and satisfies  $\varphi'(x)$ . Since  $\varphi'(x)$  has quantifier depth no more than  $k$ , and  $\wedge$  is closed under 0/1-GNN. There is a 0/1-GNN  $\mathcal{A}_1$  equivalent to  $\bar{\varphi}$  on transformed graph. Then, consider the following formula.
$$
\tilde {\varphi} (x) := \exists y \left(a u x 2 (x, y) \wedge (\exists x, a u x 1 1 (x, y) \wedge \varphi^ {\prime} (x))\right) \tag {31}
$$
$\tilde{\varphi}(x)$  evaluates on added nodes  $e_{ab}$  on transformed graph,  $e_{ab}$  satisfies it iff  $b$  satisfies  $\varphi'$
Now for a graph  $G$  and its node  $v$ , define  $n_1$  as the number of nodes on  $F(G)$  that satisfies  $\bar{\varphi}$ , and define  $n_2$  as the number of aux1-type neighbors of  $v$  on  $F(G)$  that satisfies  $\tilde{\varphi}$ . Since  $\varphi(x) = \exists^{\geq n} y(\varphi_{\emptyset}(x, y) \land \varphi'(y))$  It's easy to see that  $(G, v) \models \varphi$  if and only if  $n_1 - n_2 \geq n$ .
Formally speaking, for a node set  $S$ , let  $|S|$  denote the number of nodes in  $S$ , we define the following classifier  $c$  such that for any graph  $G$  and its node  $a$ ,  $c(F(G), a) = 1 \Leftrightarrow (G, a) \models \varphi$ .
$$
c (F (G), a) = 1 \Leftrightarrow | \{v | v \in V (F (G)), (F (G), v) \models \bar {\varphi} \} | - | \{v | v \in \mathcal {N} _ {F (G), a u x 1 1} (v), (F (G), v) \models \tilde {\varphi} \} | \geq n \tag {32}
$$
So how to construct a model  $\mathcal{A}$  to capture classifier  $c$ ? First, by induction  $\bar{\varphi}, \tilde{\varphi}$  are all formulas with quantifier depth no more than  $k$  so by previous argument there are 0/1-GNN models  $\bar{\mathcal{A}}, \tilde{\mathcal{A}}$  that capture them respectively. Then we can use feature concatenation technic introduced in Equation (6) to construct a model  $\widehat{A}$  based on  $\bar{\mathcal{A}}, \tilde{\mathcal{A}}$ , such that  $\widehat{A}$  has two-dimensional output, whose first and second dimensions have the same output as  $\bar{\mathcal{A}}, \tilde{\mathcal{A}}$  respectively.
Then, suppose  $\widehat{\mathcal{A}}$  has  $L$  layers. The only thing we need to do is to append a new layer  $L + 1$  to  $\widehat{\mathcal{A}}$ , it has output dimension 1. parameters of it are  $\{C^{(L + 1)},(A_j^{(L + 1)})_{j = 1}^K,A_{aux1}^{(L + 1)},A_{aux2}^{(L + 1)},R^{(L + 1)}\}$  as defined in Equation (5). The parameter settings are as follows:
$\mathbf{R}_{1,1}^{(L + 1)} = 1, \mathbf{A}_{aux1,(1,2)}^{(L + 1)} = -1, \mathbf{b}_1^{(L + 1)} = 1 - n.$  Other parameters are set to 0, where  $\mathbf{A}_{aux1,(1,2)}^{(L + 1)}$  denotes the value in the first row and second column of  $\mathbf{A}_{aux1}^{(L + 1)}$ .
In this construction, we have
$\mathbf{x}_v^{(L + 1)} = \max (0, \min (1, \sum_{u \in V(F(G))} \mathbf{x}_{u,1}^{(L)} - \sum_{u \in \mathcal{N}_{F(G),aux1}(v)} \mathbf{x}_{u,2}^{(L)} - (n - 1))),$  which has exactly the same output as classifier  $c$  defined above in Equation (32). Therefore,  $\mathcal{A}$  is a desired model.
G Proof of Theorem 11
Theorem 11.  $R^2$ -GNN  $\circ F \subseteq \mathcal{FOC}_2$  on any bounded graph class  $\mathcal{G}_b$ .
Before we go into theorem itself, we first introduce Lemma 28 that will be used in following proof.
Lemma 28. Let  $\varphi(x,y)$  denote a  $\mathcal{FOC}_2$  formula with two free variables, for any natural number  $n$ , the following sentence can be captured by  $\mathcal{FOC}_2$ :
There exists no less than  $n$  ordered node pairs  $(a, b)$  such that  $(G, a, b) \models \varphi$ .
Let  $c$  denote the graph classifier such that  $c(G) = 1$  iff  $G$  satisfies the sentence above.
Proof. The basic intuition is to define  $m_i, 1 \leq i < n$  as the number of nodes  $a$ , such that there are exactly  $i$  nodes  $b$  that  $\varphi(a, b)$  is true. Specially, we define  $m_n$  as the number of nodes  $a$ , such that there are at least  $n$  nodes  $b$  that  $\varphi(a, b)$  is true. Since  $\sum_{i=1}^{n} im_i$  exactly counts the number of valid ordered pairs when  $m_n = 0$ , and it guarantees the existence of at least  $n$  valid ordered pairs when  $m_n > 0$ . It's not hard to see that for any graph  $G$ ,  $c(G) = 1 \Leftrightarrow \sum_{i=1}^{n} im_i \geq n$ . Furthermore, fix a valid sequence  $(m_1, \dots, m_n)$  such that  $\sum_{i=1}^{n} im_i \geq n$ , there has to be another sequence  $(k_1, \dots, k_n)$  such that  $n \leq \sum_{i=1}^{n} ik_i \leq 2n$  and  $k_i \leq m_i$  for all  $1 \leq i \leq n$ . Therefore, We can enumerate all possibilities of valid  $(k_1, \dots, k_n)$ , and for each valid  $(k_1, \dots, k_n)$  sequence, we judge whether there are at least  $k_i$  such nodes  $a$  for every  $1 \leq i \leq n$ .
Formally,  $\varphi_{i}(x) := \exists^{[i]}y\varphi(x,y)$  can judge whether a node  $a$  has exactly  $i$  partners  $b$  such that  $\varphi(a,b) = 1$ , where  $\exists^{[i]}y\varphi(x,y)$  denotes "there are exactly  $i$  such nodes  $y$ " which is the abbreviation of formula  $(\exists^{\geq i}y\varphi(x,y)) \land (\neg \exists^{\geq i+1}y\varphi(x,y))$ . The  $\mathcal{FOC}_2$  formula equivalent to our desired sentence  $c$  is as follows:
$$
\bigvee_ {\sum_ {i = 1} ^ {n} n \leq i k _ {i} \leq 2 n} \left(\bigwedge_ {i = 1} ^ {n - 1} \exists^ {\geq k _ {i}} x \left(\exists^ {[ i ]} y \varphi (x, y)\right)\right) \wedge \left(\exists^ {\geq k _ {n}} x \left(\exists^ {\geq n} y \varphi (x, y)\right)\right) \tag {33}
$$
This  $\mathcal{FOC}_2$  formula is equivalent to our desired classifier  $c$ .
With the Lemma 28, we now start to prove Theorem 11.
Proof. By Theorem 5, it follows that  $\mathbf{R}^2$ -GNNs  $\circ F \subseteq \mathcal{FOC}_2 \circ F$ . Therefore it suffices to show  $\mathcal{FOC}_2 \circ F \subseteq \mathcal{FOC}_2$ .
By Lemma 22, it suffices to show  $\mathcal{RSFOC}_2 \circ F \subseteq \mathcal{FOC}_2$ . Since  $\wedge, \vee, \neg$  are common rules. We only need to show for any  $\mathcal{RSFOC}_2$  formula of form  $\varphi(x) := \exists^{\geq n} y(\varphi_S(x,y) \wedge \varphi'(y))$  under transformed predicate set  $P' = P \cup \{aux1, aux2, primal\}$ , there exists an  $\mathcal{FOC}_2$  formula  $\varphi^1$  such that for any graph  $G$  under  $P$  and its node  $v$ ,  $(G,v) \models \varphi^1 \Leftrightarrow (F(G), v) \models \varphi$ .
In order to show this, we consider a stronger result:
For any such formula  $\varphi$ , including the existence of valid  $\varphi^1$ , we claim there also exists an  $\mathcal{FOC}_2$  formula  $\varphi^2$  with two free variables such that the following holds: for any graph  $G$  under  $P$  and its added node  $e_{ab}$  on  $F(G)$ ,  $(G, a, b) \models \varphi^2 \Leftrightarrow (F(G), e_{ab}) \models \varphi$ . Call  $\varphi^1, \varphi^2$  as first/second discriminant of  $\varphi$ .
Now we need to prove the existence of  $\varphi^1$  and  $\varphi^2$ .
We prove by induction over quantifier depth  $m$  of  $\varphi$ . Since we only add a single unary predicate primal in  $P'$ , any  $\varphi(x)$  with quantifier depth 0 can be rewritten as  $(primal(x) \land \varphi^1(x)) \lor (\neg primal(x) \land \varphi^2(x))$ , where  $\varphi^1(x), \varphi^2(x)$  are two formulas that only contain predicates in  $P$ . Therefore,  $\varphi^1$  can be naturally seen as the first discriminant of  $\varphi$ . Moreover, since  $\varphi^2(x)$  always evaluates on non-primal nodes, it is equivalent to  $\bot$  or  $\top$  under  $\neg primal(x)$  constraint. Therefore, the corresponding  $\bot$  or  $\top$  can be seen as the second discriminant, so our theorem trivially holds for  $m = 0$ . Now assume it holds for  $m \leq k$ , we can assume quantifier depth of  $\varphi = \exists^{\geq n} y(\varphi_S(x,y) \land \varphi'(y))$  is  $m = k + 1$ .
Consider the construction rules of transformation  $F$ , for any two primal nodes in  $F(G)$ , there is no relation between them, for a primal node  $a$  and an added node  $e_{ab}$ , there is exactly a single relation of type aux1 between them. For a pair of added nodes  $e_{ab}, e_{ba}$ , there are a bunch of relations from the original graph  $G$  and an additional aux2 relation between them. Therefore, it suffices to only consider three possible kinds of  $S \subseteq P_2 \cup \{aux1, aux2\}$  according to three cases mentioned above. Then, we will construct first/second determinants for each of these three cases. Since  $\varphi'(y)$  has quantifier depth no more than  $k$ , by induction let  $\hat{\varphi}^1, \hat{\varphi}^2$  be first/second discriminants of  $\varphi'$  by induction.
1.  $S = \{\mathbf{a}\mathbf{u}\mathbf{x}\mathbf{1}\} :$
for primal node  $a$ ,  $\varphi(a)$  means the following: there exists at least  $n$  nodes  $b$ , such that there is some relation between  $a, b$  on  $G$  and the added node  $e_{ab}$  on  $F(G)$  satisfies  $\varphi'$ . Therefore, the first determinant of  $\varphi$  can be defined as following:
$$
\varphi^ {1} (x) := \exists^ {\geq n} y, \left(\bigvee_ {r \in P _ {2}} r (x, y)\right) \wedge \hat {\varphi} ^ {2} (x, y) \tag {34}
$$
for added nodes  $e_{ab}$  on  $F(G)$ ,  $\varphi(e_{ab})$  means  $a$  satisfies  $\varphi'$ , so the second determinant of  $\varphi$  is the following:
$$
n = 1: \varphi^ {2} (x, y) := \widehat {\varphi} ^ {1} (x), n > 1: \varphi^ {2} (x, y) := \perp \tag {35}
$$
$$
2. S = \{\mathbf {a u x 2} \} \cup T, T \subseteq P _ {2}, T \neq \emptyset
$$
primal nodes don't have aux2 neighbors, so first determinant is trivially false.
$$
\varphi^ {1} (x) := \perp \tag {36}
$$
For added node  $e_{ab}$ ,  $e_{ab}$  satisfies  $\varphi$  iff there are exactly relations between  $a, b$  of types in  $T$ , and  $e_{ba}$  satisfies  $\varphi'$ . Therefore the second determinant is as follows, where  $\varphi_T(x, y)$  is the relation-specification formula under  $P$  introduced in Definition 21
$$
n = 1: \varphi^ {2} (x, y) := \varphi_ {T} (x, y) \wedge \widehat {\varphi} ^ {2} (y, x), n > 1: \varphi^ {2} (x, y) := \bot \tag {37}
$$
3.  $S = \emptyset$
For a subset  $S \subseteq P_2 \cup \{aux1, aux2\}$ , let  $\varphi_S(x,y)$  denote the relation-specification formula under  $P_2 \cup \{aux1, aux2\}$  defined in Definition 21.
Since we consider on bounded graph class  $\mathcal{G}_b$ , node number is bounded by a natural number  $N$ . For any node  $a$  on  $F(G)$ , let  $m$  denote the number of nodes  $b$  on  $F(G)$  such that  $\varphi'(b) = 1$ , let  $m_0$  denote the number of nodes  $b$  on  $F(G)$  such that  $\varphi'(b) = 1$  and there is a single relation  $aux1$ , between  $(a,b)$  on  $F(G)$ , (That is equivalent to  $\varphi_{\{aux1\}}(a,b) = 1$ ). For any  $T \subseteq P_2$ , let  $m_T$  denote the number of nodes  $b$  on  $F(G)$  such that  $\varphi'(b) = 1$  and  $a,b$  has exactly relations of types in  $T \cup \{aux2\}$  on  $F(G)$ , (That is equivalent to  $\varphi_{T \cup \{aux2\}}(a,b) = 1$ ).
Note that the number of nodes  $b$  on  $F(G)$  such that  $a, b$  don't have any relation, (That is equivalent to  $\varphi_{\emptyset}(a, b) = 1$ ) and  $\varphi'(b) = 1$  equals to  $m - m_0 - \sum_{T \subseteq P_2} m_T$ . Therefore, for any transformed graph  $F(G)$  and its node  $v$ ,  $(F(G), v) \models \varphi \Leftrightarrow m - m_0 - \sum_{T \subseteq P_2} m_T \geq n$ . Since  $|V(G)| \leq N$  for all  $G$  in bounded graph class  $\mathcal{G}_b$ , transformed graph  $F(G)$  has node number no more than  $N^2$ . Therefore, we can enumerate all possibilities of  $m, m_0, m_T \leq N^2, T \subset P_2$  such that the above inequality holds, and for each possibility, we judge whether there exists exactly such number of nodes for each corresponding parameter. Formally speaking,  $\varphi$  can be rewritten as the following form:
$$
\tilde {\varphi} _ {m, m _ {0}} (x) := \left(\exists^ {[ m ]} y \varphi^ {\prime} (y)\right) \wedge \left(\exists^ {[ m _ {0} ]} y \left(\varphi_ {\{a u x 1 \}} (x, y) \wedge \varphi^ {\prime} (y)\right)\right) \tag {38}
$$
$$
\varphi (x) \equiv \bigvee_ {m - m _ {0} - \sum_ {T \subseteq P _ {2}} m _ {T} \geq n, 0 \leq m, m _ {0}, m _ {T} \leq N ^ {2}} \left(\tilde {\varphi} _ {m, m _ {0}} (x) \wedge \left(\bigwedge_ {T \subseteq P _ {2}} \exists^ {[ m _ {T} ]} y, \left(\varphi_ {T \cup \{a u x 2 \}} (x, y) \wedge \varphi^ {\prime} (y)\right)\right)\right) \tag {39}
$$
where  $\exists^{[m]}y$  denotes there are exactly  $m$  nodes  $y$ .
Since first/second determinant can be constructed trivially under combination of  $\wedge, \vee, \neg$ , and we've shown how to construct determinants for formulas of form  $\exists^{\geq n}y(\varphi_S(x,y) \wedge \varphi'(y))$  when  $S = \{aux1\}$  and  $S = \{aux2\} \cup T, T \subseteq P_2$  in the previous two cases. Therefore, in Equation (38) and Equation (39), the only left part is the formula of form  $\exists^{[m]}y\varphi'(y)$ . The only remaining work is to show how to construct first/second determinants for formula in form  $\varphi(x) := \exists^{\geq n}y\varphi'(y)$ .
Let  $m_1$  denote the number of primal nodes  $y$  that satisfies  $\varphi'(y)$  and let  $m_2$  denote the number of non-primal nodes  $y$  that satisfies  $\varphi'(y)$ . It's not hard to see that for any node  $v$  on  $F(G)$ ,  $(F(G), v) \models \varphi \Leftrightarrow m_1 + m_2 \geq n$ . Therefore,  $\varphi(x) = \exists^{\geq n} y \varphi'(y)$  that evaluates on  $F(G)$  is equivalent to the following sentence that evaluates on  $G$ : "There exists two natural numbers  $m_1, m_2$  such that the following conditions hold: 1.  $m_1 + m_2 = n$ . 2. There are at least  $m_1$  nodes  $b$  on  $G$  that satisfies  $\widehat{\varphi}^1$ , (equivalent to  $(F(G), b) \models \varphi'$ ). 3. There are at least  $m_2$  ordered node pairs  $a, b$  on  $G$  such that  $a, b$  has some relation and  $(G, a, b) \models \widehat{\varphi}^2$ , (equivalent to  $(F(G), e_{ab}) \models \varphi'$ .
Formally speaking, rewrite the sentence above as formula under  $P$ , we get the following construction for first/second determinants of  $\varphi$ .
$$
\varphi^ {1} (x) = \varphi^ {2} (x, y) = \bigvee_ {m _ {1} + m _ {2} = n} \left(\left(\exists^ {\geq m _ {1}} y, \widehat {\varphi} ^ {1} (y)\right) \wedge \overline {{\varphi}} _ {m _ {2}}\right) \tag {40}
$$
where  $\overline{\varphi}_{m_2}$  is the  $\mathcal{FOC}_2$  formula that expresses "There exists at least  $m_2$  ordered node pairs  $(a,b)$  such that  $(G,a,b) \models \widehat{\varphi}^2(x,y) \land (\bigvee_{r \in P_2} r(x,y))'$ . We've shown the existence of  $\overline{\varphi_{m_2}}$  in Lemma 28
H Proof of Theorem 15
For a graph  $G$  with  $n$  nodes, let  $\mathbb{H}^V\in \mathbb{R}^{n\times d_v}$  denote node feature matrix, and  $\mathbb{H}^E\in \mathbb{R}^{n\times n\times d_e}$  denote edge feature matrix, where  $\mathbb{H}_{ij}^E$  denote the edge feature vector from  $i$  to  $j$ .
First we need to define the GNN used in their frameworks. Note that for the comparison fairness, we add the global readout to the node feature update as we do in  $\mathbf{R}^2$ -GNNs. It recursively calculates the feature vector  $\mathbb{H}_i^{V,(l)}$  of the node  $i$  at each layer  $1 \leq l \leq L$  as follows:
$$
\mathbb {H} _ {i} ^ {V, (l)} = u ^ {(l)} \left(g ^ {(l)} \left(\left\{\left(\mathbb {H} _ {i} ^ {V, (l - 1)}, \mathbb {H} _ {j} ^ {V, (l - 1)}, \mathbb {H} _ {i j} ^ {E}\right) \mid j \in \mathcal {N} (i) \right\}\right), r ^ {(l)} \left(\left\{\left. \mathbb {H} _ {j} ^ {V, (l - 1)} \mid j \in V \right\}\right)\right)\right) \tag {41}
$$
where  $\mathcal{N}(i)$  denotes the set of all nodes that adjacent to  $i$ , and  $u^{(l)}, g^{(l)}, r^{(l)}$  are learnable functions. Note that here the GNN framework is a little different from the general definition defined in Equation (2). However, this framework is hard to fully implement and many previous works implementing time-and-graph or time-then-graph Gao and Ribeiro [2022] (Li et al. [2019], Seo et al. [2016], Chen et al. [2018], Manessi et al. [2020], Sankar et al. [2018], Rossi et al. [2020b]) don't reach the expressiveness of Equation (41). This definition is more for the theoretical analysis. In contrast, our
definition for GNN in Equation (1) and Equation (2) is more practical since it is fully captured by a bunch of commonly used models such as Schlichtkrull et al. [2018]. For notation simplicity, for a GNN  $\mathcal{A}$ , let  $\mathbb{H}^{V,(L)} = \mathcal{A}(\mathbb{H}^V,\mathbb{H}^E)$  denote the node feature outputted by  $\mathcal{A}$  using  $\mathbb{H}^V,\mathbb{H}^E$  as initial features.
Proposition 29. (Gao and Ribeiro [2022]): time-and-graph  $\subsetneq$  time-then-grahp
The above proposition is from Theorem 1 of Gao and Ribeiro [2022]. Therefore, in order to complete the proof of Theorem 15, we only need to prove  $\mathbf{R}^2$ -TGNN  $\circ F^T =$  time-then-graph.
Let  $G = \{G_1, \ldots, G_T\}$  denote a temporal knowledge graph, and  $\mathbb{A}^t \in \mathbb{R}^{n \times |P_1|}, \mathbb{E}^t \in \mathbb{R}^{n \times n \times |P_2|}, 1 \leq t \leq T$  denote one-hot encoding feature of unary facts and binary facts on timestamp  $t$ , where  $P_1, P_2$  are unary and binary predicate sets.
The updating rule of a time-then-graph model can be generalized as follows:
$$
\forall i \in V, \mathbb {H} _ {i} ^ {V} = \mathbf {R N N} ([ \mathbb {A} _ {i} ^ {1} \dots \dots \mathbb {A} _ {i} ^ {T} ]) \tag {42}
$$
$$
\forall i, j \in V, \mathbb {H} _ {i, j} ^ {E} = \mathbf {R N N} \left(\left[ \mathbb {E} _ {i, j} ^ {1} \dots \dots \mathbb {E} _ {i, j} ^ {T} \right]\right) \tag {43}
$$
$$
\mathbf {X} := \mathcal {A} \left(\mathbb {H} ^ {V}, \mathbb {H} ^ {E}\right) \tag {44}
$$
where  $\mathcal{A}$  is a GNN defined above, RNN is an arbitrary Recurrent Neural Network.  $\mathbf{X} \in \mathbb{R}^{n \times d}$  is the final node feature output of time-then-graph.
First we need to prove time-then-graph  $\subseteq \mathbb{R}^2$ -TGNN $\circ F^T$ . That is, for any time-then-graph model, we want to construct an equivalent  $\mathbb{R}^2$ -TGNN  $\mathcal{A}'$  to capture it on transformed graph. We can use nodes added after transformation to store the edge feature  $\mathbb{H}^E$ , and use primal nodes to store the node feature  $\mathbb{H}^V$ . By simulating RNN through choosing specific functions in  $\mathbb{R}^2$ -TGNN, we can easily construct a  $\mathbb{R}^2$ -TGNN  $\mathcal{A}'$  such that for any node  $i$ , and any node pair  $i,j$  with at least one edge in history,  $\mathbf{x}_i = \mathbb{H}_i^V$  and  $\mathbf{x}_{e_{ij}} = \mathbb{H}_{i,j}^E$  hold, where  $\mathbf{x}_i$  and  $\mathbf{x}_{e_{ij}}$  are features of corresponding primal node  $i$  and added node  $e_{ij}$  outputted by  $\mathcal{A}'$ .
Note that  $\mathcal{A}'$  is a  $\mathbb{R}^2$ -TGNN, it can be represented as  $\mathcal{A}_1'\ldots \mathcal{A}_T'$ , where each  $\mathcal{A}_t', 1 \leq t \leq T$  is a  $\mathbb{R}^2$ -GNN.  $\mathcal{A}'$  has simulated work of RNN, so the remaining work is to simulate  $\mathcal{A}(\mathbb{H}^V, \mathbb{H}^E)$ . We do the simulation over induction on layer number  $L$  of  $\mathcal{A}$ .