SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
f (\mathbf {X}) = \sum_ {r = 1} ^ {R} g ^ {(r)} \left(\mathbf {X} _ {\mathcal {I}}\right) \cdot \bar {g} ^ {(r)} \left(\mathbf {X} _ {\mathcal {I} ^ {\mathrm {c}}}\right), \tag {26}
$$
where  $\mathbf{X} := (\mathbf{x}^{(1)}, \ldots, \mathbf{x}^{(N)})$ ,  $\mathbf{X}_{\mathcal{I}} := (\mathbf{x}^{(i)})_{i \in \mathcal{I}}$ , and  $\mathbf{X}_{\mathcal{I}^c} := (\mathbf{x}^{(j)})_{j \in \mathcal{I}^c}$ . For  $r \in [R]$ , let  $\mathcal{B}(g^{(r)})$  and  $\mathcal{B}(\bar{g}^{(r)})$  be the grid tensors of  $g^{(r)}$  and  $\bar{g}^{(r)}$  over templates  $\mathbf{v}^{(1)}, \ldots, \mathbf{v}^{(M)}$ , respectively. That is,  $\mathcal{B}(g^{(r)})_{d_i: i \in \mathcal{I}} = g^{(r)}((\mathbf{v}^{(d_i)})_{i \in \mathcal{I}})$  and  $\mathcal{B}(\bar{g}^{(r)})_{d_j: j \in \mathcal{I}^c} = \bar{g}^{(r)}((\mathbf{v}^{(d_j)})_{j \in \mathcal{I}^c})$  for all  $d_1, \ldots, d_N \in [M]$ .
16The template vectors of a grid tensor  $\mathcal{B}(f)$  will be clear from context, thus we omit them from the notation.
By Equation (26) we have that for any  $d_1, \ldots, d_N \in [M]$ :
$$
\begin{array}{l} \boldsymbol {\mathcal {B}} (f) _ {d _ {1}, \dots , d _ {N}} = f \left(\mathbf {v} ^ {(d _ {1})}, \dots , \mathbf {v} ^ {(d _ {N})}\right) \\ = \sum_ {r = 1} ^ {R} g ^ {(r)} \big ((\mathbf {v} ^ {(d _ {i})}) _ {i \in \mathcal {I}} \big) \cdot \bar {g} ^ {(r)} \big ((\mathbf {v} ^ {(d _ {j})}) _ {j \in \mathcal {I} ^ {c}} \big) \\ = \sum_ {r = 1} ^ {R} \boldsymbol {\mathcal {B}} \left(g ^ {(r)}\right) _ {d _ {i}: i \in \mathcal {I}} \cdot \boldsymbol {\mathcal {B}} \left(\bar {g} ^ {(r)}\right) _ {d _ {j}: j \in \mathcal {I} ^ {c}}. \\ \end{array}
$$
Denoting by  $\mathbf{u}^{(r)}\in \mathbb{R}^{M^{|T|}}$  and  $\bar{\mathbf{u}}^{(r)}\in \mathbb{R}^{M^{|T^c|}}$  the arrangements of  $\pmb {\mathcal{B}}(g^{(r)})$  and  $\pmb {\mathcal{B}}(\bar{g}^{(r)})$  as vectors, respectively for  $r\in [R]$ , this implies that the matricization of  $\pmb {\mathcal{B}}(f)$  with respect to  $\mathcal{T}$  can be written as:
$$
\llbracket \boldsymbol {\mathcal {B}} (f); \mathcal {I} \rrbracket = \sum_ {r = 1} ^ {R} \mathbf {u} ^ {(r)} \left(\bar {\mathbf {u}} ^ {(r)}\right) ^ {\top}.
$$
We have arrived at a representation of  $\mathbb{[}\pmb {\mathcal{B}}(f);\mathcal{I}\mathbb{]}$  as a sum of  $R$  outer products between two vectors. An outer product of two vectors is a matrix of rank at most one. Consequently, by sub-additivity of rank we conclude:  $\mathrm{rank}[\mathbf{\mathcal{B}}(f);\mathcal{I}]\leq R = \mathrm{sep}(f;\mathcal{I})$
In the context of graph prediction, let  $\mathcal{C}^* \in \operatorname{argmax}_{\mathcal{C} \in \mathcal{S}(\mathcal{I})} \log(\alpha_{\mathcal{C}}) \cdot \rho_{L-1}(\mathcal{C}, \mathcal{V})$ . By Lemma 3, to prove that Equation (8) holds for weights  $\theta$ , it suffices to find template vectors for which  $\log(\operatorname{rank}[[\mathcal{B}(f^{(\theta, \mathcal{G})}); \mathcal{I}]]) \geq \log(\alpha_{\mathcal{C}^*}) \cdot \rho_{L-1}(\mathcal{C}^*, \mathcal{V})$ . Notice that, since the outputs of  $f^{(\theta, \mathcal{G})}$  vary polynomially with the weights  $\theta$ , so do the entries of  $[[[\mathcal{B}(f^{(\theta, \mathcal{G})}); \mathcal{I}]]]$  for any choice of template vectors. Thus, according to Lemma 9, by constructing weights  $\theta$  and template vectors satisfying  $\log(\operatorname{rank}[[\mathcal{B}(f^{(\theta, \mathcal{G})}); \mathcal{I}]]) \geq \log(\alpha_{\mathcal{C}^*}) \cdot \rho_{L-1}(\mathcal{C}^*, \mathcal{V})$ , we may conclude that this is the case for almost all assignments of weights, meaning Equation (8) holds for almost all assignments of weights. In Appendix I.3.1 we construct such weights and template vectors.
In the context of vertex prediction, let  $\mathcal{C}_t^* \in \operatorname{argmax}_{\mathcal{C} \in \mathcal{S}(\mathcal{I})} \log(\alpha_{\mathcal{C}, t}) \cdot \rho_{L-1}(\mathcal{C}, \{t\})$ . Due to arguments analogous to those above, to prove that Equation (9) holds for almost all assignments of weights, we need only find weights  $\theta$  and template vectors satisfying  $\log(\operatorname{rank}[[\mathcal{B}(f^{(\theta, \mathcal{G}, t)}); \mathcal{I}]]) \geq \log(\alpha_{\mathcal{C}_t^*, t}) \cdot \rho_{L-1}(\mathcal{C}_t^*, \{t\})$ . In Appendix I.3.2 we do so.
Lastly, recalling that a finite union of measure zero sets has measure zero as well establishes that Equations (8) and (9) jointly hold for almost all assignments of weights.
I.3.1 Weights and Template Vectors Assignment for Graph Prediction (Proof of Equation (8))
We construct weights  $\theta$  and template vectors satisfying  $\log (\mathrm{rank}[[\pmb {\mathcal{B}}(f^{(\theta ,\mathcal{G})});\mathcal{I}]])\geq \log (\alpha_{\mathcal{C}^*})\cdot$ $\rho_{L - 1}(\mathcal{C}^*,\mathcal{V})$  , where  $\mathcal{C}^*\in \operatorname {argmax}_{\mathcal{C}\in \mathcal{S}(\mathcal{I})}\log (\alpha_{\mathcal{C}})\cdot \rho_{L - 1}(\mathcal{C},\mathcal{V})$
If  $\rho_{L-1}(\mathcal{C}^*,\mathcal{V}) = 0$ , then the claim is trivial since there exist weights and template vectors for which  $\mathbb{B}\big(f^{(\theta ,\mathcal{G})};\mathcal{I}\big]$  is not the zero matrix (e.g. taking all weight matrices to be zero-padded identity matrices and choosing a single template vector holding one in its first entry and zeros elsewhere).
Now, assuming that  $\rho_{L-1}(\mathcal{C}^*,\mathcal{V}) > 0$ , which in particular implies that  $\mathcal{I}\neq \emptyset ,\mathcal{I}\neq \mathcal{V}$ , and  $\mathcal{C}^*\neq \emptyset$  we begin with the case of GNN depth  $L = 1$ , after which we treat the more general  $L\geq 2$  case.
Case of  $L = 1$ : Consider the weights  $\theta = (\mathbf{W}^{(1)}, \mathbf{W}^{(o)})$  given by  $\mathbf{W}^{(1)} := \mathbf{I} \in \mathbb{R}^{D_h \times D_x}$  and  $\mathbf{W}^{(o)} := (1, \ldots, 1) \in \mathbb{R}^{1 \times D_h}$ , where  $\mathbf{I}$  is a zero padded identity matrix, i.e. it holds ones on its diagonal and zeros elsewhere. We choose template vectors  $\mathbf{v}^{(1)}, \ldots, \mathbf{v}^{(D)} \in \mathbb{R}^{D_x}$  such that  $\mathbf{v}^{(m)}$  holds the  $m$ 'th standard basis vector of  $\mathbb{R}^D$  in its first  $D$  coordinates and zeros in the remaining entries, for  $m \in [D]$  (recall  $D := \min\{D_x, D_h\}$ ). Namely, denote by  $\mathbf{e}^{(1)}, \ldots, \mathbf{e}^{(D)} \in \mathbb{R}^D$  the standard basis vectors of  $\mathbb{R}^D$ , i.e.  $\mathbf{e}_d^{(m)} = 1$  if  $d = m$  and  $\mathbf{e}_d^{(m)} = 0$  otherwise for all  $m, d \in [D]$ . We let  $\mathbf{v}_{:D}^{(m)} := \mathbf{e}^{(m)}$  and  $\mathbf{v}_{D+1}^{(m)} := 0$  for all  $m \in [D]$ .
We prove that for this choice of weights and template vectors, for all  $d_{1},\ldots ,d_{|\mathcal{V}|}\in [D]$
$$
f ^ {(\theta , \mathcal {G})} \left(\mathbf {v} ^ {(d _ {1})}, \dots , \mathbf {v} ^ {(d _ {| \mathcal {V} |})}\right) = \left\{ \begin{array}{l l} 1 & , \text {i f} d _ {1} = \dots = d _ {| \mathcal {V} |} \\ 0 & , \text {o t h e r w i s e} \end{array} . \right. \tag {27}
$$
To see it is so, notice that:
$$
f ^ {(\theta , \mathcal {G})} \big (\mathbf {v} ^ {(d _ {1})}, \dots , \mathbf {v} ^ {(d _ {| \mathcal {V} |})} \big) = \mathbf {W} ^ {(o)} \big (\odot_ {i \in \mathcal {V}} \mathbf {h} ^ {(1, i)} \big) = \sum_ {d = 1} ^ {D _ {h}} \prod_ {i \in \mathcal {V}} \mathbf {h} _ {d} ^ {(1, i)},
$$
with  $\mathbf{h}^{(1,i)} = \odot_{j\in \mathcal{N}(i)}\bigl (\mathbf{W}^{(1)}\mathbf{v}^{(d_j)}\bigr) = \odot_{j\in \mathcal{N}(i)}\bigl (\mathbf{I}\mathbf{v}^{(d_j)}\bigr)$  for all  $i\in \mathcal{V}$ . Since  $\mathbf{v}_{:D}^{(d_j)} = \mathbf{e}^{(d_j)}$  for all  $j\in \mathcal{N}(i)$  and  $\mathbf{I}$  is a zero-padded  $D\times D$  identity matrix, it holds that:
$$
f ^ {(\theta , \mathcal {G})} \big (\mathbf {v} ^ {(d _ {1})}, \dots , \mathbf {v} ^ {(d _ {| \mathcal {V} |})} \big) = \sum_ {d = 1} ^ {D} \prod_ {i \in \mathcal {V}, j \in \mathcal {N} (i)} \mathbf {e} _ {d} ^ {(d _ {j})}.
$$
Due to the existence of self-loops (i.e.  $i \in \mathcal{N}(i)$  for all  $i \in \mathcal{V}$ ), for every  $d \in [D]$  the product  $\prod_{i \in \mathcal{V}, j \in \mathcal{N}(i)} \mathbf{e}_d^{(d_j)}$  includes each of  $\mathbf{e}_d^{(d_1)}, \ldots, \mathbf{e}_d^{(d_{|\mathcal{V}|})}$  at least once. Consequently,  $\prod_{i \in \mathcal{V}, j \in \mathcal{N}(i)} \mathbf{e}_d^{(d_j)} = 1$  if  $d_1 = \dots = d_{|\mathcal{V}|} = d$  and  $\prod_{i \in \mathcal{V}, j \in \mathcal{N}(i)} \mathbf{e}_d^{(d_j)} = 0$  otherwise. This implies that  $f^{(\theta, \mathcal{G})}(\mathbf{v}^{(d_1)}, \ldots, \mathbf{v}^{(d_{|\mathcal{V}|})}) = 1$  if  $d_1 = \dots = d_{|\mathcal{V}|}$  and  $f^{(\theta, \mathcal{G})}(\mathbf{v}^{(d_1)}, \ldots, \mathbf{v}^{(d_{|\mathcal{V}|})}) = 0$  otherwise, for all  $d_1, \ldots, d_{|\mathcal{V}|} \in [D]$ .
Equation (27) implies that  $\llbracket \mathcal{B}\big(f^{(\theta ,\mathcal{G})}\big);\mathcal{I}\rrbracket$  has exactly  $D$  non-zero entries, each in a different row and column. Thus,  $\mathrm{rank}[[\mathcal{B}(f^{(\theta ,\mathcal{G})});\mathcal{I}]] = D$ . Recalling that  $\alpha_{\mathcal{C}^*}\coloneqq D^{1 / \rho_0(\mathcal{C}^*,\mathcal{V})}$  for  $L = 1$ , we conclude:
$$
\log \left(\operatorname {r a n k} \llbracket \boldsymbol {\mathcal {B}} \left(f ^ {(\theta , \mathcal {G})}; \mathcal {I} \rrbracket\right)\right) = \log (D) = \log \left(\alpha_ {\mathcal {C} ^ {*}}\right) \cdot \rho_ {0} \left(\mathcal {C} ^ {*}, \mathcal {V}\right).
$$
Case of  $L \geq 2$ : Let  $M := \left( \begin{array}{c} D \\ \rho_{L-1}(\mathcal{C}^*, \mathcal{V}) \end{array} \right) = \binom{D + \rho_{L-1}(\mathcal{C}^*, \mathcal{V}) - 1}{\rho_{L-1}(\mathcal{C}^*, \mathcal{V})}$  be the multiset coefficient of  $D$  and  $\rho_{L-1}(\mathcal{C}^*, \mathcal{V})$  (recall  $D := \min\{D_x, D_h\}$ ). By Lemma 7, there exists  $\mathbf{Z} \in \mathbb{R}_{>0}^{M \times D}$  for which
$$
\operatorname {r a n k} \Big (\odot^ {\rho_ {L - 1} (\mathcal {C} ^ {*}, \mathcal {V})} \big (\mathbf {Z} \mathbf {Z} ^ {\top} \big) \Big) = \left(\left( \begin{array}{c} D \\ \rho_ {L - 1} (\mathcal {C} ^ {*}, \mathcal {V}) \end{array} \right)\right),
$$
with  $\odot^{\rho_{L-1}(\mathcal{C}^*,\mathcal{V})}(\mathbf{Z}\mathbf{Z}^\top)$  standing for the  $\rho_{L-1}(\mathcal{C}^*,\mathcal{V})$ 'th Hadamard power of  $\mathbf{Z}\mathbf{Z}^\top$ . For this  $\mathbf{Z}$ , by Lemma 4 below we know that there exist weights  $\theta$  and template vectors such that  $\llbracket \mathcal{B}(f^{(\theta ,\mathcal{G})});\mathcal{I}\rrbracket$  has an  $M\times M$  sub-matrix of the form  $\mathbf{S}(\odot^{\rho_{L-1}(\mathcal{C}^*,\mathcal{V})}(\mathbf{Z}\mathbf{Z}^\top))\mathbf{Q}$ , where  $\mathbf{S},\mathbf{Q}\in \mathbb{R}^{M\times M}$  are full-rank diagonal matrices. Since the rank of a matrix is at least the rank of any of its sub-matrices:
$$
\begin{array}{l} \operatorname {r a n k} \left(\llbracket \boldsymbol {\mathcal {B}} (f ^ {(\theta , \mathcal {G})}); \mathcal {I} \rrbracket\right) \geq \operatorname {r a n k} \left(\mathbf {S} \left(\odot^ {\rho_ {L - 1} (\mathcal {C} ^ {*}, \mathcal {V})} \left(\mathbf {Z} \mathbf {Z} ^ {\top}\right)\right) \mathbf {Q}\right) \\ = \operatorname {r a n k} \left(\odot^ {\rho_ {L - 1} \left(\mathcal {C} ^ {*}, \mathcal {V}\right)} \left(\mathbf {Z} \mathbf {Z} ^ {\top}\right)\right) \\ = \left(\left( \begin{array}{c} D \\ \rho_ {L - 1} (\mathcal {C} ^ {*}, \mathcal {V}) \end{array} \right)\right), \\ \end{array}
$$
where the second transition stems from  $\mathbf{S}$  and  $\mathbf{Q}$  being full-rank. Applying Lemma 8 to lower bound the multiset coefficient, we have that:
$$
\operatorname {r a n k} \Big (\llbracket \boldsymbol {\mathcal {B}} (f ^ {(\theta , \mathcal {G})}); \mathcal {I} \rrbracket \Big) \geq \left(\binom {D} {\rho_ {L - 1} (\mathcal {C} ^ {*}, \mathcal {V})}\right) \geq \left(\frac {D - 1}{\rho_ {L - 1} (\mathcal {C} ^ {*} , \mathcal {V})} + 1\right) ^ {\rho_ {L - 1} (\mathcal {C} ^ {*}, \mathcal {V})}.
$$
Taking the log of both sides while recalling that  $\alpha_{\mathcal{C}^*} \coloneqq (D - 1) \cdot \rho_{L-1}(\mathcal{C}^*, \mathcal{V})^{-1} + 1$ , we conclude that:
$$
\log (\operatorname {r a n k} \llbracket \boldsymbol {\mathcal {B}} \left(f ^ {(\theta , \mathcal {G})}; \mathcal {I} \rrbracket\right)) \geq \log \left(\alpha_ {\mathcal {C} ^ {*}}\right) \cdot \rho_ {L - 1} \left(\mathcal {C} ^ {*}, \mathcal {V}\right).
$$
Lemma 4. Suppose that the GNN inducing  $f^{(\theta, \mathcal{G})}$  is of depth  $L \geq 2$  and that  $\rho_{L-1}(\mathcal{C}^*, \mathcal{V}) > 0$ . For any  $M \in \mathbb{N}$  and matrix with positive entries  $\mathbf{Z} \in \mathbb{R}_{>0}^{M \times D}$ , there exist weights  $\theta$  and  $M+1$  template vectors  $\mathbf{v}^{(1)}, \ldots, \mathbf{v}^{(M+1)} \in \mathbb{R}^{D_x}$  such that  $[[\pmb{B}(f^{(\theta, \mathcal{G})}); \mathcal{I}]]$  has an  $M \times M$  sub-matrix  $\mathbf{S}(\odot^{\rho_{L-1}}(\mathcal{C}^*, \mathcal{V})(\mathbf{Z}\mathbf{Z}^\top))\mathbf{Q}$ , where  $\mathbf{S}, \mathbf{Q} \in \mathbb{R}^{M \times M}$  are full-rank diagonal matrices and  $(\odot^{\rho_{L-1}}(\mathcal{C}^*, \mathcal{V})(\mathbf{Z}\mathbf{Z}^\top))$  is the  $\rho_{L-1}(\mathcal{C}^*, \mathcal{V})'$ th Hadamard power of  $\mathbf{Z}\mathbf{Z}^\top$ .
Proof. Consider the weights  $\theta = (\mathbf{W}^{(1)},\dots ,\mathbf{W}^{(L)},\mathbf{W}^{(o)})$  given by:
$$
\mathbf {W} ^ {(1)} := \mathbf {I} \in \mathbb {R} ^ {D _ {h} \times D _ {x}},
$$
$$
\mathbf {W} ^ {(2)} := \left( \begin{array}{c c c c} 1 & 1 & \dots & 1 \\ 0 & 0 & \dots & 0 \\ \vdots & \vdots & \dots & \vdots \\ 0 & 0 & \dots & 0 \end{array} \right) \in \mathbb {R} ^ {D _ {h} \times D _ {h}}  ,
$$
$$
\forall l \in \{3, \ldots , L \}: \mathbf {W} ^ {(l)} := \left( \begin{array}{c c c c} 1 & 0 & \dots & 0 \\ 0 & 0 & \dots & 0 \\ \vdots & \vdots & \dots & \vdots \\ 0 & 0 & \dots & 0 \end{array} \right) \in \mathbb {R} ^ {D _ {h} \times D _ {h}}  ,
$$
$$
\mathbf {W} ^ {(o)} := ( \begin{array}{c c c c} 1 & 0 & \dots & 0 \end{array} ) \in \mathbb {R} ^ {1 \times D _ {h}}  ,
$$
where  $\mathbf{I}$  is a zero padded identity matrix, i.e. it holds ones on its diagonal and zeros elsewhere. We define the templates  $\mathbf{v}^{(1)},\ldots ,\mathbf{v}^{(M)}\in \mathbb{R}^{D_x}$  to be the vectors holding the respective rows of  $\mathbf{Z}$  in their first  $D$  coordinates and zeros in the remaining entries (recall  $D\coloneqq \min \{D_x,D_h\}$ ). That is, denoting the rows of  $\mathbf{Z}$  by  $\mathbf{z}^{(1)},\dots,\mathbf{z}^{(M)}\in \mathbb{R}_{>0}^{D}$ , we let  $\mathbf{v}_{:D}^{(m)}\coloneqq \mathbf{z}^{(m)}$  and  $\mathbf{v}_{D + 1}^{(m)}\coloneqq 0$  for all  $m\in [M]$ . We set all entries of the last template vector to one, i.e.  $\mathbf{v}^{(M + 1)}\coloneqq (1,\dots ,1)\in \mathbb{R}^{D_x}$ .
Since  $\mathcal{C}^* \in S(\mathcal{I})$ , i.e. it is an admissible subset of  $\mathcal{C}_{\mathcal{I}}$  (Definition 4), there exist  $\mathcal{I}' \subseteq \mathcal{I}, \mathcal{I}' \subseteq \mathcal{I}^c$  with no repeating shared neighbors (Definition 3) such that  $\mathcal{C}^* = \mathcal{N}(\mathcal{I}') \cap \mathcal{N}(\mathcal{I}')$ . Notice that  $\mathcal{I}'$  and  $\mathcal{I}'$  are non-empty as  $\mathcal{C}^* \neq \emptyset$  (this is implied by  $\rho_{L-1}(\mathcal{C}^*, \mathcal{V}) > 0$ ). We focus on the  $M \times M$  sub-matrix of  $\llbracket \pmb{B}(f^{(\theta, \mathcal{G})}; \mathcal{I}\rrbracket$  that includes only rows and columns corresponding to evaluations of  $f^{(\theta, \mathcal{G})}$  where all variables indexed by  $\mathcal{I}'$  are assigned the same template vector from  $\mathbf{v}^{(1)}, \ldots, \mathbf{v}^{(M)}$ , all variables indexed by  $\mathcal{I}'$  are assigned the same template vector from  $\mathbf{v}^{(1)}, \ldots, \mathbf{v}^{(M)}$ , and all remaining variables are assigned the all-ones template vector  $\mathbf{v}^{(M+1)}$ . Denoting this sub-matrix by  $\mathbf{U} \in \mathbb{R}^{M \times M}$ , it therefore upholds:
$$
\mathbf {U} _ {m, n} = f ^ {(\theta , \mathcal {G})} \Big (\big (\mathbf {x} ^ {(i)} \leftarrow \mathbf {v} ^ {(m)} \big) _ {i \in \mathcal {I} ^ {\prime}}, \big (\mathbf {x} ^ {(j)} \leftarrow \mathbf {v} ^ {(n)} \big) _ {j \in \mathcal {I} ^ {\prime}}, \big (\mathbf {x} ^ {(k)} \leftarrow \mathbf {v} ^ {(M + 1)} \big) _ {k \in \mathcal {V} \backslash (\mathcal {I} ^ {\prime} \cup \mathcal {I} ^ {\prime})} \Big),
$$
for all  $m,n\in [M]$ , where we use  $(\mathbf{x}^{(i)}\gets \mathbf{v}^{(m)})_{i\in \mathcal{I}'}$  to denote that input variables indexed by  $\mathcal{I}'$  are assigned the value  $\mathbf{v}^{(m)}$ . To show that  $\mathbf{U}$  obeys the form  $\mathbf{S}(\odot^{\rho_{L - 1}(\mathcal{C}^*,\mathcal{V})}(\mathbf{ZZ}^\top))\mathbf{Q}$  for full-rank diagonal  $\mathbf{S},\mathbf{Q}\in \mathbb{R}^{M\times M}$ , we prove there exist  $\phi ,\psi :\mathbb{R}^{D_x}\to \mathbb{R}_{>0}$  such that  $\mathbf{U}_{m,n} = \phi (\mathbf{v}^{(m)})\langle \mathbf{z}^{(m)},\mathbf{z}^{(n)}\rangle^{\rho_{L - 1}(\mathcal{C}^*,\mathcal{V})}\psi (\mathbf{v}^{(n)})$  for all  $m,n\in [M]$ . Indeed, defining  $\mathbf{S}$  to hold  $\phi (\mathbf{v}^{(1)}),\ldots ,\phi (\mathbf{v}^{(M)})$  on its diagonal and  $\mathbf{Q}$  to hold  $\psi (\mathbf{v}^{(1)}),\ldots ,\psi (\mathbf{v}^{(M)})$  on its diagonal, we have that  $\mathbf{U} = \mathbf{S}(\odot^{\rho_{L - 1}(\mathcal{C}^*,\mathcal{V})}(\mathbf{ZZ}^\top))\mathbf{Q}$ . Since  $\mathbf{S}$  and  $\mathbf{Q}$  are clearly full-rank (diagonal matrices with non-zero entries on their diagonal), the proof concludes.
For  $m, n \in [M]$ , let  $\mathbf{h}^{(l,i)} \in \mathbb{R}^{D_h}$  be the hidden embedding for  $i \in \mathcal{V}$  at layer  $l \in [L]$  of the GNN inducing  $f^{(\theta, \mathcal{G})}$ , over the following assignment to its input variables (i.e. vertex features):
$$
\left(\mathbf {x} ^ {(i)} \gets \mathbf {v} ^ {(m)}\right) _ {i \in \mathcal {I} ^ {\prime}}, \left(\mathbf {x} ^ {(j)} \gets \mathbf {v} ^ {(n)}\right) _ {j \in \mathcal {J} ^ {\prime}}, \left(\mathbf {x} ^ {(k)} \gets \mathbf {v} ^ {(M + 1)}\right) _ {k \in \mathcal {V} \backslash (\mathcal {I} ^ {\prime} \cup \mathcal {J} ^ {\prime})}.
$$
Invoking Lemma 10 with  $\mathbf{v}^{(m)},\mathbf{v}^{(n)},\mathcal{I}'$  , and  $\mathcal{J}'$  , for all  $i\in \mathcal{V}$  it holds that:
$$
\mathbf {h} _ {1} ^ {(L, i)} = \phi^ {(L, i)} \left(\mathbf {v} ^ {(m)}\right) \left\langle \mathbf {z} ^ {(m)}, \mathbf {z} ^ {(n)} \right\rangle^ {\rho_ {L - 1} \left(\mathcal {C} ^ {*}, \{i \}\right)} \psi^ {(L, i)} \left(\mathbf {v} ^ {(n)}\right), \quad \forall d \in \{2, \dots , D _ {h} \}: \mathbf {h} _ {d} ^ {(L, i)} = 0,
$$
for some  $\phi^{(L,i)},\psi^{(L,i)}:\mathbb{R}^{D_x}\to \mathbb{R}_{>0}$ . Since
$$
\begin{array}{l} \mathbf {U} _ {m, n} = f ^ {(\theta , \mathcal {G})} \left(\left(\mathbf {x} ^ {(i)} \leftarrow \mathbf {v} ^ {(m)}\right) _ {i \in \mathcal {I} ^ {\prime}}, \left(\mathbf {x} ^ {(j)} \leftarrow \mathbf {v} ^ {(n)}\right) _ {j \in \mathcal {J} ^ {\prime}}, \left(\mathbf {x} ^ {(k)} \leftarrow \mathbf {v} ^ {(M + 1)}\right) _ {k \in \mathcal {V} \backslash (\mathcal {I} ^ {\prime} \cup \mathcal {J} ^ {\prime})}\right) \\ = \mathbf {W} ^ {(o)} \left(\odot_ {i \in \mathcal {V}} \mathbf {h} ^ {(L, i)}\right) \\ \end{array}
$$
and  $\mathbf{W}^{(o)} = (1,0,\dots ,0)$  , this implies that:
$$
\begin{array}{l} \mathbf {U} _ {m, n} = \prod_ {i \in \mathcal {V}} \mathbf {h} _ {1} ^ {(L, i)} \\ = \prod_ {i \in \mathcal {V}} \phi^ {(L, i)} (\mathbf {v} ^ {(m)}) \left\langle \mathbf {z} ^ {(m)}, \mathbf {z} ^ {(n)} \right\rangle^ {\rho_ {L - 1} \left(\mathcal {C} ^ {*}, \{i \}\right)} \psi^ {(L, i)} (\mathbf {v} ^ {(n)}). \\ \end{array}
$$
Rearranging the last term leads to:
$$
\mathbf {U} _ {m, n} = \left(\prod_ {i \in \mathcal {V}} \phi^ {(L, i)} \big (\mathbf {v} ^ {(m)} \big)\right) \cdot \left\langle \mathbf {z} ^ {(m)}, \mathbf {z} ^ {(n)} \right\rangle^ {\sum_ {i \in \mathcal {V}} \rho_ {L - 1} \left(\mathcal {C} ^ {*}, \{i \}\right)} \cdot \left(\prod_ {i \in \mathcal {V}} \psi^ {(L, i)} \big (\mathbf {v} ^ {(n)} \big)\right).
$$
Let  $\phi :\mathbf{v}\mapsto \prod_{i\in \mathcal{V}}\phi^{(L,i)}(\mathbf{v})$  and  $\psi :\mathbf{v}\mapsto \prod_{i\in \mathcal{V}}\psi^{(L,i)}(\mathbf{v})$ . Noticing that their range is indeed  $\mathbb{R}_{>0}$  and that  $\sum_{i\in \mathcal{V}}\rho_{L - 1}(\mathcal{C}^*,\{i\}) = \rho_{L - 1}(\mathcal{C}^*,\mathcal{V})$  yields the sought-after expression for  $\mathbf{U}_{m,n}$ :
$$
\mathbf {U} _ {m, n} = \phi \left(\mathbf {v} ^ {(m)}\right) \left\langle \mathbf {z} ^ {(m)}, \mathbf {z} ^ {(n)} \right\rangle^ {\rho_ {L - 1} \left(\mathcal {C} ^ {*}, \mathcal {V}\right)} \psi \left(\mathbf {v} ^ {(n)}\right).
$$
I.3.2 Weights and Template Vectors Assignment for Vertex Prediction (Proof of Equation (9))
This part of the proof follows a line similar to that of Appendix I.3.1, with differences stemming from the distinction between the operation of a GNN over graph and vertex prediction. Namely, we construct weights  $\theta$  and template vectors satisfying  $\log (\mathrm{rank}[[\mathcal{B}(f^{(\theta ,\mathcal{G},t)};\mathcal{I}]])\geq \log (\alpha_{\mathcal{C}_t^*},t)\cdot \rho_{L - 1}(\mathcal{C}_t^*,\{t\})$ , where  $\mathcal{C}_t^*\in \operatorname {argmax}_{\mathcal{C}\in \mathcal{S}(\mathcal{I})}\log (\alpha_{\mathcal{C},t})\cdot \rho_{L - 1}(\mathcal{C},\{t\})$ .
If  $\rho_{L-1}(\mathcal{C}_t^*, \{t\}) = 0$ , then the claim is trivial since there exist weights and template vectors for which  $\mathbb{B}(f^{(\theta, \mathcal{G}, t)}; \mathcal{I})$  is not the zero matrix (e.g. taking all weight matrices to be zero-padded identity matrices and choosing a single template vector holding one in its first entry and zeros elsewhere).
Now, assuming that  $\rho_{L - 1}(\mathcal{C}_t^*,\{t\}) > 0$ , which in particular implies that  $\mathcal{I}\neq \emptyset ,\mathcal{I}\neq \mathcal{V}$ , and  $\mathcal{C}_t^*\neq \emptyset$  we begin with the case of GNN depth  $L = 1$ , after which we treat the more general  $L\geq 2$  case.
Case of  $L = 1$ : Consider the weights  $\theta = (\mathbf{W}^{(1)}, \mathbf{W}^{(o)})$  given by  $\mathbf{W}^{(1)} := \mathbf{I} \in \mathbb{R}^{D_h \times D_x}$  and  $\mathbf{W}^{(o)} := (1, \ldots, 1) \in \mathbb{R}^{1 \times D_h}$ , where  $\mathbf{I}$  is a zero-padded identity matrix, i.e. it holds ones on its diagonal and zeros elsewhere. We choose template vectors  $\mathbf{v}^{(1)}, \ldots, \mathbf{v}^{(D)} \in \mathbb{R}^{D_x}$  such that  $\mathbf{v}^{(m)}$  holds the  $m$ 'th standard basis vector of  $\mathbb{R}^D$  in its first  $D$  coordinates and zeros in the remaining entries, for  $m \in [D]$  (recall  $D := \min\{D_x, D_h\}$ ). Namely, denote by  $\mathbf{e}^{(1)}, \ldots, \mathbf{e}^{(D)} \in \mathbb{R}^D$  the standard basis vectors of  $\mathbb{R}^D$ , i.e.  $\mathbf{e}_d^{(m)} = 1$  if  $d = m$  and  $\mathbf{e}_d^{(m)} = 0$  otherwise for all  $m, d \in [D]$ . We let  $\mathbf{v}_{:D}^{(m)} := \mathbf{e}^{(m)}$  and  $\mathbf{v}_{D+1}^{(m)} := 0$  for all  $m \in [D]$ .
$$
f ^ {(\theta , \mathcal {G}, t)} \left(\mathbf {v} ^ {(d _ {1})}, \dots , \mathbf {v} ^ {(d _ {| \mathcal {V} |})}\right) = \left\{ \begin{array}{l l} 1 & , \text {i f} d _ {j} = d _ {j ^ {\prime}} \text {f o r a l l} j, j ^ {\prime} \in \mathcal {N} (t) \\ 0 & , \text {o t h e r w i s e} \end{array} . \right. \tag {28}
$$
$$
f ^ {(\theta , \mathcal {G}, t)} \big (\mathbf {v} ^ {(d _ {1})}, \dots , \mathbf {v} ^ {(d _ {| \mathcal {V} |})} \big) = \mathbf {W} ^ {(o)} \mathbf {h} ^ {(1, t)} = \sum_ {d = 1} ^ {D _ {h}} \mathbf {h} _ {d} ^ {(1, t)},
$$
with  $\mathbf{h}^{(1,t)} = \odot_{j\in \mathcal{N}(t)}(\mathbf{W}^{(1)}\mathbf{v}^{(d_j)}) = \odot_{j\in \mathcal{N}(t)}(\mathbf{I}\mathbf{v}^{(d_j)})$ . Since  $\mathbf{v}_{:D}^{(d_j)} = \mathbf{e}^{(d_j)}$  for all  $j\in \mathcal{N}(t)$  and  $\mathbf{I}$  is a zero-padded  $D\times D$  identity matrix, it holds that:
$$
f ^ {(\theta , \mathcal {G}, t)} \big (\mathbf {v} ^ {(d _ {1})}, \dots , \mathbf {v} ^ {(d _ {| \mathcal {V} |})} \big) = \sum_ {d = 1} ^ {D} \prod_ {j \in \mathcal {N} (t)} \mathbf {e} _ {d} ^ {(d _ {j})}.
$$