SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For every  $d \in [D]$  we have that  $\prod_{j \in \mathcal{N}(t)} \mathbf{e}_d^{(d_j)} = 1$  if  $d_j = d$  for all  $j \in \mathcal{N}(t)$  and  $\prod_{j \in \mathcal{N}(t)} \mathbf{e}_d^{(d_j)} = 0$  otherwise. This implies that  $f^{(\theta, \mathcal{G}, t)}(\mathbf{v}^{(d_1)}, \ldots, \mathbf{v}^{(d_{|\mathcal{V}|})}) = 1$  if  $d_j = d_{j'}$  for all  $j, j' \in \mathcal{N}(t)$  and  $f^{(\theta, \mathcal{G}, t)}(\mathbf{v}^{(d_1)}, \ldots, \mathbf{v}^{(d_{|\mathcal{V}|})}) = 0$  otherwise, for all  $d_1, \ldots, d_{|\mathcal{V}|} \in [D]$ .
Equation (28) implies that  $\llbracket \mathcal{B}(f^{(\theta ,\mathcal{G},t)});\mathcal{I}\rrbracket$  has a sub-matrix of rank  $D$ . Specifically, such a sub-matrix can be obtained by examining all rows and columns of  $\llbracket \mathcal{B}(f^{(\theta ,\mathcal{G},t)});\mathcal{I}\rrbracket$  corresponding to some fixed indices  $(d_i\in [D])_{i\in \mathcal{V}\setminus \mathcal{N}(t)}$  for the vertices that are not neighbors of  $t$ . Thus,  $\mathrm{rank}[\mathcal{B}(f^{(\theta ,\mathcal{G},t)});\mathcal{I}]\geq D$ . Notice that necessarily  $\rho_0(\mathcal{C}_t^*,\{t\}) = 1$ , as it is not zero and there can only be one length zero walk to  $t$  (the trivial walk that starts and ends at  $t$ ). Recalling that  $\alpha_{\mathcal{C}_t^* ,t}\coloneqq D$  for  $L = 1$ , we therefore conclude:
$$
\log \Big (\operatorname {r a n k} \llbracket \boldsymbol {\mathcal {B}} \Big (f ^ {(\theta , \mathcal {G}, t)} \Big); \mathcal {I} \rrbracket \Big) \geq \log (D) = \log \left(\alpha_ {\mathcal {C} _ {t} ^ {*}, t}\right) \cdot \rho_ {0} \left(\mathcal {C} _ {t} ^ {*}, \{t \}\right).
$$
Case of  $L \geq 2$ : Let  $M := \left(\big( \begin{array}{c} D \\ \rho_{L-1}(\mathcal{C}_t^*, \{t\}) \end{array} \right)\right) = \binom{D + \rho_{L-1}(\mathcal{C}_t^*, \{t\}) - 1}{\rho_{L-1}(\mathcal{C}_t^*, \{t\})}$  be the multiset coefficient of  $D$  and  $\rho_{L-1}(\mathcal{C}_t^*, \{t\})$  (recall  $D := \min\{D_x, D_h\}$ ). By Lemma 7, there exists  $\mathbf{Z} \in \mathbb{R}_{>0}^{M \times D}$  for which
$$
\operatorname {r a n k} \Big (\odot^ {\rho_ {L - 1} (\mathcal {C} _ {t} ^ {*}, \{t \})} \left(\mathbf {Z} \mathbf {Z} ^ {\top}\right) \Big) = \left(\binom {D} {\rho_ {L - 1} (\mathcal {C} _ {t} ^ {*}, \{t \})}\right),
$$
with  $\odot^{\rho_{L-1}}(\mathcal{C}_t^*,\{t\})(\mathbf{Z}\mathbf{Z}^\top)$  standing for the  $\rho_{L-1}(\mathcal{C}_t^*,\{t\})'$ th Hadamard power of  $\mathbf{Z}\mathbf{Z}^\top$ . For this  $\mathbf{Z}$ , by Lemma 5 below we know that there exist weights  $\theta$  and template vectors such that  $\llbracket \mathcal{B}(f^{(\theta ,\mathcal{G},t)});\mathcal{I}\rrbracket$  has an  $M\times M$  sub-matrix of the form  $\mathbf{S}(\odot^{\rho_{L-1}}(\mathcal{C}_t^*,\{t\})(\mathbf{Z}\mathbf{Z}^\top))\mathbf{Q}$ , where  $\mathbf{S},\mathbf{Q}\in \mathbb{R}^{M\times M}$  are full-rank diagonal matrices. Since the rank of a matrix is at least the rank of any of its sub-matrices:
$$
\begin{array}{l} \operatorname {r a n k} \left(\llbracket \boldsymbol {\mathcal {B}} (f ^ {(\theta , \mathcal {G}, t)}); \mathcal {I} \rrbracket\right) \geq \operatorname {r a n k} \left(\mathbf {S} \left(\odot^ {\rho_ {L - 1} \left(\mathcal {C} _ {t} ^ {*}, \{t \}\right)} (\mathbf {Z} \mathbf {Z} ^ {\top})\right) \mathbf {Q}\right) \\ = \operatorname {r a n k} \left(\odot^ {\rho_ {L - 1} \left(\mathcal {C} _ {t} ^ {*}, \{t \}\right)} \left(\mathbf {Z} \mathbf {Z} ^ {\top}\right)\right) \\ = \left(\binom {D} {\rho_ {L - 1} (\mathcal {C} _ {t} ^ {*}, \{t \})}\right), \\ \end{array}
$$
where the second transition is due to  $\mathbf{S}$  and  $\mathbf{Q}$  being full-rank. Applying Lemma 8 to lower bound the multiset coefficient, we have that:
$$
\operatorname {r a n k} \Big (\llbracket \boldsymbol {\mathcal {B}} (f ^ {(\theta , \mathcal {G}, t)}); \mathcal {I} \rrbracket \Big) \geq \left(\binom {D} {\rho_ {L - 1} (\mathcal {C} _ {t} ^ {*}, \{t \})}\right) \geq \left(\frac {D - 1}{\rho_ {L - 1} (\mathcal {C} _ {t} ^ {*} , \{t \})} + 1\right) ^ {\rho_ {L - 1} (\mathcal {C} _ {t} ^ {*}, \{t \})}.
$$
Taking the log of both sides while recalling that  $\alpha_{\mathcal{C}_t^*, t} \coloneqq (D - 1) \cdot \rho_{L-1}(\mathcal{C}_t^*, \{t\})^{-1} + 1$ , we conclude that:
$$
\log (\operatorname {r a n k} \llbracket \boldsymbol {\mathcal {B}} \left(f ^ {(\theta , \mathcal {G}, t)}\right); \mathcal {I} \rrbracket) \geq \log \left(\alpha_ {\mathcal {C} _ {t} ^ {*}, t}\right) \cdot \rho_ {L - 1} \left(\mathcal {C} _ {t} ^ {*}, \{t \}\right).
$$
Lemma 5. Suppose that the GNN inducing  $f^{(\theta, \mathcal{G}, t)}$  is of depth  $L \geq 2$  and that  $\rho_{L-1}(\mathcal{C}_t^*, \{t\}) > 0$ . For any  $M \in \mathbb{N}$  and matrix with positive entries  $\mathbf{Z} \in \mathbb{R}_{>0}^{M \times D}$ , there exist weights  $\theta$  and  $M+1$  template vectors  $\mathbf{v}^{(1)}, \ldots, \mathbf{v}^{(M+1)} \in \mathbb{R}^{D_x}$  such that  $[[\pmb{\mathcal{B}}(f^{(\theta, \mathcal{G}, t)}); \mathcal{I}]]$  has an  $M \times M$  sub-matrix  $\mathbf{S}(\odot^{\rho_{L-1}}(\mathcal{C}_t^*, \{t\})(\mathbf{Z}\mathbf{Z}^\top))\mathbf{Q}$ , where  $\mathbf{S}, \mathbf{Q} \in \mathbb{R}^{M \times M}$  are full-rank diagonal matrices and  $(\odot^{\rho_{L-1}}(\mathcal{C}_t^*, \{t\})(\mathbf{Z}\mathbf{Z}^\top))$  is the  $\rho_{L-1}(\mathcal{C}_t^*, \{t\})'$ 'th Hadamard power of  $\mathbf{Z}\mathbf{Z}^\top$ .
Proof. Consider the weights  $\theta = (\mathbf{W}^{(1)},\dots ,\mathbf{W}^{(L)},\mathbf{W}^{(o)})$  defined by:
$$
\mathbf {W} ^ {(o)} := \left( \begin{array}{c c c c} 1 & 0 & \dots & 0 \end{array} \right) \in \mathbb {R} ^ {1 \times D _ {h}}  ,
$$
where  $\mathbf{I}$  is a zero padded identity matrix, i.e. it holds ones on its diagonal and zeros elsewhere. We let the templates  $\mathbf{v}^{(1)},\ldots ,\mathbf{v}^{(M)}\in \mathbb{R}^{D_x}$  be the vectors holding the respective rows of  $\mathbf{Z}$  in their first  $D$  coordinates and zeros in the remaining entries (recall  $D\coloneqq \min \{D_x,D_h\}$ ). That is, denoting the rows of  $\mathbf{Z}$  by  $\mathbf{z}^{(1)},\dots,\mathbf{z}^{(M)}\in \mathbb{R}_{>0}^{D}$ , we let  $\mathbf{v}_{:D}^{(m)}\coloneqq \mathbf{z}^{(m)}$  and  $\mathbf{v}_{D + 1}^{(m)}\coloneqq 0$  for all  $m\in [M]$ . We set all entries of the last template vector to one, i.e.  $\mathbf{v}^{(M + 1)}\coloneqq (1,\dots ,1)\in \mathbb{R}^{D_x}$ .
Since  $\mathcal{C}_t^* \in S(\mathcal{I})$ , i.e. it is an admissible subset of  $\mathcal{C}_{\mathcal{I}}$  (Definition 4), there exist  $\mathcal{I}' \subseteq \mathcal{I}$ ,  $\mathcal{J}' \subseteq \mathcal{I}^c$  with no repeating shared neighbors (Definition 3) such that  $\mathcal{C}_t^* = \mathcal{N}(\mathcal{I}') \cap \mathcal{N}(\mathcal{J}')$ . Notice that  $\mathcal{I}'$  and  $\mathcal{J}'$  are non-empty as  $\mathcal{C}_t^* \neq \emptyset$  (this is implied by  $\rho_{L-1}(\mathcal{C}_t^*, \{t\}) > 0$ ). We focus on the  $M \times M$
sub-matrix of  $\llbracket \mathcal{B}(f^{(\theta ,\mathcal{G},t)};\mathcal{I}\rrbracket$  that includes only rows and columns corresponding to evaluations of  $f^{(\theta ,\mathcal{G},t)}$  where all variables indexed by  $\mathcal{I}'$  are assigned the same template vector from  $\mathbf{v}^{(1)},\ldots ,\mathbf{v}^{(M)}$ , all variables indexed by  $\mathcal{J}'$  are assigned the same template vector from  $\mathbf{v}^{(1)},\ldots ,\mathbf{v}^{(M)}$ , and all remaining variables are assigned the all-ones template vector  $\mathbf{v}^{(M + 1)}$ . Denoting this sub-matrix by  $\mathbf{U}\in \mathbb{R}^{M\times M}$ , it therefore upholds:
$$
\mathbf {U} _ {m, n} = f ^ {(\theta , \mathcal {G}, t)} \Big (\big (\mathbf {x} ^ {(i)} \gets \mathbf {v} ^ {(m)} \big) _ {i \in \mathcal {I} ^ {\prime}}, \big (\mathbf {x} ^ {(j)} \gets \mathbf {v} ^ {(n)} \big) _ {j \in \mathcal {J} ^ {\prime}}, \big (\mathbf {x} ^ {(k)} \gets \mathbf {v} ^ {(M + 1)} \big) _ {k \in \mathcal {V} \setminus (\mathcal {I} ^ {\prime} \cup \mathcal {J} ^ {\prime})} \Big),
$$
for all  $m,n\in [M]$ , where we use  $(\mathbf{x}^{(i)}\gets \mathbf{v}^{(m)})_{i\in \mathcal{I}'}$  to denote that input variables indexed by  $\mathcal{I}'$  are assigned the value  $\mathbf{v}^{(m)}$ . To show that  $\mathbf{U}$  obeys the form  $\mathbf{S}(\odot^{\rho_{L - 1}}(\mathcal{C}_t^*,\{t\})(\mathbf{ZZ}^\top))\mathbf{Q}$  for full-rank diagonal  $\mathbf{S},\mathbf{Q}\in \mathbb{R}^{M\times M}$ , we prove there exist  $\phi ,\psi :\mathbb{R}^{D_x}\to \mathbb{R}_{>0}$  such that  $\mathbf{U}_{m,n} = \phi (\mathbf{v}^{(m)})\langle \mathbf{z}^{(m)},\mathbf{z}^{(n)}\rangle^{\rho_{L - 1}(\mathcal{C}_t^*,\{t\})}\psi (\mathbf{v}^{(n)})$  for all  $m,n\in [M]$ . Indeed, defining  $\mathbf{S}$  to hold  $\phi (\mathbf{v}^{(1)}),\ldots ,\phi (\mathbf{v}^{(M)})$  on its diagonal and  $\mathbf{Q}$  to hold  $\psi (\mathbf{v}^{(1)}),\ldots ,\psi (\mathbf{v}^{(M)})$  on its diagonal, we have that  $\mathbf{U} = \mathbf{S}(\odot^{\rho_{L - 1}}(\mathcal{C}_t^*,\{t\})(\mathbf{ZZ}^\top))\mathbf{Q}$ . Since  $\mathbf{S}$  and  $\mathbf{Q}$  are clearly full-rank (diagonal matrices with non-zero entries on their diagonal), the proof concludes.
For  $m, n \in [M]$ , let  $\mathbf{h}^{(l,i)} \in \mathbb{R}^{D_h}$  be the hidden embedding for  $i \in \mathcal{V}$  at layer  $l \in [L]$  of the GNN inducing  $f^{(\theta, \mathcal{G}, t)}$ , over the following assignment to its input variables (i.e. vertex features):
Invoking Lemma 10 with  $\mathbf{v}^{(m)},\mathbf{v}^{(n)},\mathcal{I}'$  , and  $\mathcal{J}'$  , it holds that:
$$
\mathbf {h} _ {1} ^ {(L, t)} = \phi^ {(L, t)} \left(\mathbf {v} ^ {(m)}\right) \left\langle \mathbf {z} ^ {(m)}, \mathbf {z} ^ {(n)} \right\rangle^ {\rho_ {L - 1} \left(\mathcal {C} _ {t} ^ {*}, \{t \}\right)} \psi^ {(L, t)} \left(\mathbf {v} ^ {(n)}\right), \quad \forall d \in \{2, \dots , D _ {h} \}: \mathbf {h} _ {d} ^ {(L, t)} = 0,
$$
for some  $\phi^{(L,t)},\psi^{(L,t)}:\mathbb{R}^{D_x}\to \mathbb{R}_{>0}$ . Since
$$
\begin{array}{l} \mathbf {U} _ {m, n} = f ^ {(\theta , \mathcal {G}, t)} \left(\left(\mathbf {x} ^ {(i)} \leftarrow \mathbf {v} ^ {(m)}\right) _ {i \in \mathcal {I} ^ {\prime}}, \left(\mathbf {x} ^ {(j)} \leftarrow \mathbf {v} ^ {(n)}\right) _ {j \in \mathcal {J} ^ {\prime}}, \left(\mathbf {x} ^ {(k)} \leftarrow \mathbf {v} ^ {(M + 1)}\right) _ {k \in \mathcal {V} \backslash (\mathcal {I} ^ {\prime} \cup \mathcal {J} ^ {\prime})}\right) \\ = \mathbf {W} ^ {(o)} \mathbf {h} ^ {(L, t)} \\ \end{array}
$$
$$
\mathbf {U} _ {m, n} = \mathbf {h} _ {1} ^ {(L, t)} = \phi^ {(L, t)} \left(\mathbf {v} ^ {(m)}\right) \left\langle \mathbf {z} ^ {(m)}, \mathbf {z} ^ {(n)} \right\rangle^ {\rho_ {L - 1} \left(\mathcal {C} _ {t} ^ {*}, \{t \}\right)} \psi^ {(L, t)} \left(\mathbf {v} ^ {(n)}\right).
$$
Defining  $\phi := \phi^{(L,t)}$  and  $\psi := \psi^{(L,t)}$  leads to the sought-after expression for  $\mathbf{U}_{m,n}$ :
$$
\mathbf {U} _ {m, n} = \phi \left(\mathbf {v} ^ {(m)}\right) \left\langle \mathbf {z} ^ {(m)}, \mathbf {z} ^ {(n)} \right\rangle^ {\rho_ {L - 1} \left(\mathcal {C} _ {t} ^ {*}, \{t \}\right)} \psi \left(\mathbf {v} ^ {(n)}\right).
$$
I.3.3 Technical Lemmas
For completeness, we include the vector rearrangement inequality from [61], which we employ for proving the subsequent Lemma 7.
Lemma 6 (Lemma 1 from [61]). Let  $\mathbf{a}^{(1)},\ldots ,\mathbf{a}^{(M)}\in \mathbb{R}_{\geq 0}^{D}$  be  $M\in \mathbb{N}$  different vectors with nonnegative entries. Then, for any permutation  $\sigma :[M]\to [M]$  besides the identity permutation it holds that:
$$
\sum_ {m = 1} ^ {M} \left\langle \mathbf {a} ^ {(m)}, \mathbf {a} ^ {(\sigma (m))} \right\rangle <   \sum_ {m = 1} ^ {M} \left\| \mathbf {a} ^ {(m)} \right\| ^ {2}.
$$
Taking the  $P$ 'th Hadamard power of a rank at most  $D$  matrix results in a matrix whose rank is at most the multiset coefficient  $\left(\begin{array}{c}D\\ P\end{array}\right) := \binom{D + P - 1}{P}$  (see, e.g., Theorem 1 in [2]). Lemma 7, adapted from Appendix B.2 in [64], guarantees that we can always find a  $\left(\begin{array}{c}D\\ P\end{array}\right) \times D$  matrix  $\mathbf{Z}$  with positive entries such that  $\mathrm{rank}(\odot^P (\mathbf{Z}\mathbf{Z}^\top))$  is maximal, i.e. equal to  $\left(\begin{array}{c}D\\ P\end{array}\right)$ .
Lemma 7 (adapted from Appendix B.2 in [64]). For any  $D \in \mathbb{N}$  and  $P \in \mathbb{N}_{\geq 0}$ , there exists a matrix with positive entries  $\mathbf{Z} \in \mathbb{R}_{>0}^{\binom{D}{P}} \times D$  for which:
$$
\operatorname {r a n k} \big (\odot^ {P} \big (\mathbf {Z Z} ^ {\top} \big) \big) = \left(\left( \begin{array}{c} D \\ P \end{array} \right)\right),
$$
where  $\odot^P (\mathbf{Z}\mathbf{Z}^\top)$  is the  $P$ 'th Hadamard power of  $\mathbf{Z}\mathbf{Z}^\top$ .
Proof. We let  $M := \left( \begin{array}{c} D \\ P \end{array} \right)$  for notational convenience. Denote by  $\mathbf{z}^{(1)}, \ldots, \mathbf{z}^{(M)} \in \mathbb{R}^D$  the row vectors of  $\mathbf{Z} \in \mathbb{R}_{>0}^{M \times D}$ . Observing the  $(m,n)'$ th entry of  $\odot^P(\mathbf{ZZ}^\top)$ :
$$
\left[ \odot^ {P} \big (\mathbf {Z Z} ^ {\top} \big) \right] _ {m, n} = \left\langle \mathbf {z} ^ {(m)}, \mathbf {z} ^ {(n)} \right\rangle^ {P} = \left(\sum_ {d = 1} ^ {D} \mathbf {z} _ {d} ^ {(m)} \cdot \mathbf {z} _ {d} ^ {(n)}\right) ^ {P},
$$
by expanding the power using the multinomial identity we have that:
$$
\begin{array}{l} \left[ \odot^ {P} \left(\mathbf {Z} \mathbf {Z} ^ {\top}\right) \right] _ {m, n} = \sum_ {\substack {q _ {1}, \dots , q _ {D} \in \mathbb {N} _ {\geq 0} \\ \text {s.t.} \sum_ {d = 1} ^ {D} q _ {d} = P}} \binom {P} {q _ {1}, \dots , q _ {D}} \prod_ {d = 1} ^ {D} \left(\mathbf {z} _ {d} ^ {(m)} \cdot \mathbf {z} _ {d} ^ {(n)}\right) ^ {q _ {d}} \tag{29} \\ = \sum_{\substack{q_{1},\ldots ,q_{D}\in \mathbb{N}_{\geq 0}\\ \text{s.t.} \sum_{d = 1}^{D}q_{d} = P}}\binom {P}{q_{1},\ldots ,q_{D}}\left(\prod_{d = 1}^{D}\Bigl(\mathbf{z}_{d}^{(m)}\Bigr)^{q_{d}}\right)\cdot \left(\prod_{d = 1}^{D}\Bigl(\mathbf{z}_{d}^{(n)}\Bigr)^{q_{d}}\right), \\ \end{array}
$$
where in the last equality we separated terms depending on  $m$  from those depending on  $n$ .
Let  $(\mathbf{a}^{(q_1,\dots,q_D)} \in \mathbb{R}^M)_{q_1,\ldots,q_D \in \mathbb{N}_{\geq 0}}$  s.t  $\sum_{d=1}^{D} q_d = P$  be  $M$  vectors defined by  $\mathbf{a}_m^{(q_1,\dots,q_D)} = \prod_{d=1}^{D} (\mathbf{z}_d^{(m)})^{q_d}$  for all  $q_1,\ldots,q_D \in \mathbb{N}_{\geq 0}$  satisfying  $\sum_{d=1}^{D} q_d = P$  and  $m \in [M]$ . As can be seen from Equation (29), we can write:
$$
\odot^ {P} \left(\mathbf {Z Z} ^ {\top}\right) = \mathbf {A S A} ^ {\top},
$$
where  $\mathbf{A} \in \mathbb{R}^{M \times M}$  is the matrix whose columns are  $(\mathbf{a}^{(q_1, \ldots, q_D)})_{q_1, \ldots, q_D \in \mathbb{N}_{\geq 0}}$  s.t.  $\sum_{d=1}^{D} q_d = P$  and  $\mathbf{S} \in \mathbb{R}^{M \times M}$  is the diagonal matrix holding  $\binom{P}{q_1, \ldots, q_D}$  for every  $q_1, \ldots, q_D \in \mathbb{N}_{\geq 0}$  satisfying  $\sum_{d=1}^{D} q_d = P$  on its diagonal. Since all entries on the diagonal of  $\mathbf{S}$  are positive, it is of full-rank, i.e.  $\mathrm{rank}(\mathbf{S}) = M$ . Thus, to prove that there exists  $\mathbf{Z} \in \mathbb{R}_{>0}^{M \times D}$  for which  $\mathrm{rank}(\odot^P (\mathbf{Z} \mathbf{Z}^\top)) = M$ , it suffices to show that we can choose  $\mathbf{z}^{(1)}, \ldots, \mathbf{z}^{(M)}$  with positive entries inducing  $\mathrm{rank}(\mathbf{A}) = M$ , for  $\mathbf{A}$  as defined above. Below, we complete the proof by constructing such  $\mathbf{z}^{(1)}, \ldots, \mathbf{z}^{(M)}$ .
We associate each of  $\mathbf{z}^{(1)},\ldots ,\mathbf{z}^{(M)}$  with a different configuration from the set:
$$
\left\{\mathbf {q} = \left(q _ {1}, \dots , q _ {D}\right): q _ {1}, \dots , q _ {D} \in \mathbb {N} _ {\geq 0}, \sum_ {d = 1} ^ {D} q _ {d} = P \right\},
$$
where note that this set contains  $M = \left(\begin{pmatrix} D \\ P \end{pmatrix}\right)$  elements. For  $m \in [M]$ , denote by  $\mathbf{q}^{(m)}$  the configuration associated with  $\mathbf{z}^{(m)}$ . For a variable  $\gamma \in \mathbb{R}$ , to be determined later on, and every  $m \in [M]$  and  $d \in [D]$ , we set:
$$
\mathbf {z} _ {d} ^ {(m)} = \gamma^ {\mathbf {q} _ {d} ^ {(m)}}.
$$
Given these  $\mathbf{z}^{(1)},\ldots ,\mathbf{z}^{(M)}$ , the entries of  $\mathbf{A}$  have the following form:
$$
\mathbf {A} _ {m, n} = \prod_ {d = 1} ^ {D} \left(\mathbf {z} _ {d} ^ {(m)}\right) ^ {\mathbf {q} _ {d} ^ {(n)}} = \prod_ {d = 1} ^ {D} \left(\gamma^ {\mathbf {q} _ {d} ^ {(m)}}\right) ^ {\mathbf {q} _ {d} ^ {(n)}} = \gamma^ {\sum_ {d = 1} ^ {D} \mathbf {q} _ {d} ^ {(m)} \cdot \mathbf {q} _ {d} ^ {(n)}} = \gamma^ {\left\langle \mathbf {q} ^ {(m)}, \mathbf {q} ^ {(n)} \right\rangle},
$$
for all  $m, n \in [M]$ . Thus,  $\operatorname{det}(\mathbf{A}) = \sum_{\text{permutation } \sigma : [M] \to [M]} \operatorname{sign}(\sigma) \cdot \gamma^{\sum_{m=1}^{M} \langle \mathbf{q}^{(m)}, \mathbf{q}^{(\sigma(m))} \rangle}$  is polynomial in  $\gamma$ . By Lemma 6,  $\sum_{m=1}^{M} \langle \mathbf{q}^{(m)}, \mathbf{q}^{(\sigma(m))} \rangle < \sum_{m=1}^{M} \| \mathbf{q}^{(m)} \|^{2}$  for all  $\sigma$  which is not the identity permutation. This implies that  $\sum_{m=1}^{M} \| \mathbf{q}^{(m)} \|^{2}$  is the maximal degree of a monomial in  $\operatorname{det}(\mathbf{A})$ , and it is attained by a single element in  $\sum_{\text{permutation } \sigma : [M] \to [M]}\operatorname{sign}(\sigma) \cdot \gamma^{\sum_{m=1}^{M} \langle \mathbf{q}^{(m)}, \mathbf{q}^{(\sigma(m))} \rangle}$  — that corresponding to the identity permutation. Consequently,  $\operatorname{det}(\mathbf{A})$  cannot be the zero polynomial with respect to  $\gamma$ , and so it vanishes only on a finite set of values for  $\gamma$ . In particular, there exists  $\gamma > 0$  such that  $\operatorname{det}(\mathbf{A}) \neq 0$ , meaning  $\operatorname{rank}(\mathbf{A}) = M$ . The proof concludes by noticing that for a positive  $\gamma$  the entries of the chosen  $\mathbf{z}^{(1)}, \ldots, \mathbf{z}^{(M)}$  are positive as well.
Additionally, we make use of the following lemmas.
Lemma 8. For any  $D, P \in \mathbb{N}$ , let  $\left(\binom{D}{P}\right) := \binom{D + P - 1}{P}$  be the multiset coefficient. Then:
$$
\left(\left( \begin{array}{c} D \\ P \end{array} \right)\right) \geq \left(\frac {D - 1}{P} + 1\right) ^ {P}.
$$
Proof. For any  $N \geq K \in \mathbb{N}$ , a known lower bound on the binomial coefficient is  $\binom{N}{K} \geq \left(\frac{N}{K}\right)^{K}$ . Hence:
$$
\left(\left( \begin{array}{c} D \\ P \end{array} \right)\right) = \left( \begin{array}{c} D + P - 1 \\ P \end{array} \right) \geq \left(\frac {D + P - 1}{P}\right) ^ {P} = \left(\frac {D - 1}{P} + 1\right) ^ {P}.
$$
Lemma 9. For  $D_{1}, D_{2}, K \in \mathbb{N}$ , consider a polynomial function mapping variables  $\theta \in \mathbb{R}^{K}$  to matrices  $\mathbf{A}(\theta) \in \mathbb{R}^{D_{1} \times D_{2}}$ , i.e. the entries of  $\mathbf{A}(\theta)$  are polynomial in  $\theta$ . If there exists a point  $\theta^{*} \in \mathbb{R}^{K}$  such that  $\mathrm{rank}(\mathbf{A}(\theta^{*})) \geq R$ , for  $R \in [\min\{D_{1}, D_{2}\}]$ , then the set  $\{\theta \in \mathbb{R}^{K} : \mathrm{rank}(\mathbf{A}(\theta)) < R\}$  has Lebesgue measure zero.
Proof. A matrix is of rank at least  $R$  if and only if it has a  $R \times R$  sub-matrix whose determinant is non-zero. The determinant of any sub-matrix of  $\mathbf{A}(\theta)$  is polynomial in the entries of  $\mathbf{A}(\theta)$ , and so it is polynomial in  $\theta$  as well. Since the zero set of a polynomial is either the entire space or a set of Lebesgue measure zero [19], the fact that  $\mathrm{rank}(\mathbf{A}(\theta^{*})) \geq R$  implies that  $\{\theta \in \mathbb{R}^K : \mathrm{rank}(\mathbf{A}(\theta)) < R\}$  has Lebesgue measure zero.
Lemma 10. Let  $\mathbf{v},\mathbf{v}^{\prime}\in \mathbb{R}_{\geq 0}^{D_{x}}$  whose first  $D\coloneqq \min \{D_x,D_h\}$  entries are positive, and disjoint  $\mathcal{I}'$ $\mathcal{J}'\subseteq \mathcal{V}$  with no repeating shared neighbors (Definition 3). Denote by  $\mathbf{h}^{(l,i)}\in \mathbb{R}^{D_h}$  the hidden embedding for  $i\in \mathcal{V}$  at layer  $l\in [L]$  of a GNN with depth  $L\geq 2$  and product aggregation (Equations (2) and (5)), given the following assignment to its input variables (i.e. vertex features):
$$
\left(\mathbf {x} ^ {(i)} \gets \mathbf {v}\right) _ {i \in \mathcal {I} ^ {\prime}}, \left(\mathbf {x} ^ {(j)} \gets \mathbf {v} ^ {\prime}\right) _ {j \in \mathcal {I} ^ {\prime}}, \left(\mathbf {x} ^ {(k)} \gets \mathbf {1}\right) _ {k \in \mathcal {V} \backslash (\mathcal {I} ^ {\prime} \cup \mathcal {I} ^ {\prime})},
$$
where  $\mathbf{1} \in \mathbb{R}^{D_x}$  is the vector holding one in all entries. Suppose that the weights  $\mathbf{W}^{(1)}, \ldots, \mathbf{W}^{(L)}$  of the GNN are given by:
where  $\mathbf{I}$  is a zero padded identity matrix, i.e. it holds ones on its diagonal and zeros elsewhere. Then, for all  $l\in \{2,\ldots ,L\}$  and  $i\in \mathcal{V}$ , there exist  $\phi^{(l,i)},\psi^{(l,i)}:\mathbb{R}^{D_x}\to \mathbb{R}_{>0}$  such that:
$$
\mathbf {h} _ {1} ^ {(l, i)} = \phi^ {(l, i)} (\mathbf {v}) \left\langle \mathbf {v}: _ {D}, \mathbf {v} _ {: D} ^ {\prime} \right\rangle^ {\rho_ {l - 1} (\mathcal {C}, \{i \})} \psi^ {(l, i)} (\mathbf {v} ^ {\prime}), \quad \forall d \in \{2, \dots , D _ {h} \}: \mathbf {h} _ {d} ^ {(l, i)} = 0,
$$
where  $\mathcal{C} \coloneqq \mathcal{N}(\mathcal{I}') \cap \mathcal{N}(\mathcal{J}')$ .
Proof. The proof is by induction over the layer  $l \in \{2, \dots, L\}$ . For  $l = 2$ , fix  $i \in \mathcal{V}$ . By the update rule of a GNN with product aggregation:
$$
\mathbf {h} ^ {(2, i)} = \odot_ {j \in \mathcal {N} (i)} \left(\mathbf {W} ^ {(2)} \mathbf {h} ^ {(1, j)}\right).
$$
Plugging in the value of  $\mathbf{W}^{(2)}$  we get:
$$
\mathbf {h} _ {1} ^ {(2, i)} = \prod_ {j \in \mathcal {N} (i)} \left(\sum_ {d = 1} ^ {D _ {h}} \mathbf {h} _ {d} ^ {(1, j)}\right), \quad \forall d \in \{2, \dots , D _ {h} \}: \mathbf {h} _ {d} ^ {(2, i)} = 0. \tag {30}
$$
Let  $\bar{\mathbf{v}},\bar{\mathbf{v}}^{\prime}\in \mathbb{R}^{D_{h}}$  be the vectors holding  $\mathbf{v}_{:D}$  and  $\mathbf{v}_{:D}^{\prime}$  in their first  $D$  coordinates and zero in the remaining entries, respectively. Similarly, we use  $\bar{\mathbf{1}}\in \mathbb{R}^{D_h}$  to denote the vector whose first  $D$  entries are one and the remaining are zero. Examining  $\mathbf{h}^{(1,j)}$  for  $j\in \mathcal{N}(i)$ , by the assignment of input variables and the fact that  $\mathbf{W}^{(1)}$  is a zero padded identity matrix we have that:
$$
\begin{array}{l} \mathbf {h} ^ {(1, j)} = \odot_ {k \in \mathcal {N} (j)} \left(\mathbf {W} ^ {(1)} \mathbf {x} ^ {(k)}\right) = \left(\odot^ {| \mathcal {N} (j) \cap \mathcal {I} ^ {\prime} |} \bar {\mathbf {v}}\right) \odot \left(\odot^ {| \mathcal {N} (j) \cap \mathcal {J} ^ {\prime} |} \bar {\mathbf {v}} ^ {\prime}\right) \odot \left(\odot^ {| \mathcal {N} (j) \backslash (\mathcal {I} ^ {\prime} \cup \mathcal {J} ^ {\prime}) |} \bar {\mathbf {1}}\right) \\ = \left(\odot^ {| \mathcal {N} (j) \cap \mathcal {I} ^ {\prime} |} \bar {\mathbf {v}}\right) \odot \left(\odot^ {| \mathcal {N} (j) \cap \mathcal {J} ^ {\prime} |} \bar {\mathbf {v}} ^ {\prime}\right). \\ \end{array}
$$
Since the first  $D$  entries of  $\bar{\mathbf{v}}$  and  $\bar{\mathbf{v}}'$  are positive while the rest are zero, the same holds for  $\mathbf{h}^{(1,j)}$ . Additionally, recall that  $\mathcal{I}'$  and  $\mathcal{J}'$  have no repeating shared neighbors. Thus, if  $j \in \mathcal{N}(\mathcal{I}') \cap \mathcal{N}(\mathcal{J}') = \mathcal{C}$ , then  $j$  has a single neighbor in  $\mathcal{I}'$  and a single neighbor in  $\mathcal{J}'$ , implying  $\mathbf{h}^{(1,j)} = \bar{\mathbf{v}} \odot \bar{\mathbf{v}}'$ . Otherwise, if  $j \notin \mathcal{C}$ , then  $\mathcal{N}(j) \cap \mathcal{I}' = \emptyset$  or  $\mathcal{N}(j) \cap \mathcal{J}' = \emptyset$  must hold. In the former  $\mathbf{h}^{(1,j)}$  does not depend on  $\mathbf{v}$ , whereas in the latter  $\mathbf{h}^{(1,j)}$  does not depend on  $\mathbf{v}'$ .
Going back to Equation (30), while noticing that  $|\mathcal{N}(i)\cap \mathcal{C}| = \rho_1(\mathcal{C},\{i\})$  , we arrive at: