SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} \left[ a _ {k} \cdot x _ {i} \right] = \mathbb {E} \left[ \sum_ {j = 1} ^ {d} (a _ {k}) _ {j} \cdot (x _ {i}) _ {j} \right] = \sum_ {j = 1} ^ {d} \mathbb {E} \left[ (a _ {k}) _ {j} \right] \mathbb {E} \left[ (x _ {i}) _ {j} \right] = \sum_ {j = 1} ^ {d} 0 \cdot 0 = 0,
$$
and by definition  $\mathbb{E}\left[b_k\right] = 0$ . Since each  $(a_{k})_{j}$  and  $(x_{i})_{j}$  are independent random variables with expectation zero, we have that
$$
\operatorname {V a r} \left[ a _ {k} \cdot x _ {i} \right] = \operatorname {V a r} \left[ \sum_ {j = 1} ^ {d} (a _ {k}) _ {j} \cdot (x _ {i}) _ {j} \right] = \sum_ {j = 1} ^ {d} \operatorname {V a r} \left[ (a _ {k}) _ {j} \right] \operatorname {V a r} \left[ (x _ {i}) _ {j} \right] = \sum_ {j = 1} ^ {d} \frac {1}{2 d} \cdot 1 = \frac {1}{2},
$$
and by definition  $\operatorname{Var}\left[b_k\right] = \frac{1}{2}$ . Therefore, (iii) holds.
Computational resources All experiments were run on an NVidia A40 GPU with 48 GB of GPU memory.
E.2 Graph separation
We ran the graph separation experiment using the PyTorch Geometric [10] implementation for 'WL convolutions', and the GCN convolutions of [18], as well as their version of the TUDataset [30]. As initialization for the node features, we only took the node degree.
When moving from the abstract mathematical features that are mathematically equal  $h_i = h_j$  could end up having slightly different values. To deal with this, all computations were done in double precision. The final features computed by the MPNN were normalized to have an average norm of one, and two features  $h_i, h_j$  were deemed equal if  $|h_i - h_j| < 10^{-12}$ . In Figure 5 we show the minimum and median of the quantity  $\| h_i - h_j \|$  over all  $(i,j)$  from all graphs for which this norm was larger than the threshold of  $10^{-12}$ . This is shown for the SiLU activation, and the values are shown as a function of the hidden dimension used. We see that the minimal nonzero distance was safely larger than the threshold in all examples. Also note that the minimal distance moderately improves as the hidden dimension increases.
world to finite precision computing,
Figure 5: Minimal and median distance between non-equivalent features as a function of hidden feature dimension.