SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition 3.8. Under assumptions 1,3, let us assume that the entity embeddings of a 1-layer DGN applied on an artificially built  $k$ -NN graph follow the distribution of Equation 10. Then the Squared Error Distances  $SED(p(\pmb{h}|\pmb{c}), p(\pmb{h}|\pmb{c}'))$  and  $SED(p(\pmb{x}|\pmb{c}), p(\pmb{x}|\pmb{c}'))$  have analytical forms.
Proof. First, we have to show how we can compute the conditional probabilities  $p(\pmb{h}|\pmb{c})$  and  $p(\pmb{h}|\pmb{c}')$ , and then we derive the analytical computation of the  $SED$  divergence to verify the inequality in a similar manner of Helén and Virtanen [33].
We first work out the explicit form of  $p(h|c)$ , by marginalizing out  $x$ :
$$
\begin{array}{l} p (\boldsymbol {h} | c) = \int \sum_ {m} ^ {| \mathcal {M} |} p (\boldsymbol {h} | \boldsymbol {x}) p (\boldsymbol {x} | m, c) p (m | c) d \boldsymbol {x} (71) \\ = \sum_ {m} ^ {| \mathcal {M} |} p (m | c) \int p (\boldsymbol {h} | \boldsymbol {x}) p (\boldsymbol {x} | m, c) d \boldsymbol {x} (72) \\ = \sum_ {m} ^ {| \mathcal {M} |} p (m | c) \int \mathcal {N} (\boldsymbol {h}; \boldsymbol {x}, \boldsymbol {\Lambda} _ {\varepsilon}) \mathcal {N} (\boldsymbol {x}; \boldsymbol {\mu} _ {m c}, \boldsymbol {\Sigma} _ {m c}) d \boldsymbol {x} (73) \\ = \sum_ {m} ^ {| \mathcal {M} |} p (m | c) \underbrace {\int \mathcal {N} (\boldsymbol {h} - \boldsymbol {x} ; \boldsymbol {0} , \boldsymbol {\Lambda} _ {\varepsilon}) \mathcal {N} (\boldsymbol {x} ; \boldsymbol {\mu} _ {m c} , \boldsymbol {\Sigma} _ {m c}) d \boldsymbol {x}} _ {\text {G a u s s i a n c o n v l u t i o n}} (74) \\ = \sum_ {m} ^ {| \mathcal {M} |} p (m | c) \mathcal {N} (\boldsymbol {h}; \boldsymbol {\mu} _ {m c}, \boldsymbol {\Sigma} _ {m c} + \boldsymbol {\Lambda} _ {\varepsilon}). (75) \\ \end{array}
$$
Therefore, the distribution resulting from the  $k$ -NN neighborhood aggregation will change the input's variance in a way that is inversely proportional to the number of neighbors. In the limit of  $k \to \infty$  and finite  $\sigma_{\varepsilon}$ , it follows that  $p(\boldsymbol{h} | c) = p(\boldsymbol{x} | c)$ . Since  $p(\boldsymbol{h} | c)$  still follows a mixture of distributions of known form, we note that a repeated aggregation of the neighborhood aggregation mechanism (without any learned transformation) would only increase the values of the covariance matrix. In turn, this would make the distribution spread more and more, causing what is known in the literature as the oversmoothing effect (see Section 2).
Since both  $X_{c}$  and  $H_{c}$  follow a Gaussian mixture distribution, we apply Lemma A.4 to obtain a closed-form solution and be able to evaluate the inequality. In particular
$$
\begin{array}{l} S E D \left(\phi_ {X _ {c}}, \phi_ {X _ {c ^ {\prime}}}\right) = (76) \\ = \sum_ {\substack {m = 1 \\ m ^ {\prime} = 1}} ^ {| \mathcal {M} |} p (m | c) p \left(m ^ {\prime} \mid c\right) \mathcal {N} \left(\boldsymbol {\mu} _ {c m}; \boldsymbol {\mu} _ {c m ^ {\prime}}, \boldsymbol {\Sigma} _ {c m} + \boldsymbol {\Sigma} _ {c m ^ {\prime}}\right) (77) \\ + \sum_ {\substack {m = 1 \\ m ^ {\prime} = 1}} ^ {| \mathcal {M} |} p (m | c ^ {\prime}) p (m ^ {\prime} | c ^ {\prime}) \mathcal {N} \left(\boldsymbol {\mu} _ {c ^ {\prime} m}; \boldsymbol {\mu} _ {c ^ {\prime} m ^ {\prime}}, \boldsymbol {\Sigma} _ {c ^ {\prime} m} + \boldsymbol {\Sigma} _ {c ^ {\prime} m ^ {\prime}}\right) (78) \\ - 2 \sum_ {\substack {m = 1 \\ m ^ {\prime} = 1}} ^ {| \mathcal {M} |} p (m | c) p \left(m ^ {\prime} \mid c ^ {\prime}\right) \mathcal {N} \left(\boldsymbol {\mu} _ {c m}; \boldsymbol {\mu} _ {c ^ {\prime} m ^ {\prime}}, \boldsymbol {\Sigma} _ {c m} + \boldsymbol {\Sigma} _ {c ^ {\prime} m ^ {\prime}}\right). (79) \\ \end{array}
$$
$$
\begin{array}{l} S E D \left(\phi_ {H _ {c}}, \phi_ {H _ {c ^ {\prime}}}\right) = (80) \\ = \sum_ {\substack {m = 1 \\ m ^ {\prime} = 1}} ^ {| \mathcal {M} |} p (m | c) p \left(m ^ {\prime} | c\right) \mathcal {N} \left(\boldsymbol {\mu} _ {c m}; \boldsymbol {\mu} _ {c m ^ {\prime}}, \boldsymbol {\Sigma} _ {c m} + \boldsymbol {\Sigma} _ {c m ^ {\prime}} + 2 \boldsymbol {\Lambda} _ {\varepsilon}\right) (81) \\ + \sum_ {\substack {m = 1 \\ m ^ {\prime} = 1}} ^ {| \mathcal {M} |} p (m | c ^ {\prime}) p (m ^ {\prime} | c ^ {\prime}) \mathcal {N} \left(\boldsymbol {\mu} _ {c ^ {\prime} m}; \boldsymbol {\mu} _ {c ^ {\prime} m ^ {\prime}}, \boldsymbol {\Sigma} _ {c ^ {\prime} m} + \boldsymbol {\Sigma} _ {c ^ {\prime} m ^ {\prime}} + 2 \boldsymbol {\Lambda} _ {\varepsilon}\right) (82) \\ - 2 \sum_ {\substack {m = 1 \\ m ^ {\prime} = 1}} ^ {| \mathcal {M} |} p (m | c) p \left(m ^ {\prime} \mid c ^ {\prime}\right) \mathcal {N} \left(\boldsymbol {\mu} _ {c m}; \boldsymbol {\mu} _ {c ^ {\prime} m ^ {\prime}}, \boldsymbol {\Sigma} _ {c m} + \boldsymbol {\Sigma} _ {c ^ {\prime} m ^ {\prime}} + 2 \boldsymbol {\Lambda} _ {\varepsilon}\right). (83) \\ \end{array}
$$
As an additional remark, if  $SED(\phi_{H_c}, \phi_{H_{c'}}) > SED(\phi_{X_c}, \phi_{X_{c'}})$ , points in the DGN's latent space will be easier to classify than those in the input space because the former pair of distributions are more separable than the latter.
We also observe that  $p(\pmb{h} | c)$  is a mixture of Gaussians that differs from  $p(\pmb{x} | c)$  only in the higher variance of the individual Gaussian components. In other words, the distribution of  $p(\pmb{h} | c)$  looks "flatter" than  $p(\pmb{x} | c)$ . At least in a one or two-dimensional space, flatter distributions  $p(\pmb{h} | c)$  and  $p(\pmb{h} | c')$  should therefore increase the overlap (and therefore decrease the SED) compared to the SED in the original space. This intuition, despite its simplicity, is unfortunately hard to prove for technical reasons, for instance the dependence of the distance value on the specific parameters  $\mu$  and  $\Sigma$ .
A.4 Impact of the Metric in the Construction of  $k$ -NN Graphs
To judge how much the results might change if we used a different metric, we test the cosine similarity as done in [56]. We note that the cosine distance is less expressive than the Euclidean distance as it does not consider the magnitude of values, and that results in overall worse performances. DGNs typically work in the Euclidean space when aggregating neighbors, hence the homophily assumption can be imposed - to a certain extent - only by computing nearest neighbors in terms of Euclidean distance. In addition, building  $k$ -NN graphs using the Euclidean distance criterion has an easy interpretation in terms of "how similar the attributes of adjacent nodes are". We observe that the results below still agree with our theoretical intuitions and empirical results.
Table 5: Node classification mean and standard deviation results for a structure-agnostic baseline (MLP) compared to graph machine learning methods applied to artificially built  $k$ -NN graphs. We performed model selection w.r.t. the F1 score, but we also include the accuracy for completeness. Here,  $k$ -NN graphs are built using the cosine similarity as metric.
MLP	sDGN	GIN	GCN
F1	ACC	F1	ACC	F1	ACC	F1	ACC
Abalone	0.55±0.02	55.5±1.5	0.55±0.01	55.5±0.9	0.55±0.03	55.2±1.8	0.55±0.02	55.1 (1.9)
Adult	0.70±0.02	77.6±1.7	0.64±0.01	79.0±0.6	0.69±0.03	79.5±0.6	0.53±0.07	76.4 (0.6)
DryBean	0.78±0.04	77.4±4.0	0.77±0.06	77.0±5.0	0.79±0.02	78.2±2.0	0.78±0.06	78.2 (5.3)
eGrid	0.95±0.01	95.6±0.6	0.84±0.01	85.8±0.9	0.93±0.01	93.9±0.8	0.81±0.01	82.6 (1.4)
Isolet	0.95±0.01	95.4±0.5	0.93±0.01	92.7±1.0	0.95±0.01	94.8±1.0	0.90±0.05	90.6 (4.1)
Musk	0.99±0.01	99.3±0.3	0.96±0.01	98.0±0.6	0.93±0.04	96.2±2.5	0.93±0.04	97.1 (0.7)
Occupancy	0.99±0.01	98.9±0.4	0.99±0.01	98.6±0.3	0.99±0.01	98.9±0.3	0.95±0.1	97.6 (4.8)
Waveform	0.85±0.02	85.5±1.7	0.84±0.01	84.0±1.4	0.84±0.01	84.1±1.6	0.82±0.03	82.6 (2.3)
Cora	0.71±0.04	74.3±2.4	0.71±0.03	74.1±2.7	0.72±0.02	75.5±1.8	0.72±0.03	75.2 (2.3)
Citeseer	0.69±0.02	71.8±1.9	0.69±0.02	71.8±1.8	0.70±0.02	73.7±1.3	0.69±0.02	71.7 (2.4)
Pubmed	0.87±0.01	87.6±0.7	0.84±0.01	83.8±0.7	0.86±0.01	86.7±0.8	0.84±0.01	83.7 (0.6)
A.5 Analysis of CCNS approximation's quality
In the figure below, we visualize how the true CCNS gets closer to the MC approximation when the number of neighbors  $k$  increases for a random data distribution with 5 classes. The choice of the  $\varepsilon$  parameter for the MC estimation of the CCNS has a marginal impact here. This behavior is expected, because as  $k$  increases we get a better approximation of the posterior class distribution in the hypercube around each point, which is the one we computed in Section 3.1. When  $k$  is too small, the empirical (true) CCNS might be more unstable and dataset-dependent.
Figure 5: Comparison of the true CCNS for varying values of  $k$  and the MC approximation according to our theory for a random initialization of a 5-class data model.
A.6 Best Configurations and Ablation Studies
In the figure below, we show which are the best  $k$  and  $\varepsilon$  values picked by the models trained for the qualitative experiments.
Figure 6: We show the histograms of the learned  $k$  and  $\varepsilon$  for the different configurations of Table 4. It appears that lower values of both parameters are preferred by the trained models.
Analysis of Chosen Hyper-parameters For completeness, in the following we display the best  $k$ , number of layers, and number of hidden units chosen by the different families of models on each dataset of Table 1. We recall that for each dataset and model there are 10 best configurations due to the use of a 10-fold cross validation for risk assessment [22]. We did not find any significant pattern except for sDGN preferring lower values of  $k$  and the DGNs often choosing at least 2 convolutional layers.
Figure 7: Histograms of the  $k$  chosen by each model on the different datasets during a 10-fold cross validation for risk assessment.
Figure 8: Histograms of the number of layers chosen by each model on the different datasets during a 10-fold cross validation for risk assessment.
Figure 9: Histograms of the hidden units chosen by each model on the different datasets during a 10-fold cross validation for risk assessment.
Ablation Study We now provide an ablation study of the validation performances of the models during the model selection phase (i.e., across all possible configurations). We show the ablations for the most important architectural choices of the hyper-parameters. Also in this case, a lower  $k$  is associated with better performances, but there does not seem to be marked improvements when using more than 1 graph convolutional layer.
Figure 10: Ablation study for the  $k$  chosen by each model on the different datasets. We report the validation performances computed during model selection for all configurations tried.
Figure 11: Ablation study for the number of layers chosen by each model on the different datasets. We report the validation performances computed during model selection for all configurations tried.
Figure 12: Ablation study for the hidden units chosen by each model on the different datasets. We report the validation performances computed during model selection for all configurations tried.