SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
a _ {i m m ^ {\prime} f} = 2 \left(\mu_ {c ^ {\prime} m ^ {\prime} f} - \mu_ {i m f}\right) a n d b _ {i m m ^ {\prime} f} = 2 \sqrt {\sigma_ {i m f} ^ {2} + \sigma_ {c ^ {\prime} m ^ {\prime} f} ^ {2}}.
$$
The choice of  $\varepsilon$ , which implicitly encodes our definition of "nearest" neighbor, plays a crucial role in determining the distribution of a neighbor's class label. When the hypercube is too big, the probability that a neighbor has class  $c$  matches the true prior  $p(c)$  regardless of the class  $c'$ , that is we make a crude assumption about the neighbor's class distribution of a sample. If we instead consider a smaller hypercube, then we observe a less trivial behavior and the probability  $p_{c'}(c, \varepsilon)$  is directly proportional to the distances between the means  $\mu_{c'm'}$  and  $\mu_{cm}$ , as one would intuitively expect for simple examples.
To summarize, we can use  $p_{c'}(c, \varepsilon)$  as an approximation for  $\mathbb{E}_{p(\boldsymbol{x} | c')} [q_c'(\boldsymbol{x})]$  of Equation 2; similarly, we can use a normalized version of  $M_c(\boldsymbol{x})$  in place of  $q_c(\boldsymbol{x})$  to estimate Equation 1 via Monte Carlo sampling without the need of building a NN graph. For space reasons, in Appendix A.5, we qualitatively discuss the results for a 1-dimensional instance of a data distribution, and we investigate the quality of the CCNS approximation while varying  $k$ .
3.2 Class Separability Induced by DGNs on a  $k$ -NN Graph
This section formally investigates the properties of the embedding space created by DGNs under the  $k$ -NN graph. Our goal is to understand whether using such a graph can improve the separability between samples belonging to different classes or not. Provided that our assumptions hold, both conclusions would be interesting: if the  $k$ -NN graph helps, then we know the conditions for that to happen; if that is not the case, we have identified and formalized the need for new graph construction mechanisms.
Akin to previous works [40, 16, 5, 43], we consider a linear 1-layer DGN with the following neighborhood aggregation scheme that computes node embeddings  $\pmb{h}_u \in \mathbb{R}^D \forall u \in \mathcal{V}$ :
$$
\boldsymbol {h} _ {u} = \frac {1}{k} \sum_ {v \in \mathcal {N} _ {u}} \boldsymbol {x} _ {v}. \tag {9}
$$
The node embedding of sample  $u$  is then fed into a standard machine learning classifier, e.g., an MLP. As done in previous works, we assume that a linear (learnable) transformation  $\mathbf{W} \in \mathbb{R}^{D \times D}$  of the input  $\mathbf{x}_v$ , often used in DGN models such as the Neural Network for Graphs [46] and the Graph Convolutional Network [36], is absorbed by the subsequent classifier.
Mimicking the behavior of the  $k$ -NN algorithm, which connects similar entities together, we model the attribute distribution of a neighbor  $v \in \mathcal{N}_u$  as a normal distribution  $\mathcal{N}(\boldsymbol{x}_v; \boldsymbol{x}_u, \text{diag}(\sigma^2, \ldots, \sigma^2)$ , where  $\sigma^2$  is a hyper-parameter that ensures it is highly unlikely to sample neighbors outside of  $H_{\varepsilon}(\boldsymbol{x}_u)$ ; from now on we use the symbol  $\sigma_{\varepsilon}$  to make this connection clear. Under assumption 3, neighbors' sampling is independently repeated  $k$  times and the attributes are averaged together. Therefore, we use statistical properties of normal distributions (please refer to Lemmas A.2 and A.1) to compute the resulting node  $u$ 's embedding distribution:
$$
p \left(\boldsymbol {h} _ {u} \mid \boldsymbol {x} _ {u}\right) = \mathcal {N} \left(\boldsymbol {h} _ {u}; \boldsymbol {x} _ {u}, \boldsymbol {\Lambda} _ {\varepsilon}\right), \quad \boldsymbol {\Lambda} _ {\varepsilon} = \operatorname {d i a g} \left(\sigma_ {\varepsilon} ^ {2} / k, \dots , \sigma_ {\varepsilon} ^ {2} / k\right). \tag {10}
$$
Therefore, the more neighbors a node has (i.e., higher  $k$ ) the more skewed the resulting distribution is around  $x_{u}$  (i.e., lower variance). This is a fairly reasonable assumption if we think of an infinitely large dataset; informally, each of the  $k$  neighbors will be much more likely to lie near  $x_{u}$  than near the surface of the hypercube, hence  $h_{u}$  will be close to  $x_{u}$  with high probability.
To understand how Equation 9 affects the separability of samples belonging to different classes, we compute a divergence score between the distributions  $p(\pmb{h} | c)$  and  $p(\pmb{h} | c')$  [68, 51, 20, 8, 52]. When
this divergence is higher than that of the distributions  $p(x|c)$  and  $p(x|c')$ , then the  $k$ -NN structure and the inductive bias of DGNs are helpful for our task. Below, we show that for two mixtures of Gaussians we can obtain the analytical form of their Squared Error Distance (SED), the simplest symmetric divergence defined as  $SED(p,q) = \int (p(x) - q(x))^2 dx$ . This provides a concrete strategy to understand, regardless of training, if it would make sense to build a  $k$ -NN graph for our problem.<sup>1</sup>
Proposition 3.8. Under assumptions 1,3, let us assume that the entity embeddings of a 1-layer DGN applied on an artificially built  $k$ -NN graph follow the distribution of Equation 10. Then the Squared Error Distances  $SED(p(\pmb{h}|c), p(\pmb{h}|c'))$  and  $SED(p(\pmb{x}|c), p(\pmb{x}|c'))$  have analytical forms.
In the interest of space, we have deferred the analytic formulas to the appendix.
As an immediate but important corollary, a  $k$ -NN graph improves the ability to distinguish samples of different classes  $c, c'$  if it holds that  $SED(p(\pmb{h}|c), p(\pmb{h}|c')) > TED(p(\pmb{x}|c), p(\pmb{x}|c'))$ . Indeed, if class distributions diverge more in the embedding space, which has the same dimensionality as the input space, then they will be easier to separate by a universal approximator such as an MLP. We use this corollary in our experiments, by reverse-engineering the theory to find out "good" data models.
3.3 Limitations and Future Directions
It is extremely challenging to prove that a  $k$ -NN graph never helps to improve the classification performances using a DGN. The reason is that theoretical results of this kind rely, as is common [43], on assumptions of the true data generating distribution and on the specific assignments of parameters such as  $\mu$  and  $\Sigma$ . The true data distribution is typically unknown and hard to find [68], but assumptions 1 and 3 remain quite reasonable due to the flexibility of mixture models and the nature of the nearest neighbors construction. In addition, assumption 2 has been widely used in the graphical models literature because it is particularly attractive from a theoretical perspective [64], despite it cannot cover all real-world cases; we note, however, that it plays no role in the analysis of Section 3.2. Finally, as done in the past [40, 15, 5, 43], we have ignored the non-linearity in the neighborhood aggregation of Equation 9. As we shall see in our empirical results, the intuitions we gained from the theoretical analysis seem to hold for non-linear models as well.
Our analysis has considered hyper-cubes in the interest of simplicity, but nearest neighbor graphs typically compute hyper-balls centered at a point. In high-dimensional spaces, the volume of hyper-cubes and hyper-balls can differ in non-trivial ways [18], so future work should improve on our results by replacing hyper-cubes with hyper-balls. Similarly, the results of Section 3.1 could be used to guide the definition of new graph construction strategies based on attribute similarity criteria, for instance by proposing a good  $\varepsilon$  for the data at hand. We leave this idea to future investigations.
We conduct quantitative and qualitative experiments to support our theoretical insights. For all experiments we used a server with 32 cores, 128 GBs of RAM, and 4 GPUs with 11 GBs of memory.
Quantitative Experiments Quantitatively speaking, we want to compare a structure-agnostic baseline, namely an MLP [58], against different graph machine learning models, such as a simple DGN (sDGN) that implements Equation 9 followed by an MLP classifier, the Graph Isomorphism Network (GIN) [74] and the Graph Convolutional Network (GCN) [36]. The goal is to show that in the fully-supervised setting using a  $k$ -NN graph does not offer any concrete benefit. We consider 11 datasets, eight of which were taken from the UCI repository [2], namely Abalone, Adult, Dry Bean, Electrical Grid Stability, Isolet, Musk v2, Occupancy Detection, Waveform Database Generator v2, as well as the citation networks Cora, Citeseer [60], and Pubmed [49]. For each dataset, we build a  $k$ -NN graph, where  $k$  is a hyper-parameter, using the node attributes' similarity to find neighbors (discarding the original structure in the citation networks). We report some datasets statistics in Table 2, and the metric to optimize is the F1 score. Note that it is difficult to ascertain if our assumptions hold for these datasets because we have no access to the true data distribution; however, a robust comparison for this topic is lacking in the literature and can help us gather further insights on whether our ideas might also hold in a more general setting. For every dataset and model, we follow the rigorous and fair evaluation setup of [22]: we perform a 10-fold cross validation for risk assessment,
<sup>1</sup>The proof also makes a simple but meaningful connection to the over-smoothing problem.
Table 1: Node classification mean and standard deviation results for a structure-agnostic baseline (MLP) compared to graph machine learning methods applied to artificially built  $k$ -NN graphs. We performed model selection w.r.t. the F1 score, but we also include the accuracy for completeness.
MLP	sDGN	GIN	GCN
F1	ACC	F1	ACC	F1	ACC	F1	ACC
Abalone	0.55±0.02	55.5±1.5	0.55±0.01	55.3±1.2	0.54±0.03	54.2±1.7	0.54±0.02	54.4±1.7
Adult	0.70±0.02	77.6±1.7	0.72±0.02	77.8±1.9	0.69±0.02	76.2±1.3	0.70±0.02	78.6±0.9
DryBean	0.78±0.04	77.4±4.0	0.77±0.06	77.3±5.4	0.75±0.03	74.4±3.2	0.71±0.07	72.6±4.9
eGrid	0.95±0.01	95.6±0.6	0.95±0.01	95.8±0.4	0.93±0.01	94.0±0.6	0.82±0.02	83.1±1.4
Isolet	0.95±0.01	95.4±0.5	0.95±0.00	95.4±0.3	0.95±0.01	95.3±0.7	0.92±0.01	91.7±1.4
Musk	0.99±0.01	99.3±0.3	0.99±0.01	99.5±0.3	0.94±0.05	97.0±2.2	0.94±0.02	96.7±0.9
Occupancy	0.99±0.01	98.9±0.4	0.99±0.01	98.9±0.5	0.98±0.01	98.6±0.6	0.90±0.17	95.8±6.0
Waveform	0.85±0.02	85.5±1.7	0.85±0.02	85.3±1.5	0.84±0.02	84.5±1.7	0.84±0.03	84.0±2.2
Cora	0.71±0.04	74.3±2.4	0.71±0.03	74.1±2.5	0.72±0.03	74.9±2.4	0.70±0.04	73.0±2.6
Citeseer	0.69±0.02	71.8±1.9	0.69±0.03	72.9±2.7	0.69±0.02	72.1±2.2	0.71±0.03	74.4±2.4
Pubmed	0.87±0.01	87.6±0.7	0.87±0.01	87.6±0.6	0.87±0.01	86.6±0.9	0.83±0.01	83.5±0.9
with a hold-out model selection (90% training/10% validation) for each of the 10 external training folds. We report the hyper-parameters for the model selection phase in Table 3. For each of the 10 best models, we perform 3 final re-training runs and average their test performances on the corresponding external fold to mitigate bad initializations. The final score is the average of these 10 test results.
Qualitative Experiments We provide four qualitative experiments. First, we study how  $p_c(c', \varepsilon)$  changes for varying values of  $\varepsilon$  and different priors  $p(c)$  under specific class-conditional data distributions. Second, we analyze the first limit of Proposition 3.7 under the same conditional data distributions as before ( $D = 1$  hence assumptions are trivially satisfied). We translated one of the distributions by different values and computed the SED between them, together with the quantity  $p_c(c', \varepsilon)$  under different priors. The goal here is to show the relation between the SED and the limit. Third, we reverse-engineer the theoretical results to find, if possible, data distributions satisfying  $SED(p(h|c), p(h|c')) > SED(p(x|c), p(x|c'))$ . There is no dataset associated with this experiment, rather we learn the parameters of the graphical model of Figure 1 (left) together with  $\varepsilon, k, \sigma_{\varepsilon}$  ( $k$  is treated as a continuous value during the optimization) in order to minimize the following objective:  $\mathcal{L}_{SED} - \lambda \mathcal{L}_{CCNS}$ , where  $\lambda$  is a hyper-parameter.  $\mathcal{L}_{SED}$  sums  $SED(p(x|c), p(x|c')) - SED(p(h|c), p(h|c'))$  for all pairs of distinct classes  $c, c'$ , whereas  $\mathcal{L}_{CCNS}$  computes the lower bound for the inter-class similarity, thus acting as a regularizer that avoids the trivial solution  $p(x|c) = p(x|c')$  for class pairs  $c, c'$ . The set of configurations for this experiment is reported in Table 4, and we use Adam [35] as optimizer. In the last qualitative experiment, we compute the true CCNS and its approximations for the specific example of Figure 1 (right), showing that the quantities obtained in Proposition 3.3 and Theorem 3.5 are good representatives of the CCNS for nearest neighbor graphs. Please refer to Appendix A.5 for an additional study of this kind.
We now present our empirical results to support the theory. The interested reader can find statistics on the chosen hyper-parameters and additional ablation studies in Appendix A.6.
Quantitative Results Table 1 details the empirical comparison between the structure-agnostic MLP and the structure-aware baselines sDGN, GIN, and GCN. We observe that in terms of Micro F1 score, which is the metric all models are optimized against, there is no statistically significant improvement in performance (unpaired two samples t-test with p-value 0.05) for the different DGNs when using an artificial  $k$ -NN graph. When looking at accuracy one can observe a similar trend, but the results fluctuate more because accuracy is not the metric to be optimized. These results further confirm that using  $k$ -NN graphs does not bring a substantial contribution to the generalization performances. On the contrary, it would seem the  $k$ -NN graph can even be considered harmful for the performances of GIN and GCN (e.g., DryBean, Musk, eGrid, Waveform). We hypothesize that the information flow
$^{2}$ Code: https://github.com/nec-research/class_distributions_graphs_tabular_data
Figure 2: We show how  $p_c(c', \varepsilon)$  varies for different values of  $\varepsilon$  and prior distributions assuming the class-conditional data distributions on the left.
of message passing does not act anymore as a regularizer (we refer the reader to the discussion in Section 1) as it might happen the scarce supervision scenario [23], rather it becomes harmful noise for the learning process. This is possibly the reason why sDGN, which is also the simplest of the DGNs considered, tends to overfit less such noise and always performs on par with the MLP.
Therefore, one of the main takeaways of the quantitative analysis is that the  $k$ -NN graph might generally not be a good artificial structure for addressing the classification of tabular data using graph representation learning methods. Different artificial structures, possibly inspired by our theoretical results, should be proposed to create graphs that help DGNs to better generalize in this context. We note that this research direction is broader than tabular classification, but tabular data offers a great starting point since samples are typically assumed i.i.d. in ML and are thus easier to manipulate. Also, in Appendix A.4, we discuss the impact of a different  $\Omega$  (the cosine similarity) in the construction of the  $k$ -NN graphs.
These results also substantiate our conjecture that, under assumptions 1, 3, the  $k$ -NN graph never induces a better class separability in the embedding space: since  $p(\boldsymbol{h}|\boldsymbol{c})$  is a mixture of distributions with the same mean but higher variance than  $p(\boldsymbol{x}|\boldsymbol{c})$  (proof of Proposition 3.8), in a low-dimensional space intuition suggests that the distributions of different classes will always overlap more in the embedding space than in the input space.
Qualitative Results We present the main qualitative results in Figures 2, 3 and 4. First, as we can see from Figure 2, when  $\varepsilon$  is very small the probability that a neighbor is of the same class is high for both classes (but not the same), since there is less overlap between samples of the green ( $C = 0$ ) and blue ( $C = 1$ ) distributions. As we increase the side length of the hypercube (or equivalently, we consider more and more nearest neighbors), the curves converge to the priors that we set for this example. This result is in agreement with the second limit of Proposition 3.7.
As regards the first limit of Proposition 3.7, in Figure 3 we observe that the behavior of  $\lim_{\varepsilon \to 0}p_c(c',\varepsilon)$  is non-trivial when the two curves still overlap. In particular, the greater the overlap the closer  $p_c(c,\varepsilon)$  is to  $p(c)$ , which makes sense since it becomes harder to distinguish samples of different classes. On the other hand, the behavior becomes trivial when the curves are distant enough from each other: there, in the limit, every node will only have neighbors of the same class.
In the reverse-engineering experiment of Figure 4 (left), for each possible configuration of the models tried in Table 4, we have computed the  $\mathcal{L}_{SED}$  for varying values of  $k\in [1,500]$ . We have then normalized all values for readability, by dividing each value of  $SED(p(\pmb {h}|c),p(\pmb {h}|c'))$  by  $SED(p(\pmb {x}|c),p(\pmb {x}|c'))$ , as the latter is independent of  $k$ . This particular figure depicts the curves for the binary classification cases, but the conclusions do not change for multi-class classification, for instance with 5 classes. We find that the normalized value of  $SED(p(\pmb {h}|c),p(\pmb {h}|c'))$  on the  $y$ -axis is always upper-bounded by 1, meaning that  $SED(p(\pmb {h}|c),p(\pmb {h}|c')) < SED(p(\pmb {x}|c),p(\pmb {x}|c'))$  for all the configurations tried, even in higher dimensions. This result further indicates that it could be unlikely that a  $k$ -NN graph induces better class separability in the embedding space of a DGN. We ask machine learning practitioners to take these insights into consideration when building artificial nearest neighbor graphs.
Lastly, we go back to the data distribution of Figure 1 (right) and show how to approximate the true CCNS under the  $k$ -NN graph with our theoretical framework. We first generate 10000 data points and set w.l.o.g.  $k = 5$  to connect them. We compute the lower bound of the CCNS in the first
Figure 3: We show how  $p_c(c', \varepsilon)$  varies for different translations of the conditional data distributions  $p(x|0)$  and different priors as  $\varepsilon$  approaches 0.
Figure 4: Left: Each curve specifies the variation of  $SED(p(\pmb{h}|\pmb{c}), p(\pmb{h}|\pmb{c}'))$  against different values of  $k$  for different hyperparameters' configurations that have been trained to minimize the learning objective. The curves are normalized w.r.t their corresponding  $SED(p(\pmb{x}|\pmb{c}), p(\pmb{x}|\pmb{c}'))$ . Right: The CCNS lower bound (a), its Monte Carlo approximation for  $\varepsilon \approx 0.1$ , obtained without creating any graph (b), and the true CCNS (c) for the example of Figure 1 and a 5-NN graph. Heatmaps (a) and (b) are computed using the theory of Section 3.1. Please refer to the text for more details.
heatmap (a), a Monte Carlo (MC) approximation of Equation 1 (1000 samples per class pair) in the heatmap (b), and the true CCNS in the heatmap (c). In this case the lower bound has computed a good approximation of the inter-class CCNS, and the MC estimate is very close to the true CCNS. In Appendix A.6, we also show that the MC approximation is better for higher values of  $k$ . It is important to recall that both heatmaps (a) and (b) are computed using the theoretical results of Section 3.1 and an  $\varepsilon$  close to the empirical one, hence it seems that the theory is in accord with the evidence.
We introduced a new theoretical tool to understand how much nearest neighbor graphs can help in the classification of tabular data. Our theory and empirical evidence suggest that some attribute-based graph construction mechanisms, namely the  $k$ -NN algorithm, are not promising strategies for obtaining better DGNs' generalization performances. This is a particularly troubling result since researchers often used nearest neighbor graphs when the graph structure was unavailable. We recommend using great care in future empirical evaluations of such techniques by always comparing against structure-agnostic baselines to ascertain real improvements from fictitious ones. Moreover, we provided a theoretically principled way to model the CCNS for nearest neighbor graphs, showing its approximation in a practical example. We hope that the results of this work will foster better strategies for building artificial graphs or as a guideline for new structure learning methods.
The author would like to acknowledge, in no particular order, Sascha Saralajew, Julia Gastinger, Ammar Shaker, Filippo Grazioli, Luca Oneto, Sandra Tartarelli, and Timo Sztyler for the stimulating discussions and extremely helpful feedback. A warm thank you also goes to the reviewers for striving to further improve this work and make it ready for publication.
[78] Ziwei Zhang, Peng Cui, and Wenwu Zhu. Deep learning on graphs: A survey. IEEE Transactions on Knowledge and Data Engineering, 34(1):249-270, 2020.
[79] Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. In Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS), 2020.
A.1 Dataset Statistics
This section shows the main statistics of the datasets used in this work.
Table 2: Datasets' statistics.
# samples	# attributes	# classes
Abalone	4177	8	3
Adult	48842	14	2
DryBean	13611	17	7
eGrid	10000	14	2
Isolet	7797	617	26
Musk	6598	168	2
Occupancy	20560	7	2
Waveform	5000	40	3
Cora	2708	1433	7
Citeseer	3327	3703	6
Pubmed	19717	500	3
A.2 Hyper-Parameters Tried
The table below specifies the configurations tried by the model selection procedure for all the baselines considered in this work. We apply a patience-based early stopping technique on the validation Micro F1-score as the only regularization technique [55]. The free parameters' budget of all three baselines is the same. We used Adam to train all models.
Table 3: Hyper-parameters tried during model selection for all the benchmarks of the paper. Note that in GCN the use of the normalized Laplacian implies a mean aggregation.
# hidden units	# layers	learning rate	k	epochs	patience	aggregation
MLP	32/64/128	1/2/3	1e-2/1e-3	-	5000	1000	-
sDGN	32/64/128	1/2/3	1e-2/1e-3	1/3/5/10	5000	1000	mean
GIN	32/64/128	1/2/3	1e-2/1e-3	1/3/5/10	5000	1000	sum/mean
GCN	32/64/128	1/2/3	1e-2/1e-3	1/3/5/10	5000	1000	mean
The following table, instead, reports the configurations tried during the qualitative experiments. We still apply a patience-based early stopping technique on  $\mathcal{L}_{SED}$ .
Table 4: Hyper-parameters tried for the qualitative experiments.
D	|M|	|C|	λ	learning rate	epochs	patience
1,2,5,10,50	2,5,10	2,5	0.0, 1.0, 5.0, 10.0	1e-2	10000	3000
$$
p (c) \sum_ {m = 1} ^ {| \mathcal {M} |} p (m | c) \prod_ {f = 1} ^ {D} \left(F _ {Z _ {c m f}} \left(x _ {f} + \frac {\varepsilon}{2}\right) - F _ {Z _ {c m f}} \left(x _ {f} - \frac {\varepsilon}{2}\right)\right), \tag {11}
$$
Proof. Thanks to the conditional independence assumption, we can compute the integral of a product as a product of integrals over the independent dimensions. Defining  $a_{f} = x_{f} - \frac{\varepsilon}{2}$  and  $b_{f} = x_{f} + \frac{\varepsilon}{2}$ , we have