SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Federico Errica
NEC Laboratories Europe
Heidelberg, Germany
Researchers have used nearest neighbor graphs to transform classical machine learning problems on tabular data into node classification tasks to solve with graph representation learning methods. Such artificial structures often reflect the homophily assumption, believed to be a key factor in the performances of deep graph networks. In light of recent results demystifying these beliefs, we introduce a theoretical framework to understand the benefits of Nearest Neighbor (NN) graphs when a graph structure is missing. We formally analyze the Cross-Class Neighborhood Similarity (CCNS), used to empirically evaluate the usefulness of structures, in the context of nearest neighbor graphs. Moreover, we study the class separability induced by deep graph networks on a  $k$ -NN graph. Motivated by the theory, our quantitative experiments demonstrate that, under full supervision, employing a  $k$ -NN graph offers no benefits compared to a structure-agnostic baseline. Qualitative analyses suggest that our framework is good at estimating the CCNS and hint at  $k$ -NN graphs never being useful for such classification tasks under full supervision, thus advocating for the study of alternative graph construction techniques in combination with deep graph networks.
The pursuit of understanding real-world phenomena has often led researchers to model the system of interest as a set of interdependent constituents, which influence each other in complex ways. In disciplines such as chemistry, physics, and network science, graphs are a convenient and well-studied mathematical object to represent such interacting entities and their attributes. In machine learning, the term "graph representation learning" refers to methods that can automatically leverage graph-structured data to solve tasks such as entity (or node), link, and whole-graph predictions [12, 32, 78, 73, 4].
Most of these methods assume that the relational information, that is the connections between entities, naturally emerges from the domain of the problem and is thus known. There is also broad consensus that connected entities typically share characteristics, behavioral patterns, or affiliation, something known as the homophily assumption [45]. This is possibly why, when the structure is not available, researchers have tried to artificially build Nearest Neighbor (NN) graphs from tabular data, by connecting entities based on some attribute similarity criterion, with applications in healthcare [44, 65], fake news and spam detection [39, 7], biology [77, 42], and document classification [56] to name a few. From an information-theoretic perspective, the creation of such graphs does not add new information as it depends on the available data; that said, what makes their use plausible is that the graph construction is a form of feature engineering that often encodes the homophily assumption. Combined with the inductive bias of Deep Graph Networks (DGNs) [59, 46], this strategy aims at improving the generalization performances on tabular data compared to structure-agnostic baselines, for example, a Multi-Layer Perceptron (MLP).
Indeed, using a  $k$ -NN graph [21] has recently improved the node classification performances under the scarcity of training labels [24, 23]. This is also known as the semi-supervised setting, where one can access the features of all nodes but the class labels are available for a handful of those. A potential explanation for these results is that, by incorporating neighboring values into each entity's representation, the neighborhood aggregation performed by DGNs acts as a regularization strategy that prevents the classifier from overfitting the attributes of the few labeled nodes, similar to input jittering [57, 62, 6]. However, it is still unclear what happens when one has access to all training labels (hereinafter the fully-supervised setting), namely if these graph-building strategies grant a statistically significant advantage in generalization compared to a structure-agnostic baseline. In this respect, proper comparisons against such baselines are often lacking or unclear in previous works, an issue that has also been reported in recent papers about the reproducibility of node and graph classification experiments [61, 22, 54].
In addition, it was recently shown [43] that homophily is not required to achieve good classification performances in node classification tasks; rather, what truly matters is how different the neighborhood class label distributions of nodes of separate classes are. This resulted in the definition of the empirical Cross-Class Neighborhood Similarity (CCNS) [43, 13], an object that estimates such similarities based on the available connectivity structure. Yet, whether or not artificially built graphs can be useful for the task at hand has mainly remained an empirical question, and more theoretical conditions for which this happens are still not understood.
In this paper, we introduce a theoretical framework to approach this question, and we provide two analyses of independent interest. Inspired by the CCNS, Section 3.1 studies the neighboring class label distribution of nearest neighbor graphs. Then Section 3.2 deals with the distribution of entity embeddings induced by DGNs on a  $k$ -NN graph, and we use it to quantify class separability in both the input and the embedding spaces. Overall, our theoretical results suggest that building a  $k$ -NN graph is not a good idea. To validate the theory with empirical evidence, we robustly compare (Sections 4 and 5) four baselines across 11 tabular datasets to check that the  $k$ -NN graph construction does not give statistically significant advantages in the fully-supervised setting. In addition, we reverse-engineer the theory to learn data distributions that would make a  $k$ -NN graph useful in practice when combined with DGNs. From the empirical results, we conjecture that this is never the case. Therefore, we hope to raise awareness in the graph machine learning community about the need for alternative (either NN-based or not) graph construction techniques.
In summary, our contributions are:  $i$  ) under some assumptions, we can estimate the CCNS for nearest neighbor graphs and provide its first lower bound; ii) we study how applying a simple DGN to an artificial  $k$ -NN graph affects the class separability of the input data; iii) we carry out a robust comparison between structure-agnostic and structure-aware baselines on a set of 11 datasets that is in agreement with our theoretical results; iv) qualitative analyses based on the theory further suggest that using the  $k$ -NN graph might not be advisable.
The early days of graph representation learning date back to the end of the previous century, when researchers proposed backpropagation through structures to process directed acyclic graphs [63, 25]. These ideas laid the foundations for the adaptive processing of cyclic graphs by the recurrent Graph Neural Network [59] and the feedforward Neural Network for Graphs [46], which today's DGNs build upon. Both methods iteratively compute embeddings of the graphs' entities (also called nodes) via a local and iterative message passing mechanism [27] that propagates the information through the graph. In recent years, many neural and probabilistic DGNs have emerged [50, 36, 31, 69, 74, 3] bridging ideas from different fields of machine learning; we set up our analysis in the context of these message-passing architectures. Even more recently, transformer models have begun to appear in graph-related tasks as well [19, 37, 75]. Akin to kernel methods for graphs [71, 38], this class of methods mainly relies on feature engineering to extract rich information from the input graph, and some perform very well at molecular tasks [41]. However, the architecture of transformers is not intrinsically more expressive than DGNs, and their effectiveness depends on the specific encodings used [48]. Therefore, gaining a better understanding of the inductive bias of DGNs remains a compelling research question.
The construction of NN graphs found recent application in predicting the mortality of patients, by connecting them according to specific attributes of the electronic health records [44, 65]. In addition, it was used in natural language processing to connect messages and news with similar contents to tackle spam and fake news detection, respectively [39, 7]. In both cases, the authors computed similarity based on some embedding representation of the text, and the terms' frequency in a document was the graph-building criterion for a generic document classification task [56]. Finally,  $k$ -NN graphs have also been built based on chest computerized tomography similarity for early diagnoses of COVID-19 [77, 42] or to understand if debts will be repaid in time [29].
Most of the theoretical works on DGNs deal with the problems of over-smoothing [40, 15, 14, 11] and over-squashing [1, 66, 28] of learned representations, as well as the discriminative power of such models [47, 74, 26, 10]. In this context, researchers mostly agreed that DGNs based on message passing perform favorably for homophilic graphs and not so much for heterophilic ones [79]. However, recent works suggest a different perspective; the generalization performances depend more on the neighborhood distributions of nodes belonging to different classes [43] and on a good choice of the model's weights [67]. The Cross-Class Neighborhood Similarity (CCNS) Ma et al. [43] stands out as an effective (but purely empirical) strategy to understand whether a graph structure is useful or not for a node classification task. In our work, we take inspiration from the CCNS to study the behavior of the neighborhood class label distributions around nodes and compute the first approximation and lower bound of the CCNS for NN graphs without the need to explicitly build them.
Structure learning and graph rewiring are also related but orthogonal topics. Rather than precomputing a fixed structure, these approaches discover latent dependencies between samples [24, 72] and can enrich the original graph structure when this is available [17, 76, 66, 34]. They have been applied in contexts of scarce supervision, where a  $k$ -NN graph proved to be an effective baseline when combined with DGNs [23]. At the same time, the combinatorial nature of graphs makes it difficult and expensive to explore the space of all possible structures, making the a priori construction of the graph a sensible and efficient alternative.
We begin by introducing notions and assumptions that will be used throughout the paper. Our starting point is a classification task over a set of classes  $\mathcal{C}$ , where each sample  $u$  is associated with a vector of  $D$  attributes  $\pmb{x}_u \in \mathbb{R}^D$  and a target class label  $y_u \in \mathcal{C}$ .
A graph  $g$  of size  $N$  is a quadruple  $(\mathcal{V},\mathcal{E},\mathcal{X},\mathcal{Y})$ , where  $\mathcal{V} = \{1,\dots ,N\}$  represents the set of nodes and  $\mathcal{E}$  is the set of directed edges  $(u,v)$  from node  $u$  to node  $v$ . The symbol  $\mathcal{X} = \{X_u,\forall u\in \mathcal{V}\}$  defines the set of random variables (r.v.) with realizations  $\pmb {x}_u\in \mathbb{R}^D,u\in \mathcal{V}$ . The same definition applies to the set of target variables  $\mathcal{Y} = \{Y_u,\forall u\in \mathcal{V}\}$  and their realizations  $y_{u}\in \mathcal{C}$ ,  $u\in \mathcal{V}$ . When we talk about the subset of nodes with target label  $c\in \mathcal{C}$ , we use the symbol  $\nu_{c}$ ; also, we define the neighborhood of node  $u$  as  $\mathcal{N}_u = \{v\in \mathcal{V}|(v,u)\in \mathcal{E}\}$ .
A Gaussian (or normal) univariate distribution with mean and standard deviation  $\mu, \sigma \in \mathbb{R}, \sigma > 0$  is represented as  $\mathcal{N}(\cdot; \mu, \sigma^2)$ , using  $\pmb{\mu} \in \mathbb{R}^{D}$ ,  $\pmb{\Sigma} \in \mathbb{R}^{D \times D}$  for the multivariate case. The probability density function (p.d.f.) of a univariate normal  $r.v.$  parametrized by  $\mu, \sigma$  is denoted by  $\phi(\cdot)$  together with its cumulative density function (c.d.f.)  $F(w) = \Phi\left(\frac{w - \mu}{\sigma}\right) = \frac{1}{2}\left(1 + \operatorname{erf}\left(\frac{w - \mu}{\sigma\sqrt{2}}\right)\right)$ , where erf is the error function. Subscripts will denote quantities related to a specific random variable.
We want to transform the initial task into a node classification problem, where different i.i.d. samples become the nodes of a single graph, and the edges are computed by some nearest neighbor algorithm. We assume (assumption 1) the true data distribution  $p(X = x)$  of each sample (thus abstracting from  $u$ ) is defined by the graphical model of Figure 1 (left). This model reflects the common assumption that observables of a given class share the same distribution [30, 70], and we choose this distribution to be a mixture of Gaussian distributions. Such a choice is not restrictive since, in principle, with enough mixture components it is possible to approximate almost any continuous density [9]. Note that it is also possible to sample discrete attribute values by rounding the samples of a properly parametrized Gaussian mixture. Formally, we consider  $|\mathcal{C}|$  latent classes modeled by a categorical  $r.v. C$  with prior distribution  $p(C = c)$  and  $|\mathcal{M}|$  mixtures for each class modeled by  $M \sim p(M = m|C = c), m \in \mathcal{M}, c \in \mathcal{C}$ , such that  $p(\boldsymbol{x}|c) = \sum_{m=1}^{\mathcal{M}} p(\boldsymbol{x}|m, c)p(m|c)$ . In Section 3.1, we further require (assumption 2) that the
Figure 1: Left: we represent assumption 1 on the data distribution through a graphical model, i.e., a hierarchical mixture of distributions, where observed variables are shaded and latent ones are white circles. Right: a visual example of a data distribution generated by the graphical model, with an intuitive visualization of the class posterior mass around a point  $\boldsymbol{x}_u$ .
$D$  observables are conditionally independent when the class  $c$  and the mixture  $m$  are known, i.e.,  $p(X = \boldsymbol{x}) = \sum_{c=1}^{|C|} p(c) \sum_{m=1}^{|M|} p(m|c) \prod_{f=1}^{D} p(x_f|m,c)$ . We refer the reader to Section 3.3 for a discussion about this assumption. Figure 1 (right) depicts an example of one such data distribution for  $D = 1$ . Hereinafter we shall have  $p(X_f = x_f|m,c) = \mathcal{N}(x_f; \mu_{cmf}, \sigma_{cmf}^2)$  for the individual attributes and  $p(X = \boldsymbol{x}|m,c) = \mathcal{N}(\boldsymbol{x}; \boldsymbol{\mu}_{mc}, \boldsymbol{\Sigma}_{mc})$  for the multivariate case. We will also use diagonal covariance matrices denoted as  $\Lambda_{mc} = \text{diag}(\sigma_{mc1}^2, \ldots, \sigma_{mcD}^2)$ . Finally, akin to [43], we assume that the neighbors of node  $v$  are i.i.d. (assumption 3), meaning their attributes and class distributions only depend on the properties of node  $v$ ; such an assumption is especially reasonable when one considers how nearest neighbor structures are typically built, by picking a set of neighbors based only on their distance to the target node  $v$ .
To talk about the surroundings of a point in space we will use the notion of hypercubes (or D-cubes). A hypercube of dimension  $D$  centered at point  $\pmb{x} \in \mathbb{R}^D$  of side length  $\varepsilon > 0$  is the set of points given by the Cartesian product  $H_{\varepsilon}(\pmb{x}) = [x_1 - \frac{\varepsilon}{2}, x_1 + \frac{\varepsilon}{2}] \times \dots \times [x_D - \frac{\varepsilon}{2}, x_D + \frac{\varepsilon}{2}]$ .
3.1 Analytical Computation of the CCNS for Nearest Neighbor Graphs
The CCNS [43] computes how similar the neighborhoods of two distinct nodes are in terms of class label distribution, and it provides an aggregated result over pairs of target classes. Intuitively, if nodes belonging to distinct classes happen to have similar neighboring class label distributions, then it will be unlikely that a classifier will correctly discriminate between these two nodes after a message passing operation because the nodes' embeddings will look very similar. On the other hand, nodes of different classes with very different neighboring class label distributions will be easier to separate. This intuition implies that nodes of different classes typically have different attribute values.
Definition 3.1 (Cross Class Neighborhood Similarity, extended from [43]). Given a graph  $g$ , the cross-class neighborhood similarity between classes  $c, c' \in \mathcal{C}$  is given by
$$
s \left(c, c ^ {\prime}\right) = \mathbb {E} _ {p \left(\boldsymbol {x} \mid c\right) p \left(\boldsymbol {x} ^ {\prime} \mid c ^ {\prime}\right)} \left[ \Omega \left(q _ {c} (\boldsymbol {x}), q _ {c ^ {\prime}} \left(\boldsymbol {x} ^ {\prime}\right)\right) \right] \tag {1}
$$
where  $\Omega$  computes a (dis)similarity score between vectors and the function  $q_{c}:\mathbb{R}^{D}\to \mathbb{R}^{|C|}$  (resp.  $q_{c'}$ ) computes the probability vector that a node of class  $c$  (resp.  $c'$ ) with attributes  $\pmb{x}$  (resp.  $\pmb{x}'$ ) has a neighbor of class  $c''$ , for every  $c''\in \mathcal{C}$ .
The definition of  $q_{c}$  and  $q_{c'}$  is the key ingredient of Equation 1. In the following, we are going to show that it is possible to analytically compute these quantities when we assume an NN structure. For every norm-induced metric, which is also convex, the lower bound of Equation 1 follows from Jensen's inequality and from the linearity of expectation:
$$
s (c, c ^ {\prime}) \geq | | \mathbb {E} _ {p (\boldsymbol {x} | c) p \left(\boldsymbol {x} ^ {\prime} \mid c ^ {\prime}\right)} \left[ q _ {c} (\boldsymbol {x}) - q _ {c ^ {\prime}} \left(\boldsymbol {x} ^ {\prime}\right) \right] | | = | | \mathbb {E} _ {p (\boldsymbol {x} | c)} \left[ q _ {c} (\boldsymbol {x}) \right] - \mathbb {E} _ {p \left(\boldsymbol {x} ^ {\prime} \mid c ^ {\prime}\right)} \left[ q _ {c ^ {\prime}} \left(\boldsymbol {x} ^ {\prime}\right) \right] | |. \tag {2}
$$
This bound assigns non-zero values to the inter-class neighborhood similarity, but as we shall see one can resort to Monte Carlo approximations of Equation 1 to estimate both the inter and intra-class scores. From now on, we will use the Euclidean distance as our  $\Omega$ .
We first study the class label distribution in the surroundings of some node  $u$ . In the example of Figure 1 (right), we consider a binary classification problem with  $|\mathcal{M}| = 2$  and depict the conditional distributions  $p(\pmb{x}_u|C = 0)$ ,  $p(\pmb{x}_u|C = 1)$  with green and blue curves, respectively. A dashed black line, instead, represents  $p(\pmb{x})$  assuming a non-informative class prior. If the neighbors of  $u$  belong to the hypercube  $H_{\varepsilon}(\pmb{x}_u)$  for some  $\varepsilon$ , then the probability that a neighbor will belong to class  $c$  depends on how much class-specific probability mass, that is the shaded green and blue areas, there is in the hypercube. Since the blue area is larger than the green one, finding a neighbor of class 1 is more likely to happen. Formally, we define the (unnormized) probability of a neighbor belonging to class  $c$  in a given hypercube as the weighted posterior mass of  $C$  contained in that hypercube.
Definition 3.2 (Posterior Mass  $M_{\pmb{x}}(c,\varepsilon)$  Around Point  $\pmb{x}$ ). Given a hypercube  $H_{\varepsilon}(\pmb{x})$  centered at point  $\pmb{x} \in \mathbb{R}^{D}$ , and a class  $c \in \mathcal{C}$ , the posterior mass  $M_{\pmb{x}}(c,\varepsilon)$  is the unnormalized probability that a point in the hypercube has class  $c$ :
$$
M _ {\boldsymbol {x}} (c, \varepsilon) = \mathbb {E} _ {\boldsymbol {w} \in H _ {\varepsilon} (\boldsymbol {x})} [ p (c | \boldsymbol {w}) ] = \int_ {\boldsymbol {w} \in H _ {\varepsilon} (\boldsymbol {x})} p (c | \boldsymbol {w}) p (\boldsymbol {w}) d \boldsymbol {w} = p (c) \int_ {\boldsymbol {w} \in H _ {\varepsilon} (\boldsymbol {x})} p (\boldsymbol {w} | c) d \boldsymbol {w}, \tag {3}
$$
where the last equality follows from Bayes' theorem.
The following proposition shows how to compute  $M_{\pmb{x}}(c,\varepsilon)$  analytically. The interested reader can find all proofs of our paper in Section A.3.
Proposition 3.3. Under assumptions 1,2,  $M_{\pmb{x}}(c,\varepsilon)$  has the following analytical form
$$
p (c) \sum_ {m = 1} ^ {| \mathcal {M} |} p (m | c) \prod_ {f = 1} ^ {D} \left(F _ {Z _ {c m f}} \left(x _ {f} + \frac {\varepsilon}{2}\right) - F _ {Z _ {c m f}} \left(x _ {f} - \frac {\varepsilon}{2}\right)\right), \tag {4}
$$
where  $Z_{cmf}$  is a r.v. with Gaussian distribution  $\mathcal{N}(\cdot ;\mu_{cmf},\sigma_{cmf}^2)$ .
To reason about an entire class rather than individual samples, therefore being able to compute the two quantities on the right-hand side of Equation 2, we extend the previous definition by taking into account all samples of class  $c' \in \mathcal{C}$ . Thus, we seek to compute the average unnormalized probability that a sample belongs to class  $c$  in the hypercubes centered around samples of class  $c'$ .
Definition 3.4 (Expected Class  $c$  Posterior Mass  $M_{c'}(c, \varepsilon)$  for Samples of Class  $c'$ ). Given a hypercube length  $\varepsilon$  and a class  $c' \in \mathcal{C}$ , the unnormalized probability that a sample of class  $c'$  has another sample of class  $c$  lying in its hypercube is defined as
$$
M _ {c ^ {\prime}} (c, \varepsilon) \stackrel {\text {d e f}} {=} \mathbb {E} _ {\boldsymbol {x} \sim p (\boldsymbol {x} \mid c ^ {\prime})} [ M _ {\boldsymbol {x}} (c, \varepsilon) ]. \tag {5}
$$
It is also possible to compute  $M_{c'}(c, \varepsilon)$  in closed form, which we show in the next theorem.
Theorem 3.5. Under assumptions  $1,2$ ,  $M_{c'}(c,\varepsilon)$  has the following analytical form
$$
p (c) \sum_ {\substack {m = 1 \\ m ^ {\prime} = 1}} ^ {| \mathcal {M} |} p (m ^ {\prime} | c ^ {\prime}) p (m | c) \prod_ {f = 1} ^ {D} \left(\Phi \left(\frac {\mu_ {c ^ {\prime} m ^ {\prime} f} + \frac {\varepsilon}{2} - \mu_ {c m f}}{\sqrt {\sigma_ {c m f} ^ {2} + \sigma_ {c ^ {\prime} m ^ {\prime} f} ^ {2}}}\right) - \Phi \left(\frac {\mu_ {c ^ {\prime} m ^ {\prime} f} - \frac {\varepsilon}{2} - \mu_ {c m f}}{\sqrt {\sigma_ {c m f} ^ {2} + \sigma_ {c ^ {\prime} m ^ {\prime} f} ^ {2}}}\right)\right). \tag{6}
$$
Thanks to Theorem 3.5, we know how much class-  $c$  posterior probability mass we have, on average, around samples of class  $c'$ . To get a proper class  $c'$ -specific distribution over neighboring class labels, we apply a normalization step using the fact that  $M_{c'}(c,\varepsilon) \geq 0 \forall c \in \mathcal{C}$ .
Definition 3.6 ( $\varepsilon$ -Neighboring Class Distribution). Given a class  $c'$  and an  $\varepsilon > 0$ , the neighboring class distribution around samples of class  $c'$  is modeled as
$$
p _ {c ^ {\prime}} (c, \varepsilon) = \frac {M _ {c ^ {\prime}} (c , \varepsilon)}{\sum_ {i = 1} ^ {| c |} M _ {c ^ {\prime}} (i , \varepsilon)}. \tag {7}
$$
This distribution formalizes the notion that, in a neighborhood around points of class  $c'$ , the probability that points belong to class  $c$  does not necessarily match the true prior distribution  $p(C = c)$ . However, this becomes false when we consider an infinitely-large hyper-cube, as we show in the next result.
Proposition 3.7. Let the first  $D$  derivatives of  $\sum_{i=1}^{|C|} M_{c'}(i)$  be different from 0 in an open interval  $I$  around  $\varepsilon = 0$  and not tending to infinity for  $\varepsilon \to 0$ . Under assumptions 1,2, Equation 7 has limits
$$
\lim  _ {\varepsilon \rightarrow 0} p _ {c ^ {\prime}} (c, \varepsilon) = \frac {p (c) \sum_ {m , m ^ {\prime}} ^ {| \mathcal {M} |} p (m | c) p \left(m ^ {\prime} \mid c ^ {\prime}\right) \prod_ {f = 1} ^ {D} \phi_ {Z _ {c m m ^ {\prime} f}} (0)}{\sum_ {i} ^ {| \mathcal {C} |} p (i) \sum_ {m , m ^ {\prime}} ^ {| \mathcal {M} |} p (m | i) p \left(m ^ {\prime} \mid c ^ {\prime}\right) \prod_ {f = 1} ^ {D} \phi_ {Z _ {i m m ^ {\prime} f}} (0)}, \quad \lim  _ {\varepsilon \rightarrow \infty} p _ {c ^ {\prime}} (c, \varepsilon) = p (c) \tag {8}
$$
where  $Z_{imm'f}$  has distribution  $\mathcal{N}(\cdot; -a_{imm'}f, b_{imm'f}^2)$ ,