Title: On Class Distributions Induced by Nearest Neighbor Graphs for Node Classification of Tabular Data
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep Graph Networks

Graph Neural Networks

Graph Representation Learning

Nearest Neighbors

Node Classification

Tabular Data
Abstract: Researchers have used nearest neighbor graphs to transform classical machine learning problems on tabular data into node classification tasks to solve with graph representation learning methods. Such artificial structures often reflect the homophily assumption, believed to be a key factor in the performances of deep graph networks. In light of recent results demystifying these beliefs, we introduce a theoretical framework to understand the benefits of Nearest Neighbor (NN) graphs when a graph structure is missing. We formally analyze the Cross-Class Neighborhood Similarity (CCNS), used to empirically evaluate the usefulness of structures, in the context of nearest neighbor graphs. Moreover, we study the class separability induced by deep graph networks on a k-NN graph. Motivated by the theory, our quantitative experiments demonstrate that, under full supervision, employing a k-NN graph offers no benefits compared to a structure-agnostic baseline. Qualitative analyses suggest that our framework is good at estimating the CCNS and hint at k-NN graphs never being useful for such classification tasks under full supervision, thus advocating for the study of alternative graph construction techniques in combination with deep graph networks.

On Class Distributions Induced by Nearest Neighbor Graphs for Node Classification of Tabular Data

Abstract

Researchers have used nearest neighbor graphs to transform classical machine learning problems on tabular data into node classification tasks to solve with graph representation learning methods. Such artificial structures often reflect the homophily assumption, believed to be a key factor in the performances of deep graph networks. In light of recent results demystifying these beliefs, we introduce a theoretical framework to understand the benefits of Nearest Neighbor (NN) graphs when a graph structure is missing. We formally analyze the Cross-Class Neighborhood Similarity (CCNS), used to empirically evaluate the usefulness of structures, in the context of nearest neighbor graphs. Moreover, we study the class separability induced by deep graph networks on a k -NN graph. Motivated by the theory, our quantitative experiments demonstrate that, under full supervision, employing a k -NN graph offers no benefits compared to a structure-agnostic baseline. Qualitative analyses suggest that our framework is good at estimating the CCNS and hint at k -NN graphs never being useful for such classification tasks under full supervision, thus advocating for the study of alternative graph construction techniques in combination with deep graph networks.

1 Introduction

The pursuit of understanding real-world phenomena has often led researchers to model the system of interest as a set of interdependent constituents, which influence each other in complex ways. In disciplines such as chemistry, physics, and network science, graphs are a convenient and wellstudied mathematical object to represent such interacting entities and their attributes. In machine learning, the term 'graph representation learning' refers to methods that can automatically leverage graph-structured data to solve tasks such as entity (or node), link, and whole-graph predictions [12, 32, 78, 73, 4].

Most of these methods assume that the relational information, that is the connections between entities, naturally emerges from the domain of the problem and is thus known. There is also broad consensus that connected entities typically share characteristics, behavioral patterns, or affiliation, something known as the homophily assumption [45]. This is possibly why, when the structure is not available, researchers have tried to artificially build Nearest Neighbor (NN) graphs from tabular data, by connecting entities based on some attribute similarity criterion, with applications in healthcare [44, 65], fake news and spam detection [39, 7], biology [77, 42], and document classification [56] to name a few. From an information-theoretic perspective, the creation of such graphs does not add new information as it depends on the available data; that said, what makes their use plausible is that the graph construction is a form of feature engineering that often encodes the homophily assumption. Combined with the inductive bias of Deep Graph Networks (DGNs) [59, 46], this strategy aims at improving the generalization performances on tabular data compared to structure-agnostic baselines, for example, a Multi-Layer Perceptron (MLP).

Indeed, using a k -NN graph [21] has recently improved the node classification performances under the scarcity of training labels [24, 23]. This is also known as the semi-supervised setting, where one can access the features of all nodes but the class labels are available for a handful of those. A potential explanation for these results is that, by incorporating neighboring values into each entity's representation, the neighborhood aggregation performed by DGNs acts as a regularization strategy that prevents the classifier from overfitting the attributes of the few labeled nodes, similar to input jittering [57, 62, 6]. However, it is still unclear what happens when one has access to all training labels (hereinafter the fully-supervised setting), namely if these graph-building strategies grant a statistically significant advantage in generalization compared to a structure-agnostic baseline. In this respect, proper comparisons against such baselines are often lacking or unclear in previous works, an issue that has also been reported in recent papers about the reproducibility of node and graph classification experiments [61, 22, 54].

In addition, it was recently shown [43] that homophily is not required to achieve good classification performances in node classification tasks; rather, what truly matters is how different the neighborhood class label distributions of nodes of separate classes are. This resulted in the definition of the empirical Cross-Class Neighborhood Similarity (CCNS) [43, 13], an object that estimates such similarities based on the available connectivity structure. Yet, whether or not artificially built graphs can be useful for the task at hand has mainly remained an empirical question, and more theoretical conditions for which this happens are still not understood.

In this paper, we introduce a theoretical framework to approach this question, and we provide two analyses of independent interest. Inspired by the CCNS, Section 3.1 studies the neighboring class label distribution of nearest neighbor graphs. Then Section 3.2 deals with the distribution of entity embeddings induced by DGNs on a k -NN graph, and we use it to quantify class separability in both the input and the embedding spaces. Overall, our theoretical results suggest that building a k -NN graph is not a good idea. To validate the theory with empirical evidence, we robustly compare (Sections 4 and 5) four baselines across 11 tabular datasets to check that the k -NN graph construction does not give statistically significant advantages in the fully-supervised setting. In addition, we reverse-engineer the theory to learn data distributions that would make a k -NN graph useful in practice when combined with DGNs. From the empirical results, we conjecture that this is never the case. Therefore, we hope to raise awareness in the graph machine learning community about the need for alternative (either NN-based or not) graph construction techniques.

In summary, our contributions are: i) under some assumptions, we can estimate the CCNS for nearest neighbor graphs and provide its first lower bound; ii) we study how applying a simple DGN to an artificial k -NN graph affects the class separability of the input data; iii) we carry out a robust comparison between structure-agnostic and structure-aware baselines on a set of 11 datasets that is in agreement with our theoretical results; iv) qualitative analyses based on the theory further suggest that using the k -NN graph might not be advisable.

2 Related Work

The early days of graph representation learning date back to the end of the previous century, when researchers proposed backpropagation through structures to process directed acyclic graphs [63, 25]. These ideas laid the foundations for the adaptive processing of cyclic graphs by the recurrent Graph Neural Network [59] and the feedforward Neural Network for Graphs [46], which today's DGNs build upon. Both methods iteratively compute embeddings of the graphs' entities (also called nodes) via a local and iterative message passing mechanism [27] that propagates the information through the graph. In recent years, many neural and probabilistic DGNs have emerged [50, 36, 31, 69, 74, 3] bridging ideas from different fields of machine learning; we set up our analysis in the context of these message-passing architectures. Even more recently, transformer models have begun to appear in graph-related tasks as well [19, 37, 75]. Akin to kernel methods for graphs [71, 38], this class of methods mainly relies on feature engineering to extract rich information from the input graph, and some perform very well at molecular tasks [41]. However, the architecture of transformers is not intrinsically more expressive than DGNs, and their effectiveness depends on the specific encodings used [48]. Therefore, gaining a better understanding of the inductive bias of DGNs remains a compelling research question.

The construction of NN graphs found recent application in predicting the mortality of patients, by connecting them according to specific attributes of the electronic health records [44, 65]. In addition, it was used in natural language processing to connect messages and news with similar contents to tackle spam and fake news detection, respectively [39, 7]. In both cases, the authors computed similarity based on some embedding representation of the text, and the terms' frequency in a document was the graph-building criterion for a generic document classification task [56]. Finally, k -NN graphs have also been built based on chest computerized tomography similarity for early diagnoses of COVID-19 [77, 42] or to understand if debts will be repaid in time [29].

Most of the theoretical works on DGNs deal with the problems of over-smoothing [40, 15, 14, 11] and over-squashing [1, 66, 28] of learned representations, as well as the discriminative power of such models [47, 74, 26, 10]. In this context, researchers mostly agreed that DGNs based on message passing perform favorably for homophilic graphs and not so much for heterophilic ones [79]. However, recent works suggest a different perspective; the generalization performances depend more on the neighborhood distributions of nodes belonging to different classes [43] and on a good choice of the model's weights [67]. The Cross-Class Neighborhood Similarity (CCNS) Ma et al. [43] stands out as an effective (but purely empirical) strategy to understand whether a graph structure is useful or not for a node classification task. In our work, we take inspiration from the CCNS to study the behavior of the neighborhood class label distributions around nodes and compute the first approximation and lower bound of the CCNS for NN graphs without the need to explicitly build them.

Structure learning and graph rewiring are also related but orthogonal topics. Rather than precomputing a fixed structure, these approaches discover latent dependencies between samples [24, 72] and can enrich the original graph structure when this is available [17, 76, 66, 34]. They have been applied in contexts of scarce supervision, where a k -NN graph proved to be an effective baseline when combined with DGNs [23]. At the same time, the combinatorial nature of graphs makes it difficult and expensive to explore the space of all possible structures, making the a priori construction of the graph a sensible and efficient alternative.

3 Methodology

We begin by introducing notions and assumptions that will be used throughout the paper. Our starting point is a classification task over a set of classes C , where each sample u is associated with a vector of D attributes x u ∈ R D and a target class label y u ∈ C .

A graph g of size N is a quadruple ( V , E , X , Y ) , where V = { 1 , . . . , N } represents the set of nodes and E is the set of directed edges ( u, v ) from node u to node v . The symbol X = { X u , ∀ u ∈ V} defines the set of random variables ( r.v. ) with realizations x u ∈ R D , u ∈ V . The same definition applies to the set of target variables Y = { Y u , ∀ u ∈ V} and their realizations y u ∈ C , u ∈ V . When we talk about the subset of nodes with target label c ∈ C , we use the symbol V c ; also, we define the neighborhood of node u as N u = { v ∈ V| ( v, u ) ∈ E} .

A Gaussian (or normal) univariate distribution with mean and standard deviation µ, σ ∈ R , σ > 0 is represented as N ( · ; µ, σ 2 ) , using µ ∈ R D , Σ ∈ R D × D for the multivariate case. The probability density function ( p.d.f. ) of a univariate normal r.v. parametrized by µ, σ is denoted by ϕ ( · ) together with its cumulative density function ( c.d.f. ) F ( w ) = Φ ( w -µ σ ) = 1 2 (1 + erf ( w -µ σ √ 2 )) , where erf is the error function. Subscripts will denote quantities related to a specific random variable.

We want to transform the initial task into a node classification problem, where different i.i.d. samples become the nodes of a single graph, and the edges are computed by some nearest neighbor algorithm. We assume ( assumption 1 ) the true data distribution p ( X = x ) of each sample (thus abstracting from u ) is defined by the graphical model of Figure 1 (left). This model reflects the common assumption that observables of a given class share the same distribution [30, 70], and we choose this distribution to be a mixture of Gaussian distributions. Such a choice is not restrictive since, in principle, with enough mixture components it is possible to approximate almost any continuous density [9]. Note that it is also possible to sample discrete attribute values by rounding the samples of a properly parametrized Gaussian mixture. Formally, we consider |C| latent classes modeled by a categorical r.v. C with prior distribution p ( C = c ) and |M| mixtures for each class modeled by M ∼ p ( M = m | C = c ) , m ∈ M , c ∈ C , such that p ( x | c ) = ∑ M m =1 p ( x | m,c ) p ( m | c ) . In Section 3.1, we further require ( assumption 2 ) that the

M

X

C

data model

p(x|C = 0)

p(x|C = 1)

p(x)

xu-=

xu+=

x

2

2

D observables are conditionally independent when the class c and the mixture m are known, i.e., p ( X = x ) = ∑ |C| c =1 p ( c ) ∑ |M| m =1 p ( m | c ) ∏ D f =1 p ( x f | m,c ) . We refer the reader to Section 3.3 for a discussion about this assumption. Figure 1 (right) depicts an example of one such data distribution for D = 1 . Hereinafter we shall have p ( X f = x f | m,c ) = N ( x f ; µ cmf , σ 2 cmf ) for the individual attributes and p ( X = x | m,c ) = N ( x ; µ mc , Σ mc ) for the multivariate case. We will also use diagonal covariance matrices denoted as Λ mc = diag ( σ 2 mc 1 , . . . , σ 2 mcD ) . Finally, akin to [43], we assume that the neighbors of node v are i.i.d. ( assumption 3 ), meaning their attributes and class distributions only depend on the properties of node v ; such an assumption is especially reasonable when one considers how nearest neighbor structures are typically built, by picking a set of neighbors based only on their distance to the target node v .

To talk about the surroundings of a point in space we will use the notion of hypercubes (or D-cubes). A hypercube of dimension D centered at point x ∈ R D of side length ε > 0 is the set of points given by the Cartesian product H ε ( x ) = [ x 1 -ε 2 , x 1 + ε 2 ] ×··· × [ x D -ε 2 , x D + ε 2 ] .

3.1 Analytical Computation of the CCNS for Nearest Neighbor Graphs

The CCNS [43] computes how similar the neighborhoods of two distinct nodes are in terms of class label distribution, and it provides an aggregated result over pairs of target classes. Intuitively, if nodes belonging to distinct classes happen to have similar neighboring class label distributions, then it will be unlikely that a classifier will correctly discriminate between these two nodes after a message passing operation because the nodes' embeddings will look very similar. On the other hand, nodes of different classes with very different neighboring class label distributions will be easier to separate. This intuition implies that nodes of different classes typically have different attribute values.

Definition 3.1 (Cross Class Neighborhood Similarity, extended from [43]) . Given a graph g , the cross-class neighborhood similarity between classes c, c ′ ∈ C is given by

where Ω computes a (dis)similarity score between vectors and the function q c : R D → R |C| (resp q c ′ ) computes the probability vector that a node of class c (resp. c ′ ) with attributes x (resp. x ′ ) has a neighbor of class c ′′ , for every c ′′ ∈ C .

The definition of q c and q c ′ is the key ingredient of Equation 1. In the following, we are going to show that it is possible to analytically compute these quantities when we assume an NN structure. For every norm-induced metric, which is also convex, the lower bound of Equation 1 follows from Jensen's inequality and from the linearity of expectation:

This bound assigns non-zero values to the inter-class neighborhood similarity, but as we shall see one can resort to Monte Carlo approximations of Equation 1 to estimate both the inter and intra-class scores. From now on, we will use the Euclidean distance as our Ω .

We first study the class label distribution in the surroundings of some node u . In the example of Figure 1 (right), we consider a binary classification problem with |M| = 2 and depict the conditional distributions p ( x u | C = 0) , p ( x u | C = 1) with green and blue curves, respectively. A dashed black line, instead, represents p ( x ) assuming a non-informative class prior. If the neighbors of u belong to the hypercube H ε ( x u ) for some ε , then the probability that a neighbor will belong to class c depends on how much class-specific probability mass, that is the shaded green and blue areas, there is in the hypercube. Since the blue area is larger than the green one, finding a neighbor of class 1 is more likely to happen. Formally, we define the (unnormalized) probability of a neighbor belonging to class c in a given hypercube as the weighted posterior mass of C contained in that hypercube.

Definition 3.2 (Posterior Mass M x ( c, ε ) Around Point x ) . Given a hypercube H ε ( x ) centered at point x ∈ R D , and a class c ∈ C , the posterior mass M x ( c, ε ) is the unnormalized probability that a point in the hypercube has class c :

where the last equality follows from Bayes' theorem.

The following proposition shows how to compute M x ( c, ε ) analytically. The interested reader can find all proofs of our paper in Section A.3.

Proposition 3.3. Under assumptions 1,2, M x ( c, ε ) has the following analytical form

where Z cmf is a r.v. with Gaussian distribution N ( · ; µ cmf , σ cmf ) .

To reason about an entire class rather than individual samples, therefore being able to compute the two quantities on the right-hand side of Equation 2, we extend the previous definition by taking into account all samples of class c ′ ∈ C . Thus, we seek to compute the average unnormalized probability that a sample belongs to class c in the hypercubes centered around samples of class c ′ .

Definition 3.4 (Expected Class c Posterior Mass M c ′ ( c, ε ) for Samples of Class c ′ ) . Given a hypercube length ε and a class c ′ ∈ C , the unnormalized probability that a sample of class c ′ has another sample of class c lying in its hypercube is defined as

It is also possible to compute M c ′ ( c, ε ) in closed form, which we show in the next theorem.

Theorem 3.5. Under assumptions 1,2, M c ′ ( c, ε ) has the following analytical form

Thanks to Theorem 3.5, we know how much classc posterior probability mass we have, on average, around samples of class c ′ . To get a proper class c ′ -specific distribution over neighboring class labels, we apply a normalization step using the fact that M c ′ ( c, ε ) ≥ 0 ∀ c ∈ C .

Definition 3.6 ( ε -Neighboring Class Distribution) . Given a class c ′ and an ε > 0 , the neighboring class distribution around samples of class c ′ is modeled as

This distribution formalizes the notion that, in a neighborhood around points of class c ′ , the probability that points belong to class c does not necessarily match the true prior distribution p ( C = c ) . However, this becomes false when we consider an infinitely-large hyper-cube, as we show in the next result.

Proposition 3.7. Let the first D derivatives of ∑ |C| i =1 M c ′ ( i ) be different from 0 in an open interval I around ε = 0 and not tending to infinity for ε → 0 . Under assumptions 1,2, Equation 7 has limits

The choice of ε , which implicitly encodes our definition of 'nearest' neighbor, plays a crucial role in determining the distribution of a neighbor's class label. When the hypercube is too big, the probability that a neighbor has class c matches the true prior p ( c ) regardless of the class c ′ , that is we make a crude assumption about the neighbor's class distribution of a sample. If we instead consider a smaller hypercube, then we observe a less trivial behavior and the probability p c ′ ( c, ε ) is directly proportional to the distances between the means µ c ′ m ′ and µ cm , as one would intuitively expect for simple examples.

To summarize, we can use p c ′ ( c, ε ) as an approximation for E p ( x | c ′ ) [ q ′ c ( x )] of Equation 2; similarly, we can use a normalized version of M c ( x ) in place of q c ( x ) to estimate Equation 1 via Monte Carlo sampling without the need of building a NN graph. For space reasons, in Appendix A.5, we qualitatively discuss the results for a 1-dimensional instance of a data distribution, and we investigate the quality of the CCNS approximation while varying k .

3.2 Class Separability Induced by DGNs on a k -NN Graph

This section formally investigates the properties of the embedding space created by DGNs under the k -NN graph. Our goal is to understand whether using such a graph can improve the separability between samples belonging to different classes or not. Provided that our assumptions hold, both conclusions would be interesting: if the k -NN graph helps, then we know the conditions for that to happen; if that is not the case, we have identified and formalized the need for new graph construction mechanisms.

Akin to previous works [40, 16, 5, 43], we consider a linear 1-layer DGN with the following neighborhood aggregation scheme that computes node embeddings h u ∈ R D ∀ u ∈ V :

The node embedding of sample u is then fed into a standard machine learning classifier, e.g., an MLP. As done in previous works, we assume that a linear (learnable) transformation W ∈ R D × D of the input x v , often used in DGN models such as the Neural Network for Graphs [46] and the Graph Convolutional Network [36], is absorbed by the subsequent classifier.

Mimicking the behavior of the k -NN algorithm, which connects similar entities together, we model the attribute distribution of a neighbor v ∈ N u as a normal distribution N ( x v ; x u , diag ( σ 2 , . . . , σ 2 ) , where σ 2 is a hyper-parameter that ensures it is highly unlikely to sample neighbors outside of H ε ( x u ) ; from now on we use the symbol σ ε to make this connection clear. Under assumption 3 , neighbors' sampling is independently repeated k times and the attributes are averaged together. Therefore, we use statistical properties of normal distributions (please refer to Lemmas A.2 and A.1) to compute the resulting node u 's embedding distribution:

Therefore, the more neighbors a node has (i.e., higher k ) the more skewed the resulting distribution is around x u (i.e., lower variance). This is a fairly reasonable assumption if we think of an infinitely large dataset; informally, each of the k neighbors will be much more likely to lie near x u than near the surface of the hypercube, hence h u will be close to x u with high probability.

To understand how Equation 9 affects the separability of samples belonging to different classes, we compute a divergence score between the distributions p ( h | c ) and p ( h | c ′ ) [68, 51, 20, 8, 52]. When

this divergence is higher than that of the distributions p ( x | c ) and p ( x | c ′ ) , then the k -NN structure and the inductive bias of DGNs are helpful for our task. Below, we show that for two mixtures of Gaussians we can obtain the analytical form of their Squared Error Distance (SED), the simplest symmetric divergence defined as SED ( p, q ) = ∫ ( p ( x ) -q ( x )) 2 dx . This provides a concrete strategy to understand, regardless of training, if it would make sense to build a k -NN graph for our problem. 1

Proposition 3.8. Under assumptions 1,3, let us assume that the entity embeddings of a 1-layer DGN applied on an artificially built k -NN graph follow the distribution of Equation 10. Then the Squared Error Distances SED ( p ( h | c ) , p ( h | c ′ )) and SED ( p ( x | c ) , p ( x | c ′ )) have analytical forms.

In the interest of space, we have deferred the analytic formulas to the appendix.

As an immediate but important corollary, a k -NN graph improves the ability to distinguish samples of different classes c, c ′ if it holds that SED ( p ( h | c ) , p ( h | c ′ )) > SED ( p ( x | c ) , p ( x | c ′ )) . Indeed, if class distributions diverge more in the embedding space, which has the same dimensionality as the input space, then they will be easier to separate by a universal approximator such as an MLP. We use this corollary in our experiments, by reverse-engineering the theory to find out 'good' data models.

3.3 Limitations and Future Directions

It is extremely challenging to prove that a k -NN graph never helps to improve the classification performances using a DGN. The reason is that theoretical results of this kind rely, as is common [43], on assumptions of the true data generating distribution and on the specific assignments of parameters such as µ and Σ . The true data distribution is typically unknown and hard to find [68], but assumptions 1 and 3 remain quite reasonable due to the flexibility of mixture models and the nature of the nearest neighbors construction. In addition, assumption 2 has been widely used in the graphical models literature because it is particularly attractive from a theoretical perspective [64], despite it cannot cover all real-world cases; we note, however, that it plays no role in the analysis of Section 3.2. Finally, as done in the past [40, 15, 5, 43], we have ignored the non-linearity in the neighborhood aggregation of Equation 9. As we shall see in our empirical results, the intuitions we gained from the theoretical analysis seem to hold for non-linear models as well.

Our analysis has considered hyper-cubes in the interest of simplicity, but nearest neighbor graphs typically compute hyper-balls centered at a point. In high-dimensional spaces, the volume of hypercubes and hyper-balls can differ in non-trivial ways [18], so future work should improve on our results by replacing hyper-cubes with hyper-balls. Similarly, the results of Section 3.1 could be used to guide the definition of new graph construction strategies based on attribute similarity criteria, for instance by proposing a good ε for the data at hand. We leave this idea to future investigations.

4 Experiments

We conduct quantitative and qualitative experiments to support our theoretical insights. For all experiments we used a server with 32 cores, 128 GBs of RAM, and 4 GPUs with 11 GBs of memory.

Quantitative Experiments Quantitatively speaking, we want to compare a structure-agnostic baseline, namely an MLP [58], against different graph machine learning models, such as a simple DGN (sDGN) that implements Equation 9 followed by an MLP classifier, the Graph Isomorphism Network (GIN) [74] and the Graph Convolutional Network (GCN) [36]. The goal is to show that in the fully-supervised setting using a k -NN graph does not offer any concrete benefit. We consider 11 datasets, eight of which were taken from the UCI repository [2], namely Abalone, Adult, Dry Bean, Electrical Grid Stability, Isolet, Musk v2, Occupancy Detection, Waveform Database Generator v2, as well as the citation networks Cora, Citeseer [60], and Pubmed [49]. For each dataset, we build a k -NN graph, where k is a hyper-parameter, using the node attributes' similarity to find neighbors (discarding the original structure in the citation networks). We report some datasets statistics in Table 2, and the metric to optimize is the F1 score. Note that it is difficult to ascertain if our assumptions hold for these datasets because we have no access to the true data distribution; however, a robust comparison for this topic is lacking in the literature and can help us gather further insights on whether our ideas might also hold in a more general setting. For every dataset and model, we follow the rigorous and fair evaluation setup of [22]: we perform a 10 -fold cross validation for risk assessment,

with a hold-out model selection (90% training/10% validation) for each of the 10 external training folds. We report the hyper-parameters for the model selection phase in Table 3. For each of the 10 best models, we perform 3 final re-training runs and average their test performances on the corresponding external fold to mitigate bad initializations. The final score is the average of these 10 test results.

Qualitative Experiments We provide four qualitative experiments. First, we study how p c ( c ′ , ε ) changes for varying values of ε and different priors p ( c ) under specific class-conditional data distributions. Second, we analyze the first limit of Proposition 3.7 under the same conditional data distributions as before ( D = 1 hence assumptions are trivially satisfied). We translated one of the distributions by different values and computed the SED between them, together with the quantity p c ( c ′ , ε ) under different priors. The goal here is to show the relation between the SED and the limit. Third, we reverse-engineer the theoretical results to find, if possible, data distributions satisfying SED ( p ( h | c ) , p ( h | c ′ )) > SED ( p ( x | c ) , p ( x | c ′ )) . There is no dataset associated with this experiment, rather we learn the parameters of the graphical model of Figure 1 (left) together with ε, k, σ ε ( k is treated as a continuous value during the optimization) in order to minimize the following objective: L SED -λ L CCNS , where λ is a hyper-parameter. L SED sums SED ( p ( x | c ) , p ( x | c ′ )) -SED ( p ( h | c ) , p ( h | c ′ )) for all pairs of distinct classes c, c ′ , whereas L CCNS computes the lower bound for the inter-class similarity, thus acting as a regularizer that avoids the trivial solution p ( x | c ) = p ( x | c ′ ) for class pairs c, c ′ . The set of configurations for this experiment is reported in Table 4, and we use Adam [35] as optimizer. In the last qualitative experiment, we compute the true CCNS and its approximations for the specific example of Figure 1 (right), showing that the quantities obtained in Proposition 3.3 and Theorem 3.5 are good representatives of the CCNS for nearest neighbor graphs. Please refer to Appendix A.5 for an additional study of this kind. 2

5 Empirical Results

We now present our empirical results to support the theory. The interested reader can find statistics on the chosen hyper-parameters and additional ablation studies in Appendix A.6.

Quantitative Results Table 1 details the empirical comparison between the structure-agnostic MLP and the structure-aware baselines sDGN, GIN, and GCN. We observe that in terms of Micro F1 score, which is the metric all models are optimized against, there is no statistically significant improvement in performance (unpaired two samples t-test with p-value 0 . 05 ) for the different DGNs when using an artificial k -NN graph. When looking at accuracy one can observe a similar trend, but the results fluctuate more because accuracy is not the metric to be optimized. These results further confirm that using k -NN graphs does not bring a substantial contribution to the generalization performances. On the contrary, it would seem the k -NN graph can even be considered harmful for the performances of GIN and GCN (e.g., DryBean, Musk, eGrid, Waveform). We hypothesize that the information flow

0

2

4

6

x

0.0

0.2

0.4

0.6

0.8

1.0

pdf

p

(

x

|

C

=0)

p

(

x

|

C

=1)

0

2

4

6

8

10

0.35

0.40

0.45

0.50

0.55

0.60

0.65

pc

(

c

′

,

)

p

(

c

) = [0.5, 0.5]

p

0(0,

)

p

0(1,

)

p

1(0,

)

p

1(1,

)

0

2

4

6

8

10

0.2

0.3

0.4

0.5

0.6

0.7

0.8

pc

(

c

′

,

)

p

(

c

) = [0.3, 0.7]

p

0(0,

)

p

0(1,

)

p

1(0,

)

p

1(1,

)

of message passing does not act anymore as a regularizer (we refer the reader to the discussion in Section 1) as it might happen the scarce supervision scenario [23], rather it becomes harmful noise for the learning process. This is possibly the reason why sDGN, which is also the simplest of the DGNs considered, tends to overfit less such noise and always performs on par with the MLP.

Therefore, one of the main takeaways of the quantitative analysis is that the k -NN graph might generally not be a good artificial structure for addressing the classification of tabular data using graph representation learning methods. Different artificial structures, possibly inspired by our theoretical results, should be proposed to create graphs that help DGNs to better generalize in this context. We note that this research direction is broader than tabular classification, but tabular data offers a great starting point since samples are typically assumed i.i.d. in ML and are thus easier to manipulate. Also, in Appendix A.4, we discuss the impact of a different Ω (the cosine similarity) in the construction of the k -NN graphs.

These results also substantiate our conjecture that, under assumptions 1 , 3 , the k -NN graph never induces a better class separability in the embedding space: since p ( h | c ) is a mixture of distributions with the same mean but higher variance than p ( x | c ) (proof of Proposition 3.8) , in a low-dimensional space intuition suggests that the distributions of different classes will always overlap more in the embedding space than in the input space.

Qualitative Results We present the main qualitative results in Figures 2, 3 and 4. First, as we can see from Figure 2, when ε is very small the probability that a neighbor is of the same class is high for both classes (but not the same), since there is less overlap between samples of the green ( C = 0 ) and blue ( C = 1 ) distributions. As we increase the side length of the hypercube (or equivalently, we consider more and more nearest neighbors), the curves converge to the priors that we set for this example. This result is in agreement with the second limit of Proposition 3.7.

As regards the first limit of Proposition 3.7, in Figure 3 we observe that the behavior of lim ε → 0 p c ( c ′ , ε ) is non-trivial when the two curves still overlap. In particular, the greater the overlap the closer p c ( c, ε ) is to p ( c ) , which makes sense since it becomes harder to distinguish samples of different classes. On the other hand, the behavior becomes trivial when the curves are distant enough from each other: there, in the limit, every node will only have neighbors of the same class.

In the reverse-engineering experiment of Figure 4 (left), for each possible configuration of the models tried in Table 4, we have computed the L SED for varying values of k ∈ [1 , 500] . We have then normalized all values for readability, by dividing each value of SED ( p ( h | c ) , p ( h | c ′ )) by SED ( p ( x | c ) , p ( x | c ′ )) , as the latter is independent of k . This particular figure depicts the curves for the binary classification cases, but the conclusions do not change for multi-class classification, for instance with 5 classes. We find that the normalized value of SED ( p ( h | c ) , p ( h | c ′ )) on the y -axis is always upper-bounded by 1 , meaning that SED ( p ( h | c ) , p ( h | c ′ )) < SED ( p ( x | c ) , p ( x | c ′ )) for all the configurations tried, even in higher dimensions. This result further indicates that it could be unlikely that a k -NN graph induces better class separability in the embedding space of a DGN. We ask machine learning practitioners to take these insights into consideration when building artificial nearest neighbor graphs.

Lastly, we go back to the data distribution of Figure 1 (right) and show how to approximate the true CCNS under the k -NN graph with our theoretical framework. We first generate 10000 data points and set w.l.o.g. k = 5 to connect them. We compute the lower bound of the CCNS in the first

0

1

2

3

4

5

x

0.00

0.25

0.50

0.75

1.00

pdf

p

(

x

|

C

=0)

p

(

x

|

C

=1)

3

2

1

0

1

2

3

translation of

p

(

x

|0)

2

4

6

8

SED

(

p

(

x

|0),

p

(

x

|1))

3

2

1

0

1

2

3

translation of

p

(

x

|0)

0.00

0.25

0.50

0.75

1.00

lim

0

pc

(

c

′

,

)

p

(

c

) = [0.5, 0.5]

p

0(0,

)

p

0(1,

)

p

1(0,

)

p

1(1,

)

3

2

1

0

1

2

3

translation of

p

(

x

|0)

0.00

0.25

0.50

0.75

1.00

lim

0

pc

(

c

′

,

)

p

(

c

) = [0.3, 0.7]

p

0(0,

)

p

0(1,

)

p

1(0,

)

p

1(1,

)

0

100

200

300

400

500

0.5

1.0

SED

(

H

0,

H

1)

0

1

(a)

0

1

0

1

1

0

0

1

(b)

0.27

1

1.1

0.3

0

1

(c)

0.29

1.1

1.1

0.3

k

heatmap (a), a Monte Carlo (MC) approximation of Equation 1 ( 1000 samples per class pair) in the heatmap (b), and the true CCNS in the heatmap (c). In this case the lower bound has computed a good approximation of the inter-class CCNS, and the MC estimate is very close to the true CCNS. In Appendix A.6, we also show that the MC approximation is better for higher values of k . It is important to recall that both heatmaps (a) and (b) are computed using the theoretical results of Section 3.1 and an ε close to the empirical one, hence it seems that the theory is in accord with the evidence.

6 Conclusions

We introduced a new theoretical tool to understand how much nearest neighbor graphs can help in the classification of tabular data. Our theory and empirical evidence suggest that some attributebased graph construction mechanisms, namely the k -NN algorithm, are not promising strategies for obtaining better DGNs' generalization performances. This is a particularly troubling result since researchers often used nearest neighbor graphs when the graph structure was unavailable. We recommend using great care in future empirical evaluations of such techniques by always comparing against structure-agnostic baselines to ascertain real improvements from fictitious ones. Moreover, we provided a theoretically principled way to model the CCNS for nearest neighbor graphs, showing its approximation in a practical example. We hope that the results of this work will foster better strategies for building artificial graphs or as a guideline for new structure learning methods.

Acknowledgments and Disclosure of Funding

The author would like to acknowledge, in no particular order, Sascha Saralajew, Julia Gastinger, Ammar Shaker, Filippo Grazioli, Luca Oneto, Sandra Tartarelli, and Timo Sztyler for the stimulating discussions and extremely helpful feedback. A warm thank you also goes to the reviewers for striving to further improve this work and make it ready for publication.

References

Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. In 9th International Conference on Learning Representations (ICLR) , 2021.

Arthur Asuncion and David Newman. Uci machine learning repository, 2007. URL https: //archive.ics.uci.edu/ .

Davide Bacciu, Federico Errica, and Alessio Micheli. Probabilistic learning on graphs via contextual architectures. Journal of Machine Learning Research , 21(134):1-39, 2020.

Davide Bacciu, Federico Errica, Alessio Micheli, and Marco Podda. A gentle introduction to deep learning for graphs. Neural Networks , 129:203-221, 9 2020.

Aseem Baranwal, Kimon Fountoulakis, and Aukosh Jagannath. Graph convolution for semisupervised classification: Improved linear separability and out-of-distribution generalization. In Proceedings of the 38th International Conference on Machine Learning (ICML) , 2021.

Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of machine learning research , 7(11), 2006.

Adrien Benamira, Benjamin Devillers, Etienne Lesot, Ayush K Ray, Manal Saadi, and Fragkiskos D Malliaros. Semi-supervised learning and graph neural networks for fake news detection. In Proceedings of the IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) , 2019.

Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning. In Proceedings of the 2018 ACM Conference on Computer and Communications Security (SIGSAC) , 2018.

Christopher M Bishop. Pattern recognition and machine learning . Springer, 2006.

Cristian Bodnar, Fabrizio Frasca, Nina Otter, Yuguang Wang, Pietro Lio, Guido F Montufar, and Michael Bronstein. Weisfeiler and lehman go cellular: Cw networks. In Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS) , 2021.

Cristian Bodnar, Francesco Di Giovanni, Benjamin Paul Chamberlain, Pietro Lio, and Michael M. Bronstein. Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in GNNs. In Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS) , 2022.

Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond Euclidean data. IEEE Signal Processing Magazine , 34 (4):18-42, 2017.

Andrea Cavallo, Claas Grohnfeldt, Michele Russo, Giulio Lovisotto, and Luca Vassio. 2-hop neighbor class similarity (2ncs): A graph structural metric indicative of graph neural network performance. Workshop on Graphs and more Complex structures for Learning and Reasoning (AAAI) , 2023.

Ben Chamberlain, James Rowbottom, Maria I Gorinova, Michael Bronstein, Stefan Webb, and Emanuele Rossi. Grand: Graph neural diffusion. In Proceedings of the 38th International Conference on Machine Learning (ICML) . PMLR, 2021.

Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI) , pages 3438-3445, 2020.

Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In Proceedings of the 37th International Conference on Machine Learning (ICML) , 2020.

Yu Chen, Lingfei Wu, and Mohammed Zaki. Iterative deep graph learning for graph neural networks: Better and robust node embeddings. In Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS) , 2020.

Pedro Domingos. A few useful things to know about machine learning. Communications of the ACM , 55(10):78-87, 2012.

Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. Workshop on Deep Learning on Graphs: Methods and Applications (AAAI) , 2021.

Gamaleldin Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large margin deep networks for classification. In Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS) , 2018.

David Eppstein, Michael S Paterson, and F Frances Yao. On nearest-neighbor graphs. Discrete &Computational Geometry , 17:263-282, 1997.

Federico Errica, Marco Podda, Davide Bacciu, and Alessio Micheli. A fair comparison of graph neural networks for graph classification. In 8th International Conference on Learning Representations (ICLR) , 2020.

Bahare Fatemi, Layla El Asri, and Seyed Mehran Kazemi. Slaps: Self-supervision improves structure learning for graph neural networks. In Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS) , 2021.

Luca Franceschi, Mathias Niepert, Massimiliano Pontil, and Xiao He. Learning discrete structures for graph neural networks. In Proceedings of the 36th International Conference on Machine Learning (ICML) , 2019.

Paolo Frasconi, Marco Gori, and Alessandro Sperduti. A general framework for adaptive processing of data structures. IEEE Transactions on Neural Networks , 9(5):768-786, 1998.

Floris Geerts, Filip Mazowiecki, and Guillermo Perez. Let's agree to degree: Comparing graph convolutional networks in the message-passing framework. In Proceedings of the 38th International Conference on Machine Learning (ICML) , 2021.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning (ICML) , 2017.

Alessio Gravina, Davide Bacciu, and Claudio Gallicchio. Anti-symmetric dgn: a stable architecture for deep graph networks. In 11th International Conference on Learning Representations (ICLR) , 2023.

Xiawei Guo, Yuhan Quan, Huan Zhao, Quanming Yao, Yong Li, and Weiwei Tu. Tabgnn: Multiplex graph neural network for tabular data prediction. Workshop on Deep Learning Practice for High-Dimensional Sparse Data (KDD) , 2021.

Jacques A Hagenaars and Allan L McCutcheon. Applied latent class analysis . Cambridge University Press, 2002.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS) , 2017.

William L. Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. IEEE Data Engineering Bulletin , 40(3):52-74, 2017.

Marko Helén and Tuomas Virtanen. Query by example of audio signals using euclidean distance between gaussian mixture models. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2007.

Anees Kazi, Luca Cosmo, Seyed-Ahmad Ahmadi, Nassir Navab, and Michael M Bronstein. Differentiable graph module (dgm) for graph convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(2):1606-1617, 2022.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations (ICLR) , 2015.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations (ICLR) , 2017.

Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. In Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS) , 2021.

Nils M Kriege, Fredrik D Johansson, and Christopher Morris. A survey on graph kernels. Applied Network Science , 5(1):1-42, 2020.

Ao Li, Zhou Qin, Runshi Liu, Yiqun Yang, and Dong Li. Spam review detection with graph convolutional networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM) , 2019.

Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI) , 2018.

Yi-Lun Liao and Tess Smidt. Equiformer: Equivariant graph attention transformer for 3d atomistic graphs. In 11th International Conference on Learning Representations (ICLR) , 2023.

Siyuan Lu, Ziquan Zhu, Juan Manuel Gorriz, Shui-Hua Wang, and Yu-Dong Zhang. Nagnn: classification of covid-19 based on neighboring aware representation from deep graph neural network. International Journal of Intelligent Systems , 37(2):1572-1598, 2022.

Yao Ma, Xiaorui Liu, Neil Shah, and Jiliang Tang. Is homophily a necessity for graph neural networks? In 10th International Conference on Learning Representations (ICLR) , 2022.

Brandon Malone, Alberto Garcia-Duran, and Mathias Niepert. Learning representations of missing data for predicting patient outcomes. In Workshop on Deep Learning on Graphs: Method and Applications (AAAI) , 2021.

Miller McPherson, Lynn Smith-Lovin, and James M Cook. Birds of a feather: Homophily in social networks. Annual review of sociology , 27(1):415-444, 2001.

Alessio Micheli. Neural network for graphs: A contextual constructive approach. IEEE Transactions on Neural Networks , 20(3):498-511, 2009.

Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI) , volume 33, 2019.

Luis Müller, Mikhail Galkin, Christopher Morris, and Ladislav Rampášek. Attending to graph transformers. arXiv preprint arXiv:2302.04181 , 2023.

Galileo Mark Namata, Ben London, Lise Getoor, and Bert Huang. Query-driven active surveying for collective classification. In Proceedings of the Workshop on Mining and Learning with Graphs (MLG) , 2012.

Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural networks for graphs. In Proceedings of the 33rd International Conference on Machine Learning (ICML) , 2016.

Il-Seok Oh, Jin-Seon Lee, and Ching Y. Suen. Analysis of class separation and combination of class-dependent features for handwriting recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 21(10):1089-1094, 1999.

Luca Oneto, Sandro Ridella, and Davide Anguita. Do we really need a new theory to understand over-parameterization? Neurocomputing , 543, 2023.

KB Petersen and MS Pedersen. The matrix cookbook, version 20121115. Technical report, Technical University of Denmark, 2012.

Oleg Platonov, Denis Kuznedelev, Michael Diskin, Artem Babenko, and Liudmila Prokhorenkova. A critical look at the evaluation of GNNs under heterophily: Are we really making progress? In 11th International Conference on Learning Representations (ICLR) , 2023.

Lutz Prechelt. Early stopping-but when? Neural networks: tricks of the trade: second edition , pages 53-67, 2012.

Rahul Ragesh, Sundararajan Sellamanickam, Arun Iyer, Ramakrishna Bairi, and Vijay Lingam. Hetegcn: heterogeneous graph convolutional networks for text classification. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining (WSDM) , 2021.

Russell Reed and Robert J MarksII. Neural smithing: supervised learning in feedforward artificial neural networks . Mit Press, 1999.

Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review , 65(6):386, 1958.

Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks , 20(1):61-80, 2009.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina EliassiRad. Collective classification in network data. AI magazine , 29(3):93-93, 2008.

Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. Workshop on Relational Representation Learning, Neural Information Processing Systems (NeurIPS) , 2018.

Vikas Sindhwani, Partha Niyogi, and Mikhail Belkin. Beyond the point cloud: from transductive to semi-supervised learning. In Proceedings of the 22nd International Conference on Machine Learning (ICML) , 2005.

Alessandro Sperduti and Antonina Starita. Supervised neural networks for the classification of structures. IEEE Transactions on Neural Networks , 8(3):714-735, 1997.

Charles Sutton, Andrew McCallum, et al. An introduction to conditional random fields. Foundations and Trends® in Machine Learning , 4(4):267-373, 2012.

Catherine Tong, Emma Rocheteau, Petar Veliˇ ckovi´ c, Nicholas Lane, and Pietro Liò. Predicting patient outcomes with graph representation learning. In AI for Disease Surveillance and Pandemic Intelligence: Intelligent Disease Detection in Action , pages 281-293. Springer, 2022.

Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M Bronstein. Understanding over-squashing and bottlenecks on graphs via curvature. In 10th International Conference on Learning Representations (ICLR) , 2022.

Domenico Tortorella and Alessio Micheli. Leave graphs alone: Addressing over-squashing without rewiring. In Proceedings of the 1st Conference on Learning on Graphs (LOG) , 2022.

Vladimir Vapnik and Vlamimir Vapnik. Statistical learning theory wiley. New York , 1(624):2, 1998.

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In 6th International Conference on Learning Representations (ICLR) , 2018.

Jeroen K Vermunt and Jay Magidson. Latent class cluster analysis. Applied latent class analysis ,

11(89-106):60, 2002.

S Vichy N Vishwanathan, Nicol N Schraudolph, Risi Kondor, and Karsten M Borgwardt. Graph kernels. Journal of Machine Learning Research , 11(Apr):1201-1242, 2010.

Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog) , 38(5):1-12, 2019.

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems , 32(1):4-24, 2020.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In 7th International Conference on Learning Representations (ICLR) , 2019.

Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? In Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS) , 2021.

Donghan Yu, Ruohong Zhang, Zhengbao Jiang, Yuexin Wu, and Yiming Yang. Graph-revised convolutional network. In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD) , 2021.

Xiang Yu, Siyuan Lu, Lili Guo, Shui-Hua Wang, and Yu-Dong Zhang. Resgnet-c: A graph convolutional neural network for detection of covid-19. Neurocomputing , 452:592-605, 2021.

Ziwei Zhang, Peng Cui, and Wenwu Zhu. Deep learning on graphs: A survey. IEEE Transactions on Knowledge and Data Engineering , 34(1):249-270, 2020.

Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. In Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS) , 2020.

A Appendix

A.1 Dataset Statistics

This section shows the main statistics of the datasets used in this work.

A.2 Hyper-Parameters Tried

The table below specifies the configurations tried by the model selection procedure for all the baselines considered in this work. We apply a patience-based early stopping technique on the validation Micro F1-score as the only regularization technique [55]. The free parameters' budget of all three baselines is the same. We used Adam to train all models.

The following table, instead, reports the configurations tried during the qualitative experiments. We still apply a patience-based early stopping technique on L SED .

A.3 Proofs

Proposition 3.3. Under assumptions 1,2, M x ( c, ε ) has the following analytical form

where Z cmf is a r.v. with Gaussian distribution N ( · ; µ cmf , σ cmf ) .

Proof. Thanks to the conditional independence assumption, we can compute the integral of a product as a product of integrals over the independent dimensions. Defining a f = x f -ε 2 and b f = x f + ε 2 , we have

where Z cmf ∼ N ( · ; µ cmf , σ 2 cmf )

and the last equality follows from the known fact p ( a ≤ X ≤ b ) = F ( b ) -F ( a ) .

We introduce the following lemmas that will be useful in Theorem 3.5 and Theorem 3.8, which are about addition, linear transformation, and marginalization involving mixtures of distributions.

Lemma A.1. Let X,Y be two independent random variables with corresponding mixture distributions ϕ X ( w ) = ∑ I i =1 α i f i ( w ) and ϕ Y ( w ) = ∑ J j =1 β j d j ( w ) . Then Z = X + Y still follows a mixture distribution.

Proof. By linearity of expectation the moment generating function of X (and analogously Y ) has the following form

where X i is the r.v. corresponding to a component of the distribution. Using the fact that the moment generating function of Z = X + Y is given by M Z ( t ) = M X ( t ) M Y ( t ) , we have that

Therefore, Z follows a mixture model with IJ components where each component follows the distribution associated with the random variable Z ij = X i + Y j .

Lemma A.2. Let X be a r.v. with multivariate Gaussian mixture distribution ϕ X ( w ) = ∑ I i =1 α i N ( w ; µ i , Σ i ) , w ∈ R D . Then Z = Λ X, Λ ∈ R D × D still follows a mixture distribution.

Proof. Using the change of variable z = Λ w we have

By expanding the terms, we see that the distribution of Z is still a mixture of distributions of the following form

Lemma A.3. Let X,Y be two independent random variables with corresponding Gaussian mixtures of distributions ϕ X ( w ) = ∑ I i =1 α i N ( w ; µ X i , σ 2 X i ) and ϕ Y ( w ) = ∑ J j =1 β j N ( w ; µ Y j , σ 2 Y j ) . Then

Proof. It is useful to look at this integral from a probabilistic point of view. In particular, we know that

and that, by marginalizing over all possible values of X,

Therefore, finding the solution corresponds to computing p ( Y -X ≤ 0) . Because X,Y are independent, the resulting variable Z = Y -X is distributed as (using Lemma A.1 and Lemma A.2)

and hence, using the fact that the c.d.f. of a mixture of Gaussians is the weighted sum of the individual components' c.d.f.s :

Theorem 3.5. Under assumptions 1,2, M c ′ ( c, ε ) has the following analytical form

Proof. The proof relies on the previous Lemma A.3. We begin by expanding the formula and using the result of Proposition 3.3. We define the r.v. Z imf ∼ N ( w ; µ imf , σ 2 imf ) to write

We first note that F Z cmf ( x f + ε 2 ) = F Y ( x f ) where Y follows distribution N ( w ; µ cmf -ε 2 , σ 2 cmf ) (and symmetrically for F Z cmf ( x f -ε 2 ) ), so we can apply Lemma A.3 and obtain

Proposition 3.7. Let the first D derivatives of ∑ |C| i =1 M c ′ ( i ) be different from 0 in an open interval I around ε = 0 and not tending to infinity for ε → 0 . Then, under assumptions 1,2, Equation 7 has the following limits (the first of which requires the assumptions)

where Z imm ′ f has distribution N ( · ; -a imm ′ f , b 2 imm ′ f ) , a imm ′ f = 2( µ c ′ m ′ f -µ imf ) and b imm ′ f = 2 √ σ 2 imf + σ 2 c ′ m ′ f .

Proof. The second limit follows from lim x → + ∞ Φ ( x ) = 1 and lim x →-∞ Φ ( x ) = 0 . As regards the first limit, we first expand the terms

where a imm ′ f = 2( µ c ′ m ′ f -µ imf ) and b imm ′ f = 2 √ σ 2 imf + σ 2 c ′ m ′ f ∀ i ∈ C to simplify the notation. By defining Z imm ′ f ∼ N ( · ; -a imm ′ f , b 2 imm ′ f ) , we can rewrite the limit as

Both the numerator and denominator tend to zero in the limit, so we try to apply L'H ˆ opital's rule, potentially multiple times.

For reasons that will become clear soon, we show that the limit of the first derivative of each term in the product is not zero nor infinity:

In addition, let us consider the n -th derivative of the product of functions (by applying the generalized product rule):

and we note that, as long as n < D there will exist an assignment to j f = 0 ∀ f in the summation, and thus the limit of each product will always tend to 0 when ε goes to 0 since the individual terms cannot tend to infinity for our assumptions. However, when we take the D -th derivative, there exists one term in the summation that does not go to 0 in the limit, which is

Therefore, by applying the L'H ˆ opital's rule D times, we obtain

However, to be valid, L'H ˆ opital's rule requires that the derivative of the denominator never goes to 0 for points different from ε = 0 . This is part of our assumptions, hence we conclude the proof.

For n = 1 we can show that this last requirement holds as d (1) imm ′ f ( ε ) > 0 ∀ ε , d imm ′ f ( ε ) ≥ 0 , and d imm ′ f ( ε ) = 0 if and only if ε = 0 . In fact,

̸

is a sum over terms that are all greater than 0 for ε = 0 .

Analytical form of the n -th derivative of ∑ |C| i =1 M c ′ ( i ) To verify if the hypotheses of Proposition 3.7 is true given the parameters of all features' distributions, one could compute the n -th derivative of the denominator w.r.t. ε and check that it is not zero around 0. We start again from

and we proceed using the fact that the n -th derivative of the standard normal distribution S ∼ N (0 , 1) has a well-known form in terms of the n -th (probabilist) Hermite polynomial He n ( x ) . We note the following equivalence:

hence we can write

We use this result in the expansion of Equation 50

However, we can readily see that computing the derivative for a single ε has combinatorial complexity, which makes the application of the above formulas practical only for small values of D .

Class Separability Analysis of Section 3.2 We now present a result that will help us compute the SED divergence between two Gaussian mixtures of distributions.

Lemma A.4. Let X,Y be two independent random variables with corresponding Gaussian mixture distributions ϕ X ( w ) = ∑ I i =1 α i f X i ( w ) = ∑ I i =1 α i N ( w ; µ X i , Σ X i ) and ϕ Y ( w ) = ∑ J j =1 β j f Y j ( w ) = ∑ J j =1 β j N ( w ; µ Y j , Σ Y j ) , w ∈ R D . Then the SED divergence between ϕ X ( w ) and ϕ Y ( w ) can be computed as

Finally, we can compute the integral of the product of two Gaussians as follows (Section 8.1.8 of [53])

to obtain the desired value.

Proposition 3.8. Under assumptions 1,3, let us assume that the entity embeddings of a 1-layer DGN applied on an artificially built k -NN graph follow the distribution of Equation 10. Then the Squared Error Distances SED ( p ( h | c ) , p ( h | c ′ )) and SED ( p ( x | c ) , p ( x | c ′ )) have analytical forms.

Proof. First, we have to show how we can compute the conditional probabilities p ( h | c ) and p ( h | c ′ ) , and then we derive the analytical computation of the SED divergence to verify the inequality in a similar manner of Helén and Virtanen [33].

Proof.

We first work out the explicit form of p ( h | c ) , by marginalizing out x :

Therefore, the distribution resulting from the k -NN neighborhood aggregation will change the input's variance in a way that is inversely proportional to the number of neighbors. In the limit of k →∞ and finite σ ε , it follows that p ( h | c ) = p ( x | c ) . Since p ( h | c ) still follows a mixture of distributions of known form, we note that a repeated aggregation of the neighborhood aggregation mechanism (without any learned transformation) would only increase the values of the covariance matrix. In turn, this would make the distribution spread more and more, causing what is known in the literature as the oversmoothing effect (see Section 2).

Since both X c and H c follow a Gaussian mixture distribution, we apply Lemma A.4 to obtain a closed-form solution and be able to evaluate the inequality. In particular

and

As an additional remark, if SED ( ϕ H c , ϕ H c ′ ) > SED ( ϕ X c , ϕ X c ′ ) , points in the DGN's latent space will be easier to classify than those in the input space because the former pair of distributions are more separable than the latter.

We also observe that p ( h | c ) is a mixture of Gaussians that differs from p ( x | c ) only in the higher variance of the individual Gaussian components. In other words, the distribution of p ( h | c ) looks 'flatter' than p ( x | c ) . At least in a one or two-dimensional space, flatter distributions p ( h | c ) and p ( h | c ′ ) should therefore increase the overlap (and therefore decrease the SED) compared to the SED in the original space. This intuition, despite its simplicity, is unfortunately hard to prove for technical reasons, for instance the dependence of the distance value on the specific parameters µ and Σ .

A.4 Impact of the Metric in the Construction of k -NN Graphs

To judge how much the results might change if we used a different metric, we test the cosine similarity as done in [56]. We note that the cosine distance is less expressive than the Euclidean distance as it does not consider the magnitude of values, and that results in overall worse performances. DGNs typically work in the Euclidean space when aggregating neighbors, hence the homophily assumption can be imposed - to a certain extent - only by computing nearest neighbors in terms of Euclidean distance. In addition, building k -NN graphs using the Euclidean distance criterion has an easy interpretation in terms of 'how similar the attributes of adjacent nodes are'. We observe that the results below still agree with our theoretical intuitions and empirical results.

A.5 Analysis of CCNS approximation's quality

In the figure below, we visualize how the true CCNS gets closer to the MC approximation when the number of neighbors k increases for a random data distribution with 5 classes. The choice of the ε parameter for the MC estimation of the CCNS has a marginal impact here. This behavior is expected, because as k increases we get a better approximation of the posterior class distribution in the hypercube around each point, which is the one we computed in Section 3.1. When k is too small, the empirical (true) CCNS might be more unstable and dataset-dependent.

0

1

2

3

4

True CCNS for

k

=5

0

1

2

3

4

0.59

0.6

0.59

0.59

0.59

0.6

0.61

0.6

0.61

0.61

0.59

0.6

0.59

0.61

0.6

0.59

0.61

0.61

0.56

0.6

0.59

0.61

0.6

0.6

0.58

0

1

2

3

4

True CCNS for

k

=25

0

1

2

3

4

0.38

0.39

0.39

0.39

0.37

0.39

0.39

0.39

0.4

0.39

0.39

0.39

0.39

0.42

0.39

0.39

0.4

0.42

0.35

0.39

0.37

0.39

0.39

0.39

0.34

0

1

2

3

4

True CCNS for

k

=50

0

1

2

3

4

0.34

0.35

0.35

0.35

0.34

0.35

0.35

0.35

0.36

0.36

0.35

0.35

0.35

0.38

0.35

0.35

0.36

0.38

0.3

0.36

0.34

0.36

0.35

0.36

0.3

0

1

2

3

4

True CCNS for

k

=100

0

1

2

3

4

0.31

0.32

0.32

0.33

0.31

0.32

0.33

0.33

0.34

0.33

0.32

0.33

0.32

0.36

0.32

0.33

0.34

0.36

0.28

0.34

0.31

0.33

0.32

0.34

0.28

0

1

2

3

4

0

1

2

3

4

0.28

0.3

0.29

0.3

0.28

0.29

0.31

0.3

0.32

0.31

0.29

0.3

0.28

0.32

0.29

0.3

0.32

0.32

0.25

0.31

0.29

0.31

0.29

0.31

0.25

MC estimate of CCNS

A.6 Best Configurations and Ablation Studies

In the figure below, we show which are the best k and ε values picked by the models trained for the qualitative experiments.

0

1

2

3

4

5

k

0

10

20

30

Count

2

4

6

8

10

12

14

16

ε

0

10

20

30

Count

Analysis of Chosen Hyper-parameters For completeness, in the following we display the best k , number of layers, and number of hidden units chosen by the different families of models on each dataset of Table 1. We recall that for each dataset and model there are 10 best configurations due to the use of a 10-fold cross validation for risk assessment [22]. We did not find any significant pattern except for SDGN preferring lower values of k and the DGNs often choosing at least 2 convolutional layers.

1

3

5

10

k

0

1

2

3

4

5

6

Count

abalone

model

MLP

sDGN

GIN

GCN

1

3

5

10

k

0

2

4

6

8

10

Count

adult

model

MLP

sDGN

GIN

GCN

1

3

5

10

k

0

2

4

6

8

10

Count

drybean

model

MLP

sDGN

GIN

GCN

1

3

5

10

k

0

2

4

6

8

10

Count

electricalgrid

model

MLP

sDGN

GIN

GCN

1

3

5

10

k

0

2

4

6

8

10

Count

isolet

model

MLP

sDGN

GIN

GCN

1

3

5

10

k

0

2

4

6

8

10

Count

musk

model

MLP

sDGN

GIN

GCN

1

3

5

10

k

0

1

2

3

4

5

Count

occupancy

model

MLP

sDGN

GIN

GCN

1

3

5

10

k

0

2

4

6

8

Count

waveform

model

MLP

sDGN

GIN

GCN

1

3

5

10

k

0

1

2

3

4

5

6

Count

cora

model

MLP

sDGN

GIN

GCN

1

3

5

10

k

0

1

2

3

4

5

Count

citeseer

model

MLP

sDGN

GIN

GCN

1

3

5

10

k

0

2

4

6

8

10

Count

pubmed

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

0

2

4

6

8

Count

abalone

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

0

2

4

6

8

10

Count

adult

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

0

2

4

6

8

10

Count

drybean

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

0

2

4

6

8

10

Count

electricalgrid

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

0

2

4

6

8

10

Count

isolet

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

0

2

4

6

Count

musk

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

0

2

4

6

Count

occupancy

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

0

1

2

3

4

5

6

Count

waveform

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

0

2

4

6

8

10

Count

cora

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

0

2

4

6

8

Count

citeseer

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

0

2

4

6

8

10

Count

pubmed

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

0

2

4

6

8

10

Count

abalone

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

0

2

4

6

Count

adult

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

0

2

4

6

Count

drybean

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

0

1

2

3

4

5

6

Count

electricalgrid

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

0

2

4

6

8

Count

isolet

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

0

1

2

3

4

5

6

Count

musk

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

0

2

4

6

Count

occupancy

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

0

1

2

3

4

5

6

Count

waveform

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

0

2

4

6

8

Count

cora

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

0

1

2

3

4

5

6

Count

citeseer

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

0

1

2

3

4

5

6

Count

pubmed

model

MLP

sDGN

GIN

GCN

Ablation Study We now provide an ablation study of the validation performances of the models during the model selection phase (i.e., across all possible configurations). We show the ablations for the most important architectural choices of the hyper-parameters. Also in this case, a lower k is associated with better performances, but there does not seem to be marked improvements when using more than 1 graph convolutional layer.

0

1

3

5

10

k

20

30

40

50

60

70

vl_acc

abalone

model

MLP

sDGN

GIN

GCN

0

1

3

5

10

k

65

70

75

80

85

vl_acc

adult

model

MLP

sDGN

GIN

GCN

0

1

3

5

10

k

0

20

40

60

80

100

vl_acc

drybean

model

MLP

sDGN

GIN

GCN

0

1

3

5

10

k

40

60

80

100

vl_acc

electricalgrid

model

MLP

sDGN

GIN

GCN

0

1

3

5

10

k

40

60

80

100

vl_acc

isolet

model

MLP

sDGN

GIN

GCN

0

1

3

5

10

k

60

70

80

90

100

vl_acc

musk

model

MLP

sDGN

GIN

GCN

0

1

3

5

10

k

70

80

90

100

vl_acc

occupancy

model

MLP

sDGN

GIN

GCN

0

1

3

5

10

k

40

60

80

vl_acc

waveform

model

MLP

sDGN

GIN

GCN

0

1

3

5

10

k

55

60

65

70

75

80

85

vl_acc

cora

model

MLP

sDGN

GIN

GCN

0

1

3

5

10

k

55

60

65

70

75

80

vl_acc

citeseer

model

MLP

sDGN

GIN

GCN

0

1

3

5

10

k

77.5

80.0

82.5

85.0

87.5

90.0

vl_acc

pubmed

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

10

20

30

40

50

60

vl_acc

abalone

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

65

70

75

80

85

vl_acc

adult

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

0

20

40

60

80

100

vl_acc

drybean

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

40

60

80

100

vl_acc

electricalgrid

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

40

60

80

100

vl_acc

isolet

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

60

70

80

90

100

vl_acc

musk

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

70

80

90

100

vl_acc

occupancy

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

20

40

60

80

100

vl_acc

waveform

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

60

65

70

75

80

85

vl_acc

cora

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

55

60

65

70

75

80

vl_acc

citeseer

model

MLP

sDGN

GIN

GCN

1

2

3

# layers

77.5

80.0

82.5

85.0

87.5

90.0

vl_acc

pubmed

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

20

30

40

50

60

70

vl_acc

abalone

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

65

70

75

80

85

vl_acc

adult

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

0

20

40

60

80

100

vl_acc

drybean

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

40

60

80

100

vl_acc

electricalgrid

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

40

60

80

100

vl_acc

isolet

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

60

70

80

90

100

vl_acc

musk

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

70

80

90

100

vl_acc

occupancy

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

20

40

60

80

vl_acc

waveform

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

55

60

65

70

75

80

85

vl_acc

cora

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

55

60

65

70

75

80

vl_acc

citeseer

model

MLP

sDGN

GIN

GCN

32

64

128

# hidden units

77.5

80.0

82.5

85.0

87.5

90.0

vl_acc

pubmed

model

MLP

sDGN

GIN

GCN

Peer reviews:
comments: SUMMARY:
This paper studies the machine learning problems on tabular data and convert the problem into node classification. Then, they formally analyze the Cross-Class Neighborhood Similarity (CCNS) and validate the performance on benchmark datasets.

STRENGTHS:
S1. The studied problem is very important. 

S2. The paper offers theoretical analysis. 

S3. The paper is well written and the idea is easy to follow.

WEAKNESSES:
W1. More baselines are needed, such as other SOTA solutions in long-tail node classification. 

W2. Neighbors’ sampling for balanced classes seems to be less novel [1]. The technical contribution is weak.

W3. Introducing graph data for tabular data has been studied a lot [2]. 

[1] Implicit Class-Conditioned Domain Alignment for Unsupervised Domain Adaptation, ICML 20

[2] TabGNN: Multiplex graph neural network for tabular data prediction.

QUESTIONS:
See weakness.

LIMITATIONS:
See weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-16

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies quantitative measures to evaluate the usefulness of k-NN graphs when performing classification in datasets that do not originally come with a known graph topology. The authors argue that the use of k-NN graphs is not useful.\n\nSTRENGTHS:\nA study of applying k-NN to construct a graph and then using standard GNN models for node classification. This is compared to using the baseline MLP that does not make use of any neighboring node information.\n\nWEAKNESSES:\n1. The paper is very heavy on assumptions, especially using the use of Gaussian models to derive certain results. There is no empirical evidence that such models hold in the datasets used.\n\n2. The numerical experiments are not convincing enough. Maybe I misunderstood how the experiments are conducted. A major concern is how the k-NN graphs are constructed and how the GNN models are applied.\n\nQUESTIONS:\n1. In the definition (1), q_c does not seem to depend on the graph structure, unlike the original definition in [39]. How do you incorporate the notion of \"neighborhood\" here?\nIn fact, the whole development in Section 3.1 does not need to assume that we are performing node classification in a graph. Only in (9) do we use the notion of a node's neighborhood. \n\n1. What is the purpose of the derivations in Prop 3.3 - 3.7? These are never used anywhere else in the paper or experiments (except for a very simple synthetic example). While I understand that the authors \"leave these ideas to future investigations\", there should be empirical studies to demonstrate that the results are of interest or useful.\n\n1. The distribution p(\\omega | c) in (3) has not been stated. Looking through the proofs and Prop 3.3, it seems to be assumed to be a Gaussian. Is this reasonable and is there empirical evidence for it?\n\n1. The definition of SED in Prop 3.8 was never properly given. One has to look at the proof to realize the measure used is Lebesgue measure. Why is that when p(\\omega | c) above is assumed to be Gaussian?\n\n1. In the experiments, what distance metric is assumed to construct the k-NN graphs? The performance of different GNN models depends very much on the graph topology. Have different metrics been considered and do the authors' conclusions hold regardless of the metric?\n\nLIMITATIONS:\nTheoretical assumptions have not been comprehensively tested.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shows that using a k-NN graph in tabular data classification is not beneficial compared to MLP. Specifically, based on Cross-Class Neighborhood Similarity, the authors theoretically induce the analytical forms of the squared error distance (SED) of two approaches. In the experiment, they demonstrate that the SED of the method using a k-NN graph is smaller than the SED of the method using attributes directly. Furthermore, in fully-supervised setting, they show that MLP exhibits comparable performance with methods using k-NN graphs.\n\nSTRENGTHS:\n- They propose a theoretical framework to analyze the method using a k-NN graph.\n- They show that the SED for using a k-NN graph is smaller than the SED for using attributes directly in synthetic data.\n- They demonstrate that MLP exhibits comparable performance with the approaches using a k-NN graph in fully-supervised setting.\n\nWEAKNESSES:\n- [W1] As the authors mentioned, the assumptions seem strong. Especially, for assumption 1, it would not hold that attributes are independent in the real-world data. Also, they fix GNN architecture as GCN in Equation 9, but there exists many effective GNNs.\n- [W2] I understand that SED can be related with performance. However, it seems not enough to exploit the statement that the methods with large SED would show better performance than the methods with small SED.\n- [W3] As far as I understand, the proposed framework gives only the analytic forms of SED and the empirical evidence for SED difference is also suggested in the synthetic setting. It seems not enough to assert that using a k-NN graph offers no benefits compared to a structure-agnostic baseline.\n- [W4] The analysis about the analytic forms does not provided in the main paper.\n\nQUESTIONS:\n- Following [W1], could you provide the correlation between attributes in the real-world datasets?\n- Following [W2], would you provide eivdences about the relation between SED and classification performance?\n- Following [W3], could you provide more evidences that the SED for the methods using a k-NN graph is smaller than the SED of the methods using attributes directly?\n- Following [W4], would you provide the detailed description about analytic forms? Also, I'm curious about the conditions when SED of methods using a k-NN graph gets large based on the analytic forms?\n\nLIMITATIONS:\nAs the authors mentioned, the assumptions seem strong [W1].", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This submission provides a theoretical framework to explore the benefits of Nearest Neighbor (NN) graphs in the absence of a predefined graph structure. The Cross-Class Neighborhood Similarity (CCNS) is formally analyzed to evaluate the usefulness of structures within nearest neighbor graphs. The study also investigates the class separability induced by deep graph networks on a k-NN graph. Quantitative experiments conducted under full supervision demonstrate that employing a k-NN graph provides no advantages compared to a structure-agnostic baseline. Qualitative analyses indicate that the framework effectively estimates CCNS and suggests that k-NN graphs are not useful for such classification tasks. This highlights the need to explore alternative graph construction techniques.

STRENGTHS:
1. The theoretical analysis is interesting.

2. The motivation is clear.

WEAKNESSES:
To be honest, I am not familiar with this topic. I can only provide high-level comments as follows,

1. Equation (2) is not clear. For example, how to derive the equivalent between two absolute values?

2. Figure 2 left is not very easy to be understood. For example, what are the colors mean, and what is the meaning of each line?

QUESTIONS:
N/A

LIMITATIONS:
The authors have discussed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main contributions of this paper are as follows:\n\n1. The authors of the paper study Cross-Class Neighborhood Similarity (CCNS) for nearest neighbor graphs and provide its first lower bound. \n2. They apply a Deep Graph Network (DGN) to an artificial k-NN graph and study how it affect the class separability of the input data. \n3. They carry out a robust comparison between structure-agnostic and structure-aware baselines on a set of 11 datasets, and their results are in agreement with their theoretical results. \n4. Qualitative analyses based on the theory further suggest that using the k-NN graph might not be advisable.\n\nOverall, the authors introduce a theoretical framework to understand the benefits of Nearest Neighbor (NN) graphs when a graph structure is missing and provide both theoretical and empirical evidence to support their conclusions. They try to raise awareness about the potential limitations of k-NN graphs for node classification tasks and advocates for the study of alternative graph construction techniques.\n\nSTRENGTHS:\n1. The authors introduce a theoretical framework to understand the benefits of Nearest Neighbor (NN) graphs when a graph structure is missing.\n2. They provide a formal analysis of the Cross-Class Neighborhood Similarity (CCNS) in the context of nearest neighbor graphs and provide an analytical computation of the CCNS for Nearest Neighbor Graphs.\n3. They carry out a robust comparison between structure-agnostic and structure-aware baselines on a set of 11 datasets that is in agreement with their theoretical results.\n\nOverall, this paper provides both theoretical and empirical evidence to support its conclusions and makes a contribution to the understanding of the benefits and limitations of using k-NN graphs for node classification tasks.\n\nWEAKNESSES:\nAs the authors mentioned in the paper, they assume that the attributes are conditionally independent when conditioned on a specific mixture and class. These assumptions may not hold for all datasets and could limit the generalizability of their results.\n\nAdditionally, the authors’ analysis is focused on the fully-supervised setting, where all training labels are available. It is not clear how their conclusions would apply to other settings, such as the semi-supervised setting where only a subset of training labels is available.\n\nOverall, while this paper provides valuable insights into the benefits and limitations of using k-NN graphs for node classification tasks, its conclusions are based on specific assumptions and conditions and may not apply to all situations.\n\nQUESTIONS:\n1. How does your work relate to other dataset formats more than tabular data?\n\n2. How would your conclusions apply to other settings, such as the semi-supervised setting where only a subset of training labels is available?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to investigate the practice of using nearest neighbour graphs build upon tabular data in order to classify samples (then relying on a node classification model). The authors note that this practice has been shown to be relevant in the semi-supervised case, but not with full supervision - and that it often exploits the common idea that homophily is a desired graph property for node classification models. However, recent works shows that the difference between the distributions of labels of neighbours for nodes of separate class (measured by the Cross-Class Neighborhood Similarity, CCNS) is what matters for their efficiency. This brings the authors to propose (1) a formal model for computing an approximation of these distributions under several assumptions (2) under the same assumptions, study the separability of classes induced by a simple node representation model relying on message passing on the NN-graph. \nThe authors then propose two sets of experiments to draw conclusions: (1) an study of the results of learning classification models directly on tabular data or on NN-graphs for 11 tasks (2) qualitative experiments relying on their theoretical work, verifying if, for a toy task of binary classification based on simple distributions built following their assumptions: (a) there exists of configuration where classes are more separable using a NN-graph structure (b) the quantities obtained via their approximation of CCNS concurr to the conclusion of (a).\n\nSTRENGTHS:\n- This paper is very well written, and easy to follow, despite the difficulty of the subject.\n- This paper demonstrates experimentally that using nearest neighbour graphs build upon tabular data in order to classify samples does not help performance. \n- This paper provides well-motivated theoretical arguments showing that a model built on message-passing for a k-NN graph will not improve class separability.\n- Finally, this paper provides a theoretical framework (under assumptions) to build a tool, allowing to approximate the CCNS (an indicator of potential success of node classification models).\n\nWEAKNESSES:\n- While the paper clearly discusses the limitations brought by the assumptions it makes, it would be interesting and useful to explore how significant they are.\n- The utility of the theoretical developments for the experimental demonstration is not made completely clear.\n\nQUESTIONS:\n- The purpose of the CCNS approximation proposed is a little unclear. Is it supposed to be a practical tool for testing the relevance of the k-NN graph for a classification task ?\n\nLIMITATIONS:\n- Limitations have been adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-26"}