Graph Neural Networks Formed via Layer-wise Ensembles of Heterogeneous Base Models
Authors: Jiuhai Chen
Jonas Mueller
Vassilis N. Ioannidis
Tom Goldstein
David Wipf
Venue: Accepted by TMLR ()

Abstract: Graph Neural Networks (GNNs) with numerical node features and graph structure as inputs have demonstrated superior performance on various semi-supervised learning tasks with graph data. However, the numerical node features utilized by GNNs are commonly extracted from raw data which is of text or tabular (numeric/categorical) type in most real-world applications. 
The best models for such data types in most standard supervised learning settings with IID (non-graph) data are not simple neural network layers and thus are not easily incorporated into a GNN. Here we propose a robust stacking framework that fuses graph-aware propagation with arbitrary models intended for IID data, which are ensembled and stacked in multiple layers. Our layer-wise framework leverages bagging and stacking strategies to enjoy strong generalization, in a manner which effectively mitigates label leakage and overfitting. Across a variety of graph datasets with tabular/text node features, our method achieves comparable or superior performance relative to both tabular/text and graph neural network models, as well as existing state-of-the-art hybrid strategies that combine the two.

Published in Transactions on Machine Learning Research (2/2024)
Graph Neural Networks Formed via Layer-wise Ensembles of Heterogeneous Base Models
Jiuhai Chen University of Moryland
jehen weteen66@@mmdccomm
Jonas Mueller
jonasmacilerrecsati mit. edu
Cleanlab
Vassilis N. Ioannidis Amazon
bussilsnicammidiie@gaaa.cc
Tom Goldstein
tomg@und edu
University of Morylond
David Wipf
dovidwipff@gmaai edu
Amazon
Reviewed OpenReview:
https: https://openrevveeu openrevieww net/ forum? id= L002YHorad
Abstract
Graph Neural Networks (GNNs) with numerical node features and graph structure as inputs have demonstrated superior performance on various semi-supervised learning tasks with graph data. However, the numerical node features utilized by GNNs are commonly extracted from raw data which is of text or tabular applications. The best models for such data types in most standard supervised learning setlings with IID (non-graph) data are not simple neural network layers and thus not easily incorporated into a GNN. Here we propose robust stacking framework that fuses graph-aware propagation with arbitrary models intended for IID data, which are ensembbeed and stacked in multiple layers. Our layer- wise framework leverages bagging and stacking strategies enjoy strong generalization, in a manner which effectively mitigates label leakage and overfitting. Across a variety of graph datasets with tabular/texs node features, our method achieves comparable or superior performance relative to both tabular/text and graph neural network models, as well as existing State-of-the-ate hybrid strategies that combine the two.
type in most real-world
1 Introduction
Graph datasets comprise nodes of various data types and modalities linked by edges that encapsulate non-lID conditional dependencies between them. While it is often assumed that graph neural networks (GNN) KKipf & Welling 2016 Veličković et al. 2017] are preferable for handling such data relative to models originally designed for IID instances, GNNs are nonetheless subject to various limitations. In particular, the best architecture may be data-set specific and require appropriately setting many attendant structural
Work done during internship Amazon Web Services Shanghai Tab

---

Published in Transactions on Machine Learning Research (2/2024)
hyperparameters, e.g.; note the complex assortment of GNN architectures that populate the top of the Open Graph Benchmark (OGB) leaderboard (Hu et al. 2020) Moreover, most GNNs implicitly assume that node features are numerical, and may struggle to remain competitive with more complex text, tabular, or composite alternatives.
In fact, with richer node feature sets it has even been observed that models tailored to ITD data (which our setting simply operate on individual node features as though they were independent of the others) can at times outperform GNNs if they are combined with simple graph propagation operations to account for the graph structure (Huang et al. 2020 Chen et al. 2021 Moreover, for graph data with text features, Chien et al. (2021) has demonstrated that leveraging a BERT Transformer in addition to GNN can greatly improve performance. And beyond these considerations, real-workd applications of ML typically involve more than just a single model, GNN or otherwise. Instead they usually require an ML pipeline composed of data preprocessing and braniinng/tuinn/gggen
of many models achieve the best results.
In this paper, we investigale how to adapt ML pipelines designed for supervised learning with IID data (e.g., Transformers for text, gradient boosted decision trees related for tabular data) to node classifica- tion/regression tasks with graph-structured statistical dependencies between node features. We focus on using K-fold bagging (Breiman 1996), i.c. pross-validation. to avoid label leakage issues, with stack ensembling methods for maximal flexibility (Wolpert 1992 Van der Laan et. al.) 2007). These techniques are particularly effective for achieving high accuracy across diverse IID datasets, and are utilized in many popular AutoML
frameworks (Erickson et al. 2020 LeDell & Poirier 2020: Feurer et al. 2015), but have largely been ignored within the context of graph data.
Within this context, our goal is to design single architecture that integrates graph propagation mcssage passing steps and stacked ensembles of arbitrary base models to flexibly accommodate diverse types within a unified framework. In doing so, our contributions are as follows:
We propose a framework of stack ensembling with graph propagation called BestowGNN for Bagged, Ensembled, Stacked Training Of Well-balanced GNNs (see Figure that can bestow arbitrary (uon-graph) base models intended for IID data with the capability of producing highly accurate node predictions in the graph (i.e., non-IID) setting.
Using only a single, unified architecture, our proposed methodology can match or outperform bespoke datasct-specific models that top competitive leaderboards for popular node coassiiication/rerressttt tasks (e.g., on OGB and elsewhere completely different network architectures typically dominate the top positions for different datasets and data types).
Label leakage is an unavoidable issue for many layer-wise training strategies (SAGN Sun & Wu 2021 and GAMLP (Zhang et al. 2. To address this potential shortcoming, we formalize how our bagging and stacking framework can effectively mitigate the label Icakage issuc within the graph sctting using analytical tools from differential privacy. This is the first work establishing that bagging with graph-based predictors can be useful for ameliorating label leakage.
2 Related Work
2.1 From Scalability to Layer-wise Training
Currently, GNN training suffers from high computational cost as the number of layers grows. To improve the scalability, the graph sampling scheme GraphSAGE (Hamilton et al. 2017]] proposes to uniformly sampling a fixed number of neighbours for batch of nodes. Meanwhile, Cluster- clustering algorithms to sample block of nodes that form a dense subgraph and then runs SGD-based algorithms on these subgraphs. Quite differently, L2-GON (You et al. 2020 proposes a layer-wise training framework by disentangling feature aggregation and feature transformations to reduce time and memory complexity.
(Chiang et al. 2019) uses graph
Other approaches include SAGN (unn & Wu 2021 , which iteratively trains models in several stages by applying a graph structure-aware attention mechanism on node features and also combines a self-training
2

---

Published in Transactions on Machine Learning Research (2/2024)
approach with label propagation to further improve performance. GAMLP Zhang al 2021 instead proposes two attention mechanisms to explore the relation between features with different propagation steps. Both SAGN and GAMLP achieve strong performance on two large open graph benchmarks (ogbn-produce and ogbn-papers 100M), demonstrating the high scalability and efficiency of layer-wise training strategies. However, SAGN and GAMLP suffer from the risk of label leakage: label information is included in the enhanced training set, and can cause performance degradation if the model extracts and relies on these labels. SAGN empirically shows that sufficient propagation depth can effectively alleviate label leakage, thus they only use label information at one fixed propagation step. Meanwhile, GAMLP passes label information between propagation steps using residual connections. Wang et al. (202T) further randomly masks nodes during every training epoch to mitigate the label leakage issue. Even so, the efficacy of these methods in mitigating label leakage issue remains somewhat speculative. In contrast, by using utilizing differential privacy tools, we rigorously evaluate how our proposed bagging and stacking framework can explicitly reduce label leakage concerns in node classification scttings.
2.2 Graph models with Multifaceted Node Features
Traditional GNN models are mostly studied for graphs with homogencous sparse node features, and leading
GNN models often fail to achieve competitive results for heterogeneous features with tabular or text node attributes [Ivanov & Prokhorenkova, 2021 Huang et al. 2020 Chen et al. 2021p. To remedy this, [vanov & Prokhorenkova (2021) y jointly train Gradient Boosted Decision Trees (GBDT) and a GNN in an end-to-end lashion, demonstrating a significant increase in performance on graph data with tabular node features. Chen
el al. [2021) removes the need for a GNN allogether, proposing a generalized framework for ilerating boosting with parameter-free graph propagation steps that share uode/sample information across edges connecting related samples. Still, both Ivanov & Prokhorenkova 2021 and Chen et al. (2021) h only study regimes where tabular node features favor the using of boosted decision trees, and are unlikely to remain competitive in broader use cases, e.g., textural node features or numerical embeddings. In contrast, we demonstrate a flexible, unified framework producing competitive performance across multiple domains involving tabular, numerical, and/or textual node features. Correct and Smooth (C&S) (Huang et al. 2020) is a simple post- past-processing method that applies label propagation to incorporate graph information into the outputs of a learning algorithm that otherwise agnostic to graph structure.
Turning to common GNN models that take numerical node features as inputs, one must establish way to extract numerical embeddings from raw data such as text and images. For example, the embeddings of ogbn-arxiv data are computed by running the skip-gram model (Mikolov et al. 2013). Chien et al. (2021) proposes self-supervised learning to fully utilize correlations wetween graph nodes, and extracts the embeddings of three open graph benchmark datasets (ogbu-arxiv, gebn-producte and (abo--pppes1000)) Chien et al. (0211) demonstrates the superior performance of these new embeddings for the Open Graph Benchmark datasets. [Lin et al. (2021) proposes BertGCN, which combines the Bert model and transductive learning for text classification in end-to-end fashion and achieves superior performance on range of text. classification tasks. However as with the boosting methods mentioned above, these techniques focus solely on one task, in contrast to our approach, where the in- built flexibiliiyy to incorporate different base models allows our hod to adapt to a variety of input lypes with a unified overarching design.
3 Background
Consider undirected graph g (V,E) with | nodes. The node feature matrix is denoted by X E Iaadd and the corresponding node label matrix is Y E Raxe with d and e being the dimension of features and labels respectively. The unweighted adjacency matrix is A E Rnx* For training purposes we only have access to the labels of a subset of nodes (y:hee, with L c V. Given feature values of all nodes (av,, label data (yrhec, and the connectivity of the graph E, the task is to prodict the labels of the unlabeled nodes heuu, with 2 = V.C. We denote the labeled dataset {x,, Y hee as De and the unlabeled dataset fieu as Dy-
3

---

Published in Transactions on Machine Learning Research (2/2024)
3.1 Bagging, Ensembling, and Stacking
with ITD (non-graph) data, bagging, ensembling, and stacking represent practical tools that can be combined in various ways to produce more accurate predictions relative to other strategies
For classification/regree
across diverse tabular and text datascts (Shi ct al. 2021; Blohm ct al.) 2020 Yoo ct al., 2020 Fakoor
ct al. 2020 Bezrukavnikov & Linder 2021 Feldman 2021). For example, in each stacking layer of an
ansemble-basee architecture, bagging simply trains the same types of base models with out-of-fold predictions from the previous layer models (obtained via bagging) as extra predictive features. These base models might
include various Gradient Boosted Decision Trees (Ke et al., 2017 Prokhorenkova et al. 2018), fully-connected
neural networks (MTP),, K Nearest Neighbors (Frickson et al.
2020). or pretrained Electra Transformer
models CCark et al. 2020). For instance, the AutoML package AutoGluon (Erickson et al. 2020 is an open-source code which is capable of exploiting these techniques.
3.2
Graph-Aware Propagation Layers as Energy Function Minimization
Recently there has been a surge of interest GNN architectures with layers defined in one-to-one correspondence with descent iterations that minimize a principled class of graph-regularized energy functions [Klicpera et al.
2018 Ma et al. 2020: Pan et al., 2021 Yang et al. 2021 Zhang et al., 2020 Zhu et al., 2021). In this way GNN models can benefit. from the inductive bias afforded by energy function minimizers (or close approximations thereof) whose specific form can be controlled by trainable parameters. For our purposes later in Section 4 attractive feature of this approach is that graph propagation can be conducted across arbitrary number of
without encountering undesirable oversmoothing effects (for a reproscntative empirical remonstration of this capability, please see Yang et al. (202T)) The latter can degrade the performance deep
GNN models by pushing all node embeddings to similar values [Li et al. 2018 Oono & Suzuki 2020
Following Zhou al. (2004), one relovant energy function capable of inducing such graph-aware propagation is given by
ey (Y) (1 (1- 1) m (X:0)1I Atr Y LY
(1)
where X E (0,1) a weight that determincs the tradc-off between the two terms. Y Rund is a learnable d-dimensional embedding across n nodes, and m (X; denotes a base model (parameterized by 0) that computes initial target embedding based on the node features X. L E pax" is the graph Laplacian of G, meaning L = D A, where D represents the degree matrix.
Intuitively, the first term of ① encourages Y to be close to initial target embedding, while the second term introduces the smoothness over the whole graph. On the positive side, the foored-form optimal solution of energy function
[ can be easily derived as
(x:0) a arg min (Y) P'm (X:0),
(2)
Y
with P A AL) However, for large graphs the requisite inverse is impractical to compute, and alternatively itcrative approximations are more practically-feasible. To this end, we may initialize as (()) = m (X;0), and it follows that Y can be approximated by iterative descent in the direction of the negative gradient. Given that
oly(Y) 2XLY + +2 2m (X:0),
(3)
OY
the 1-th iteration of gradient descent becomes
y00 = - - a + T)Y((-1) {
(4)
where serves as the effective step size. Considering that L is generally sparse, computation of 4 can leverage efficient sparse matrix multiplications, and we may also introduce modifications such as Jacobi preconditioning to speed convergence (Axelsson 1996 Yang et al. 2021).
Furthermore, based on well-known propertics of gradient descent, is sufficiently large and a is small enough, then
m (X:0) m (X; A p()) (m (X:0)),
(5)
4

---

Published in Transactions on Machine Learning Research (2/2024)
where the operator p() (.) computes t gradient steps via (4. The structure of these propagation steps, well as related variants based on normalized modifications of gradient descent, are analogous to principled GNN layers, such as those used by GCN (Kipf & Welling 2016), ; APPNP (Klicpera et al. 2018), and many others. And per the energy function association, these steps can be trained within a broader bilevel optimization framework without the risk of oversmoothing as described next.
4 Stack Ensembling for Graph Data (BestowGNN)
For node prediction tasks (either regression or classification), each (non-graph) base model is trained within our BestowGNN framework by simply treating cach node and its label as separate IID training example and fitting the model in the usual manner. Such model may informatively encode tabular or text features from the nodes, but its predictions will be uniformed by the additional information available in the graph structure. To enhance such models with graph information we utilize graph-aware propagation.
4.1 Incorporating Graph-Aware Propagation
Let y Yc, Yu denote the predictions of labeled (i.e. training) nodes and unlabeled (i.e. validation/test) nodes, respectively. In node classification tasks, these may be prodicted class probability vectors. Via iterative application of the update in 4), we can apply graph-aware propagation to predictions (Y c. Yu) } in order to ensure they reflect statistical dependencies between nodes encoded by the graph structure. We denote p(0) A (Ýc, c, Yuh. and for each propagation step we compute the update p(t) = ( E 4
y-00) via 4 In our method, Y may actually be predictions from multiple models concatenated together at each node, but the propagation procedure remains identical in this case.
4.2 Stack Ensembling
In stack ensembling, the predictions output by individually trained base models are concatenated together as features that are subsequently used to train a slacker model whose target is still to predict the original labels (Wolpert) 1992 Ting & Witten 1997). A good stacker model learns how to nonlinearly combine the predictions of base models into an even more accurate prediction. This process can be iterated in multiple layers, a strategy that has been used to win
prediction competitions with IID data (Koren 2009
In this work, we follow the stacking methodology of Erickson al. 2020)) but adapt it for graphs rather than IID data. We allow stacker models to access the original node features X by concatenating X with the base models' predictions when forming the features used to train each stacker model. To produce final prediction for each node, we aggregate predictions from the topmost layer models via a simple weighted combination where weights are Iearned via the efficient Ensemble Selection technique of Caruana et al. (2004). Our base models before the first stacking layer are those which can effectively encode the original tabular or text features observed the nodes (like Gradient Boosted Decision Trees for tabular features and Transformers for text features). Our stacker models are simply chosen as the same types of models as the base models.
4.3 Repeated K-fold Bagging to Mitigate Over-fitting
A problem that arises in the aforementioned stacking strategy is label leakage. If base model is even slightly overfit to its training data such that its predictions memorize parts of the training labels, then subsequent stacker models will have low accuracy due to distribution shift in their features between training and inference time (their features will be highly correlated with the labels during training but not necessarily during inference). This issue is remedied by ensuring stacker models are only trained on features comprised of base model predictions on held-out nodes omitted from the base model's training set.
We achiove this while still being able to train stacker models using all labeled nodes by levoraging K-fold bagging (i.e. cross-validation of all models (Van der Laan et. al.) 20077] Parmanto al. 1996 Erickson el al., 2020). Here the training nodes are partitioned into K disjoint chunks and K copies of each (non-graph-aware) model m are trained with different data chunk held-out {x-* y-*) held out from each copy. After training all K copies of model m, we can produce out-of-fold (OOF) predictions & for each chunk x²
5

---

Published in Transactions on Machine Learning Research (2/2024)
by feeding it into the model copy from which it was previously held-out. We repeat this K-fold bagging procedure over N different random partitions of the training data to further reduce variance and distribution shift that arises in stack ensembling with bagging. Thus for a labeled training node, the OOF prediction from a model of type m is averaged over N different partitions (this node is held-out from exactly one model copy each partition):
N
K
Y
k
(6)
N
m.n
k=1
Since unlabeled (validationn/lst) nodes were technically held-out from every model copy, can feed them through any copy without harming stacking performance. For particular type of model m, we simply make predictions Yu for unlabeled nodes by averaging over all N bagging repeats and all K copics of the model within each repeat:
K
N
Yu
k m.n.
KN
For IID data, this stack ensembling procedure with bagging can produce powerful predictors, both in theory (Van der Laan et al. 2007) and in practice (Frickson et. al. 2020
4.4 Stacking with Graph-Aware Propagation
To extend this methodology to graph data, our proposed training strategy is precisely detailed in Algorithm The main idea is to apply graph-aware propagation on the predictions of models at each intermediate layer of the stack. Different amounts of propagation lead to different characteristics of the data being captured in the resulting prediction (few steps of propagation means predictions are only influenced by local neighbors, whereas many propagation steps allow predictions to be influenced by more distant nodes as well). Thus we can further enrich the feature set of our stacker models by concatenating together the predictions produced after different numbers of propagation steps. With this expanded feature set, our stacker models learn to aggregate not only the predictions of different models, but differently smoothed versions of these predictions well. This allows the stacker model to adaptively decide how to best account for dependencies induced by the graph structure.
More precisely, given F(t), the predictions (concatenated across all base model types) for labeled and unlabeled nodes after smoothing steps, then the feature input to each stacker model is given by the original node features X concatenated with [F(O) ..... oblained via bagging. Another fundamental difference between our approach and stack ensembling the IID setting is the use of unlabeled (test) nodes at each intermediate layer of the stack. By including unlabeled nodes in the propagation, these nodes inffuence the features used to train subsoquent stacker models at labeled nodes. This can even further reduce potential distribution shift in the stacker models' features between the labeled and unlabeled nodes, which ensures better generalization.
(..0,,,..))) F(T)}. Here the predictions for labeled nodes are always OOF,
Graph machine learning models for non-lID data typically do not use bagging, seemingly because there has not been a rigorous study on the effect of bagging in relation to propagation models. Furthermore, bagging traditionally serves as a mcans of variance reduction which only brings limited performance benefits for large datasets (Breiman (996). In contrast, our stacking framework adopts bagging primarily as a means to mitigate the catastrophic effects of label leakage. While bagging can effectively mitigate label information from being directly encoded in stacker model features in the IID setting, it is not clear whether this property still holds with graph-structured dependence between nodes. A particular concern is the fact that the propagation of base model predictions across the graph implies label information is shared across the folo chunks used to hold-out some nodes from some models. In the next section, we theoretically study this issue and prove that bagging can still mitigate the effects of label leakage even in the non-IID graph setting. Our subsequent experiments (see Table 4 reveal that bagging produces substantial performance gains in practical applications of stack ensembling with graph propagation.

---

Published in Transactions on Machine Learning Research (2/2024)
4.5 Consideration of Alternative GNN-based Message Passing
While have adopted the energy-based message passing from Section 3.2 into framework, it may initially seem plausible to replace these graph propagation layers with a more traditional GNN architecture. However, to incorporate GNNs in this way would require, at each stacking layer, a separate inner-loop training process, meaning multiple epochs forward and backward passes through the GNN model to train all the parameters. contrast, the descent steps of the graph-reeuularied energy functions (1) we adopt lead efficient, parameter-Fe free message passing, so no inner-loop training is needed. Computationally speaking, our approach akin to just single forward pass of GNN, opposed numerous forward and backward passes would be required with training.
And incidentally, if we were to remove the GNN model parameters and simply incorporate the graph- propagation that remains, this would exacerbate the well-known oversmoothing problem mentioned in Section 3.2 whereby all node embeddings converge to similar values. In contrast, this does not occur with the emergy-based graph propagation we adopt, where even an infinite number of propagation steps does not, produce oversmoothing. To reiterate, this is possible because the energy minimizer itself explicitly designed not to oversmooth, and extra propagation steps only move closer to this minimum. Please supporting
references to this effect (Ahn et al. 2022
Ma et al. 2020 Pan et al. 2021
Yang et al. Zhang al.
2021
2020
Zhu et al. 2021
Output
Algorithm BestowGNN Training Strategy
Input: Node features and labels (X, Y) from graph G with labeled (training) nodes L and unlabeled validation/test nodes 44 family of models intended for ID data M, L stacking layers, N-repeated, K- fold bagging, for - 1 to L do {stacking)
Stacking Layer
propagation sleps.
Input
+
(F(O), F(1) F(D}
for = to N do (repeated bagging)
Randomly split data into K chunks
[x* kkk ===
T' steps Graph Propagation for F(0)
for k 1 K do
[(r)) MEM
{Pm)
MEM
Train model m E M on fx~* Make predictions P m,n on OOF data x
end for
OOF Prediction y
Test Validation Prediction y
end for
for MM do
Get OOF predictions y for labeled nodes via
6
Get predictions Y. for unlabeled nodes via
Model m E M
7
end for
Concatenate all models' predictions: F()) & HY C JMEM, { ImmM]
Chunk
Chunk
Chunk 3
for t = 0 to T do {propagation]
JeM Compute F(() [( (0) MEm ing 4 end for X +- concatenate (X, {F((O) F
Input Data
end for Output: weighted prediction
Figure 1: BestowGNN with single base learner m, 2 stacking layers, and 3-fold bagging (repeated bagging not depicted here). The stacking layer repeats the operations depicted between it and the input data.
Σ
am mm
mEM
with {am) fitted via. Ensemble Selection

---

Published in Transactions on Machine Learning Research (2/2024)
5 Theoretical Analysis of Label Leakage
Label utilization is a common technique in which the outputs of a model are concatenated with input features and then used to train a stacking layer. Unfortunately, layer-wise training with label utilization is susceptible to the label leakage problem. Recall that previons layer-wise training methods SAGN (Sun & Wu 2021 and GAMLP (Zhang et al.) 2021) both suffer from the risk of label leakage: the first layer, SAGN and GAMLP train the model and predict the label of training points, then predicted labels are included in the enhanced training set. In the second layer, they use the enhanced training set the new feature to train new model. Notice here including predicted labels as training features can cause performance degradation if the model extracts and relies on these predicted labels. The second layer training upon first-layer predictions could amplify over-fitting issues and introduce the covariate shift at test time. Although prior work (Sun & Wu 2021 Zhang et al. 2021) has mentioned heuristic ways to address label leakage via graph propagation, it is unclear how generally applicable this strategy in practice. Morcover, there natural tradc-off between avoiding label leakage via graph propagation, and well-known oversmoothing effects in GNN models.
In this section employ powerful theoretical tool, Differential Privacy (Mironov 2017), to showcase the advantage of bagging in our proposed BestowGNN. Our analysis will show that BestowGNN enjoys strong generalization under the Rényi Differential Privacy framework. In fact this is the first work that establishes that bagging in graph predictors is useful and mitigates label leakage. Specifically, Bestow can preserve the privacy (or information sharing) of labels between bags, that would otherwise be compromised by graph propagation.
To this end, we first introduce the definition of Rényi Differential Privacy, which is relaxation of Differential Privacy based on the Rényi Divergence.
Definition 1. (Rényi Differential Privacy (Mironor) 2017)) Consider randomized algorithm M mapping from D to real-value R. Such on algorithm is said to have e- e-Rényi Differential Privacy of order a if for any D' D' E DD with du (D, D') = 1, where du is the Hamming distance (D, D' are also referred to as adjacent datasets), we have that
Da(M())IM(D))
M(D) M(D') Se.
)²
log FEzMIDD)
In plain words, this definition establishes that the output of an algorithm does not change significantly, as measured by the Rényi divergence Da (MM(I))))) when the data changes slightly. The idea behind this framework is that if each individual data sample has only a small effect on the resulting model, the model cannot be used to infer information about any single individual.
We then have the following result:
Theorem 1. Assume base model is malli-layer (two-layeer) perceplron and that node features X are sampled from multivariate Gaussian as in [Jia 8 Benson 2021):
X ~~N10 X~N(0,T
T= + CoL,
where In is an idenlily matria and L is the normalized graph Laplacian. Here ei controls a noise level and O2 the smoothness over the whole graph. E(ag; Dc) and F(x; Dz) are predictions produced by BestowGNN for a data point xo with and without bagging mode, respectively. If E has sensitivity 1 and lower magnitude bound L, i.e., for any two adjacent D' E D: E(z;; -) D')l < and EE| 2 L, then E satisfies ()) 1o-1 + RRényi Differential Privacy, where o² depends on graph structure G. Meanwhhile, F has no privacy guarantee, i.e., the Rényi differential privacy loss (8) is unbounded.
The proof is deferred to the appendix. Theorem I indicates that bagging with graph propagation can well preserve the privacy of Dr = {æ., y. y.j bet different chunks while non-bagging would have a high risk of leaking the information of Dc. For layer-wise training with label utilization, the output of the model E(z;; Dz) concatenated with input features and then used to train next stacking layer, and bagging can effectively mitigate the label leakage issue since the information of true label is well preserved at the first layer, while no-bagging exposes the true labels and can lead to wverffitting issue for next stacking layer.
8

---

Published in Transactions on Machine Learning Research (2/2024)
6 Experiments
Datasets. We study the effectiveness of our approach by comparing performance against a. variety of baselines in node regression and classification tasks. For node regression with tabular node features, we consider four real-world graph datasets used for benchmarking by [vanov & Prokhorenkova (2021): House, County, VK and Avazu. As node classification tasks, we adopt one dataset with numerical features: Reddit; and two datasets with raw text featurcs: OGB-Arxiv and OGB-Products. We also consider the OGB dalasets using original OGB provided features. More details about the dalasets are provided in the appendix.
Baseline approaches. We compare our method against various baselines, starting with purely tabular bascline models or language models where the graph structure is ignored. Our first bascline is Autogluon Ericksson et al. 2020), an AutoML system for IID tabular or text data that is completely unaware of the graph structure (here we simply treat nodes ITD.. Next, we consider AutoGluon + C&S, which performs Correct and Smooth (Huang et al.) 2020) post hoc processing step on top of AutoGluon's predictions, in order to at least account for the graph structure during inference. For node regression tasks we also consider some popular GNN models: GCN (Kipf & Welling, 2016), , GAT (Veličković et al.) 2017), and a hybrid strategy BGNN QIvanov & Prokhorenkova 2021), which combincs Gradient Boosted Decision Trees (also a model intended for IID data) with GNNs via end-lo-end training in a manner that is graph-aware.
For node classification using Reddit with original numerical features, we also compare with GraphSAGE (Hamilton et al., 2017) and PCAPass + Tree (Sadowski et al., 2022), which combines PCA and message passing to generale node embeddings and leverages tree-based model lor node classification.
For OGB-Arxiv and OGB-Products with original OGB features (pre-computed low-dimensional text em- beddings as node features provided by OGB), we consider standard GNN models: GCN (Kipf & Welling 2016) and GAT (Veličković et. al. 2017) variants, and Ensemble GCN, a natural which divides all training nodes into K chunks, trains a GCN model for each chunk and then ensembles the results. We also compare against SOTA models for OGB-Arxiv and OGB-Products from the OGB leaderboard (restricted to original features, text augmentation) when initially started the experiments BoONN++BTTssl--KK
for Arxiv and
for Products).
For OGB-Arxiv and OGB-Products with raw text as node features (these are necessarily more informa- tive than the compressed original OGB features), we chose the method that topped the respective OGB leaderboard at the time of our submission: this was TAPE+ and BLEM--GIATT-SAGN-SSCC
(He et al., 2023) for OGB-Arxiv (Zhao et al., (023) for OGB-Products. To assess the adaptability of these two approaches, we retrain both models with standard hyperparameter tuming on the opposile dataset. for which they topped the leaderboad, meaning we also tested RevGATT on OGB-Products and GLENA+GIANTSSSANNNSCCC on OGB-Arxiv. This is a reasonable scenario since both models are designed to handle text features, and yet given their complex, composite natures, it is unclear how well they might reliably transfer to different datasets.
BestowGNN details. We evaluate our method RestowGNN, which incorporates base models and graph information through propagation operations within each stacking layer. For the base models, consider LightGBM boosted Tress (GBM) (Ke et al., 2017), CatBoost boosted trees (CAT) (Prokhorenkova et al. 2018), fully-connected neural networks (NN), Extremely Randomized Trees (RT), Random Forests (RF), K Nearest. Neighbors (KNN), Label Propagation (I.P) (Huang et al. 2020) and Transformer with electra pretrained model (Text) (Training epoch is 12) (Clark et al. 2020). , For the first layer, we keep the typical models, for example, Gradient Boosted Docision Trees for Tabular data, Transformer models for text data. For second stacking layer, we use all of models except extremely low-efficient models for large dataset, for example, KNN and Catboost slow down the training procedure for the OGB-products dataset. All details regarding the base models, well as the minor/standard hyperparameter tuning used by BestowGNN to fit all datasets, can be found in the appendix.
Results. In Table 1 we present the results for the node regression task with tabular node features. The bascline GNN models are challenged by the tabular node features. AutoGluon an ensemble of various base models (e.g., Gradient Boosted Decision Trees, fully-connected neural networks) intended for IID data without.
9

---

Published in Transactions on Machine Learning Research (2/2024)
Data set
House
County
Vk
Avazu
GCN
0.63 + 0.01
1.48 + 0.08
7.25 + 0.19
0.1141 + 0.02
GAT
0.54 + 0.01
1.45 + 0.06
7.22 + 0.19
0.1134 + 0.01
BGNN
0.50 = 0.01
1.26 + 0.08
6.95 + 0.21
0.109 + 0.01
AutoGluon
0.618 = 0.01
1.379 = 0.08
7.176 + 0.21
0.117 = 0.018
AutoGluon + C&tS
0.477 = 0.01
1.162 = 0.09
6.995 + 0.21
0.107 + 0.015
BestowGNN
0.467 0.007 1.145 + 0.083
6.918 0.220
0.105 0.013
Table 1:
Mean squared error of different methods for four different node regression datasets (The top model is boldfaced, the second-best model is underlined).
Table Node classification accuracy for Reddit with numerical node features (The best model boldfaced, the second-best model is underlined).
Method
Reddit
PCAPass + XGBoost
96,26 + 0.02
GraphSAGE
95.40 = 0.22
AutoGluon
95.83 + 0.00
AutoGluon + C&S
96.00 + 0.00
BestowGNN
96.44 10.00
considering graph structure. We observe that Autogluon + C&S outperforms Autogluon, demonstrating that graph information can greatly boost the performance of models intended for IID data. Incorporating the graph structure at each stacking layer, our BestowGNN method performs better than prior baselines on all datascts. While the performance difference relative to the top performing alternatives may be modest at times, as we will soon observe, BestowGNN consistently at or near the lop spanning all datasels and scenarios. As such, a cumulative case can be made for the efficacy of BestowGNN relative other approaches that excel only in limited domains.
Next, Table 2 shows that BeStowGNN outperforms the baselines for the Reddit dalaset with numerical embeddings, while PCAPass + XGBoost is second. Turning to Table 3 we display results for OGB-Arxiv and OGB-Products, and highlight the superior performance models trained using raw text features as would be expected. Of particular note though is the performance of the top OGB leaderboard methods when retrained on a different dataset. Specifically, while the time of our submission), when transferred of 77.50 here would only rank 16th on the current Arxiv leaderboard). Similarly, although + may have been tops for AGIvA it is not especially competitive places only 30th as of the current OGB-Products leaderboard). We also note that AutoGlnuonn&&ss which was competitive on the tabular benchmarks in Table [ is far less competitive using OGB data, especially on OGG-Products. In contrast, BestowGNN uses essentially the same core architecture, which includes the incorporation of graph information within each stacking layer, to fit all datasets spanning Tables
may be best on DGB-Products (at. OGii-Axiv there is considerable drop-off (its accuracy
OGB-Proodcct (its accuracy of 82.16
2 and
The key ingredients of our framework are bagging/ensemmbln /ensemblingg and graph propagation. Table shows an ablation study involving these components using OGB-Arxiv with original OGB embeddings. From the above results, we observe that in each case the training performance under the no-bagging set lenting is always higher than bagging as would be expected if some degree of overfitting were occurring. In contrast, on the test (and validation) scts, the situation is reversed and no-bagging now outperforms bagging. This indicates that bagging has helped mitigate some of the effects of overfitting by reducing the gap between training and testing accuracy. For additional ablations, please the appendix.
Ablation.
10

---

Published in Transactions on Machine Learning Research (2/2024)
Table Node classification accuracy for AGB-Arxiv and OGB-Products (The best model boldfaced, the second- best model is underlined). Rows labeled TEXT contain involve models trained on raw text embeddings/ features, while those labeled OGB indicate models trained on precomputed numerical embeddings provided by OGB as node features. As of the time of our submission RevE+ was the top performing model on OGB-Arxiv while GLEM+GIANT++SANN +SCR was the top performing model on OGB-Products however, in both cases relative performance degrades when shifting to different dataset, despite the similar text-based modeling scenarios. Meanwhile, BestowGNN maintains competitive performance spanning these and other datasets/scenant
OGB-Arxiv
OGB-Productss
(Validation)
Foature
Test
(Validation)
GCN GAT + CAS
73.06 1 0.21 (71.12 + 0.12)
DCeperGNN FLAG
81.93 0.31 (91.08 0.01)
OGB
OGB
73.86 1: 0.14 (74.84 1 0.07)
GAT + FLAG
81.76
0.45 (92.51
0.06)
AGDN BoT s C&s
74.31
0.14 (75.18 + 0.09)
GAMLP RLU SCR CCKS
85.20
0.08 (93.04 0.05)
Enwemble GCN
73:22 + 0.12 (74.64 + 0.01)
Ensemble GAT
80.01
+0.20 (93.24 + 0.06)
TEXT
GLEBI-GLANT+SAGN--EE TAPE: RevGAT
0.08)
TEXT
CLAM--GIAT-SAGNN TAPE: ReyGAT
(94.00 0.03)
77.50 :: 0.12 (77.85 a: 0.16)
82.16 0.28 (92 10 0.09)
TEXT
AutoGhoo
73.05 +: 0.00 (74.33 + 0.00)
TEXT
AstoGlaon
17.10
0.00
(91.78 0.83)
AutoGlacu + C&8
75.31 1 0.00 (76.67 + 0.00)
AutoGluon C&S
79.03
0.12 (93.02 0.03)
TEXT
BNNtowGNN
78.19 4: 0.02 (77.25 + 0.05)
TEXT
BestowCNN
85.48 0.03 (90.90 0.02)
Table 4: BestowGNN ablation study with (V) and without bagging (X). Here T' is the number of graph propagation steps, thus T' = represents a baseline model that completely ignores graph structure.
STEP T
TRAIN
VALIDATION
TEST
X
0.64
0.68
0.58
0.56
0.56
0.54
0.76
0.78
0.67
0.66
0.67
0.65
0.77
0.78
0.70
0.68
0.70
0.68
0.77
0.78
0.71
0.69
0.70
0.68
0.77
0.79
0.71
0.70
0.70
0.69
0.79
0.81
0.72
0.71
0.71
0.69
Table 5: Training time Lested on AWS g4dn. g4dn.12xlaal 12xlarge machine.
DATASET
BASE MODEL
TIMR(S)
HOUSE
GBM, NN
52
COUNTY
GBM, NN
18
VK
GBM, NN
119
AVAZU
GBM, NN
15
OGB-A
NN
199
PEBB-PROUCTS
NN
837
Computing cost. The computing cost depends the ensemble models select (e.g., transformer models can take more computing resources relying on the implementation, including more emsemble models lcads to more computing cost). As a result, it is difficult to consistently measure the training/inferee memory consumption. Nonetheless, the computing cost lies in a competitive range since the integration of the bagging and ensembling parts key to our model can be efficiently implemented, c.g., via open source packages like AutoGluon that we used. In Table [ we present the training time for different datasets with basic ensemble models on AWS g4dn.12c 12x Large machine.
time or
11

---

Published in Transactions on Machine Learning Research (2/2024)
7 Discussion
While real-world graph data come with heterogeneous feature types, existing GNN models are primarily suited for (adequately preprocessed) numerical features. For IID supervised learning, it is well-known that the best models for different feature types vary based on dataset and data-type, and that a learning system aiming to output good predictions across variety of datasets should leverage a heterogeneous collection of different types of models (Erickson et al. 2020). There little reason the situation should be different for graph data. In this paper, demonstrate the first working system that utilize arbitrary helerogeneous collections of models for arbitrary graph datasets with heterogeneous feature-types (numerical, categorical, text). This is achieved by means of a novel graph-aware stack ensembling technique that takes the graph structure into account without restricting how individual models are trained. Our graph-aware propagation techniques leverage specific properties of stack ensembling that allow our proposed methodology to outperform both many complex GNNs as well as existing approaches in which propagation only applied to the predictions output by an ITD base model (e.g., AuloGluon + +C&S, elc.).
12

---

Published in Transactions on Machine Learning Research (2/2024)
References
Ahn, H., Yang, Y., Gan, Q. Moon, T., and Wipf, D. P. Descent steps of a relation-aware energy produce heterogencous graph neural networks. In Advances in Neural Information Processing Systems, volume 35, 2022.
Axelsson, O. Iterative Solution Methods. Cambridge University Press, 1996.
Bezrukavnikov, O. and Linder, R. A neophyle with automl: Evaluating the promises of automatic machine learning tools. arXiv preprint arXiv:2101.058400 2021.
Blohm, M., Hanussek, M., and Kintz, M. Leveraging automated machine learning for text classification: Evaluation of automl tools and comparison with human performance. arXiv preprint arXiv:2012.0357 2020.
Breiman, L. Bagging predictors. Machine learning, 24(2):123-14), 1996.
Caruana, R., Niculescu-Mizil, A., Crew, G., and Ksikes, A. Ensemble selection from librarics of models. In Proceedings of the twenty-fire first international conference on Machine learning, pp. 18, 2004.
Chen, J., Mueller, J., Toannidis, V. N., Adeshina, S., Wang, Y., Goldstein, T., and Wipf, D. Convergent boosted smoothing for modeling graph data with tabular node features. arXiv preprint arXiv:2110.13./ 2021.
Chiang, W.-L,, Liu, X., Si, S., Li, Y., Bengio, S., and Hsieh, C.-J. Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 257-266, 2019.
Chien, E., Chang, W.-C., Hsieh, C.-J,, Yu, H.-F., Zhang, J., Milenkovic, O., and Dhillon, S. Node feature extraction by self-supervised multi-scale neighborhood prediction. arXiv preprint arXiv:2111.000 2021.
Clark, K., Luong, M.-T,, Le, Q. V., and Manning, C. D. ELECTRA: Pre-training text encoders discriminators rather than generators. In International Conference on Learning Representations, 2020.
Erickson, N., Mueller, J., Shirkov, A., Zhang, H., Larroy, P., Li, M., and Smola, A. Autogluon-tabular: Robust and accurate automl for structured data. arXiv preprint arXiv:2003. 066505, 2020.
Fakoor, R., Mueller, J., Erickson, N., Chaudhari, P., and Smola, A. J. Fast, accurate, and simple models for tabular data via augmented distillation. In Advances in Neural Information Processing Systems, 2020.
Feldman, Which machine learning classifiers are best for small datasets? An empirical study. https:
//www. //www.date dawa-cowboasss....ooooooooooa... sitiraaccoooosscc/blllo/// 2021.
Feurer, M., Klein, A.,
Eggensperger, K., Springenberg, J., Blum, M., and Hutter, F. Efficient and robust. automated machine learning. Advances in neural information processing systems, 28, 2015.
Hamilton, W. L., Ying, R., and Leskovec, J. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 1025 1035, 2017.
He, X., Bresson, X., Laurent, T., Perold, A., LeCun, Y., and Hooi, B. Harnessing explanations: Lim-to-Im interpreter for enhanced text- aext-attributed graph representation learning, 2023.
Hu, W., Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., and Leskovee, J. Open graph benchmark: Datasets for machine learning on graphs. Xr preprint arXic:2005.006 00687,, 2020.
Huang, Q., He, H., Singh, A., Lim, S.-N., and Benson, A. R. Combining label propagation and simple models out-performs graph neural networks. arXiv preprinl ar 201: 13993, 2020.
Ivanov, S. and Prokhorenkova, L. Boost then convolve: Gradient boosting meets graph neural networks. arXiv preprint ar arxi:22101.880 2021.
13

---

Published in Transactions on Machine Learning Research (2/2024)
Jia, J. and Benson, A. R. A unifying generative model for graph learning algorithms: Label propagation, graph convolutions, and combinations. arXie preprint 2Xiv:2101.07730 2021.
Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q and Liu, T.-Y. LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems, 2017.
Kipf, T. N. and Welling, M. Supi-super classification with graph convolutional networks. arXiv preprint arv: wwXii:1609.000 609. 029007 2016.
Kliepera, J., Bojchevski, A., and Günnemann, S. Predict then propagate: Graph neural networks meet personalizod pagerank. arXiv preprint arXi::1810.0500 859907997, 2018.
Koren, Y. The bellkor solution to the netflix grand prize, 2009. URL https://wwo2. seas. gww.edu/ -simhaweb/ http:///w..aaaagg/gaaaa/aaa cf httpp/pperr//oooeeeeoorrrro pdf
LeDell, E. and Poirier, S. H2o automl: Scalable automatic machine learning. In Proceedings of the AutoML Workshop at ICML, volume 2020, 2020.
Li, Q. Han, Z., and Wu, X.M. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Lin, Y., Meng, Y., Sun, X., Han, Q., Kuang, K., Li, J., and Wu, F. Bertgen: Transductive text classification by combining gen and bert, 2021.
Ma, Y., Liu, X., Zhao, T., Liu, Y., Tang, J., and Shah N. A unified view graph neural networks graph signal denoising. arXiv preprint arXic:2010.01771 2020.
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111-3119, 2013.
Mironov, 1. Rényi differential privacy, In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pp. 263-275. IEEE, 2017.
Oono, K. and Suzuki, T. Graph neural networks exponentially lose expressive power for node classification. In 8th International Conference on Learning Representations, ICLR, 2020.
Pan, X., Song, S., and Huang, G. A unified framework for convolution-based graph neural networks, 2021. URL https://openrevier wwt.:eoootndddiileeeeee http://paarrrieiee.eeeeraaaaaaaa
200fformmmmd222000--P-P33 -b9Btt
Parmanto, B., Munro, P. W., and Doyle, H. R. Reducing variance of committee prediction with resampling techniques. Connection Science, 8(3-4):405-422 1996.
Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., and Gulin, A. CatBoost: unbiased boosting with categorical features. In Advances in Neural Information Processing Systems, 2018.
Sadowski, K., Szarmach, M., and Maltia, E. Dimensionality reduction meets message passing for graph node embeddings. arXiv preprint ar erXi:22002.00 00408, 2022.
Shi, X., Mueller, J., Erickson, N., Li, M., and Smola, A. J. Benchmarking multimodal automl for tabular data with text fields. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarkks, 2021.
Sun, C. and Wu, G. Scalable and adaptive graph neural networks with self-labeleehhaneed training. arXiv preprinl ar Xiv:21104 093776 2021.
Ting, K. M. and Witten, I. H. Stacking bagged and dagged models. In International Conference Machine Learning, 1997.
Van der Laan, M. J., Polley, E. C., and Hubbard, A. E. Super learner. Statistical applications in genetics and molecular biology, 6(1), 2007.
14

---

Published in Transactions on Machine Learning Research (2/2024)
Van Erven, T. and Harremos, P. Rényi divergence and Kullback-leibler divergence. 1EEE Transactions Information Theory, 60(7):3797-3888 2014.
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., and Bengio, Y. Graph attention networks. arXiv preprint ar 10900, 2017.
Wang, Y., Jin, J., Zhang, W., Yu, Y., Zhang, Z. and Wipf, D. Bag of tricks for node classífication with graph neural networks. arXiv preprint arX 13355, 2(3), 2021.
Wolpert, D. H. Stacked generalization. Neural networks, 5(2):241-22) 2599, 1992.
Yang, Y., Liu, T., Wang, Y., Zhou, J., Gan, Q. Wei, 7.,, Zhang, Z.,, Huang, Z., and Wipf, D. Graph neural networks inspired by classical iterative algorithms. arXiv preprint orXiv:2103.0606, 2021.
Yoo, J., Joseph, T., Yung, D., Nasseri, S. A., and Wood, F. Ensemble squared: A meta automl system. arXiv preprint Xiv: 05390, 2020.
You, Y., Chen, T., Wang, Z., and Shen, Y. L2-gen: Layer-wise and learned efficient training of graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1127-2135 2020.
Zhang. H., Yan, T., Xie, Z., Xia, Y., and Zhang, Y. Revisiting graph convolutional network semi-supervised node classification from optimization perspective. arXiv preprint Xitx:2009.11469 2020.
Zhang, W., Yin, 7., Sheng, 7., Ouyang, W., Li, X., Tao, Y., Yang, Z., and Cui, B. Graph attention multi-layer perceptron. arXiv preprint ar arXiv:1008.10097 2021.
Zhao, J., Qu, M., Li, C., Yan, H., Liu, Q., Li, R., Xie, X., and Tang, J. Learning on large-scale text-attributed graphs via variational inference, 2023.
Zhou, D., Bousquet, O., Lal, T. N., Weston, J., and Schölkopf, B. Learning with local and global consistency. Advances in Newral Information Processing Systems, 2004.
Zhu, M., Wang, X., Shi, C., Ji, H., and Cui, P. Interpreting and unifying graph neural networks with an optimization framework. arXiv preprinl 2xx::2101.118333 2021.
15

---

Appendix:
A Proof of Theorem 1.
Preliminary 1. Firstly, derive the format of Exxo; De) and F(wo; Dz). Suppose BestowGNN randomly splits the labeled nodes Dr into 2 disjoint chunks D₁ = (X,, Y 11),D. D2 = 2x Y2). BestowGNN trains a model m E M with a different data chunk held-out. Model m is defined by a set of parameters collected in θ namely, which defined as m(X;0). In the following, we will express the predicted labels from model m under the bagging and non-bagging settings. We compare the predicted labels under both settings and establish that our bagging solution is less amenable to label leakage.
The model will learn different parameters for each chunk and those denoted as 01 for the chunk I and O2 for the chunk II, namely 0, = (Di) and 0₂ (D))) Next, BestowGNN produces prediction Y,, Y2 on out-of-fold data, i.e., Yı = m(X;; and Y 2 m2x2; 01). The prediction for unlabeled nodes is Yu = 2 [mm((xxr 01) m Xu; 02)] as explained in 7 Consider one data point 2o from the unlabeled dataset.
Du, the prediction of zo is given by Ún = (m b1) + m( 02)]. Next, we perform one step graph-aware propagation on Do-
30
Σ (o
Σ
D
Σ
VEN(ZO)) D1
wEN(
(9)
Σ
[m(x 01) + m(x 02)1 +
m(x 02)
m(<w; 02),
2
(Z nD
WEN(Za)DD:
where yo is the aggregated results from one-hop neighbor (xxo), which may belongs to Du, Di and D2.
Next, consider the no-bagging mode, where the predictions of X Xa are changed into Y m X 1; 01) and Y2- = ml [xx 22:00 02). Notice that with bagging mode we use the parameters from a different bag, while without bagging we use the parameters from the same bag. The prediction of the test point ag is once again yo f : 0 + m( 02)], which is identical to the bagging mode. We perform the same graph-aware propagation Vo.
36
Σ
yn +
yr +
(E
wEN(Z,)nDz
(10)
Σ
(m( 01) m( 02)l +
Σ
m( 01) 01)+
Σ
m( 02).
.EN(ZE))
WEN
The first term Enn {mm( 01) m(x 02)] is the same for (9) and (10) and can be cancelled. order to facilitate the exposition of the theoretical contributions we will define functions for the different terms in g
Next, we compare the terms among the predicted labels from the two settings, namely and 10
and (10p. We define E(x;; Dz), that is a function formulating the relation between training data De and the prediction for test data æg under bagging mode.
Elzo: := :=
Σ )DD
m(x O(D)))
Σ
m( (D))).
(11)
Similarly, we define the function Faro; De) formulating the relation between training data De and the prediction for test data Do under the no-bagging mode:
F( :=
Σ
m(x +(D)))
Σ
(12)
(2
Notice here ((2)) is the model parameters of Chunk involving information of true label Y1. We aim examine bagging and stacking strategies effectively preserve the information of label Y, via introducing

---

Published in Transactions on Machine Learning Research (2/2024)
randomness to the function E D2) while Fæu;; Dz) has high risk of leaking the information of true label Y1.
We first reiterate the definition of Rényi Differential Privacy.
Definition 1. (Rényi Differential Privacy (Mironool ). Consider randomized algorithm M mapping from DD to real-value R. Such an algorithm is said to have e-Rényi Differential Privacy of order a (a > 1) if any D,D' D' E D with du (D, = 1, where du is the Hamming distance (D, D' are also referred to as adjacent datasets):
M(D)
Da(MD))IM()) a - log EE-M(D'
c.
(13)
To proceed in quantifiable way, we rely some preliminary results for Rényi Differential privacy and generative model for graph learning algorithms.
Proposition 1. Rényi differential privacy is preserved by post-processing (Mironov, 2017/. If F(-) has e-Rényi Differential Privacy, then for any randomized or determänistic function 9, 9(F satisfies e-Rényi Differential Privacy.
Proposition 2.
The closed--frrm expression of the Rényi divergence belween any two Gaussian distributions = a(u ppo) I-a In
is given by Da (N
an oo provided that o² = (1- aajo + tof >
20
Van Erven & Harremos 2014)
Proposition 3. Assume f has sensitivity 1 and lower magnitude bound L, i.e., for any pair of adjacent datasets D' e D: - f(D') 1 and Ifl 2 L, and define the Gaussian maltiplicative mechanism
GM ((D) F(D)N
Then GM satisfies ( -utا + 217 -ennyi Differential Privacy.
Proof. According to Proposition 2
D1y2 (N(f(D) - μ. (I
20 (P² +m1, 1 -00
(S(D) I(D))
((D)F(D")
+ +n1, ( (PDD)++(100))
20
(J((D)
1
[(D)((D) +In ((D))
20
+ ((((D)
2F(D)((D)
<
(2(D) + (2(D") + In(
2T((D)(())) +1)
20
1
< 20
+ In(
+1)
212
1
< 40
+ 2L
The first inequality follows from |((D)- 1 take square for both side ((()++ < $142) (DD)
20d (AD)))A(( and (D()) (DD)) I . +1, the first inequality Then we have 2 ((( holds.
I 1
Proposition 4. If f has sensitivity 1, i.e. for any pair of adjacent datasets D, E D: (f(D) f(D') < 1. Define the Gaussian additive mechanism
GA f(D)++(((..
then Ganssian additive mechanism GA.S satisfies (a, ) RRényi Differential Privacy
Mironoe 2017
17

---

Published in Transactions on Machine Learning Research (2/2024)
Proposition 5. Consider multivariate Gaussian distribution, and the rondom variables partitioned inlo two groups (zp, zQ), the distribulion is block malria formal
2P 2Q)
N
(
Ep [
Ppp For Tool
where Tpp TrQ TQp Tool can be wrillen as
is precision (inverse covariance) matria. Then the marginal and conditional distribution
2p N (2p, (Tpp- IPq Q0 poP))
(14)
zr|zq 2q ~ N (zp T =Q))
(15)
Before proceeding to our specific results in the main paper, we also need to describe the graph setting.
Preliminary 2. Let g = (V, E) be an undirected graph, where V the set n nodes and is the set of edges. The adjacency matrix G is W Raxn the diagonal degree matrix is E Raxn, The normalize graph Laplacian can be written as N D=I--"/WWD matrix, where p is the dimension of features. We assume all vertex features X are jointly sampled from multivariate Gaussian distribution (Jia & Benson 2021), namely
S. We use X Raxp for the feature
X ~ N~N(O,
T= F=al+eg +c2N
(16)
where I, is identical matrix, N is normalized graph Laplacian. Here controls noise level and controls the smoothness over the whole graph.
We now proceed to our specific results in the main paper.
Theorem Assume base model m to a muulii-laye (two-layer) perceptron and node features X is sampled from multivariate Gaussian as in Jia Benson 2021
X X~N(O,T
T= +
where In is identity matric and L is the normalized graph Laplacian. Here eı controls noise level and 02 controls the smoolkmess over the whole graph. E(wo; Dz) and F(æn; Dc) are predictions produced by BestowGNN for data point va with and without bagging mode, respectively. If E has sensitivity 1 and lower magnitude bound L, i.e., for any two adjacent D, D' E D: [E(w;; D) E(vo; D')l S 1 and |E| > L, then E satisfies (2, 13-12 + 21,2 RRéngii Differential Privacy, where o² depends on graph structure G. Meanwwhile, F has no pricucy guarantee, i.e., the Rényii differential privacy loss ( unbounded..
Proof. Given the definition function E from above, we have that
E(xo; Dc)= =
Σ m(x:; O(D2)) +
Σ
m(# (()))
WEA
(17)
Σ
m(x,0(D,)) +
Σ
m(x ((D)))
(E)
"EN(T.))DD
where the second equality follows from the MLP assumption. Similarly for F we have
F( D2)
Σ
m(
0
(DD
(2
(18)
Σ
m m(z.0(Di))
+
Σ
m(x O(D2))
wEN((E,)DD;
We now define the adjacent datasets D and D' follows. Assume D1; one data point (x'.y)} then randomly selected from Chunk I and removed (æ', yf} from D, forming D' Dı Di v'}. Meanwhile, the
18

---

Published in Transactions on Machine Learning Research (2/2024)
Table Training time tested on AWS g4dn. 12xlarge machine.
DATASET
BASE MODEL TIME(S)
HOUSE
GBM, NN
52
COUNTY
GBM, NN
18
VK
GBM, NN
119
AVAZU
GBM, NN
15
OGB-ARXIV
NN
199
OGB-PRODUCTS
837
unlabeled set Du and D2 remain the same.. Our goal is to examine the extent to which E and F may leak information pertaining to {o', y when {a', y'} removed from D₁ as described above.
I Fm
Denote v. Tw as training data in chunk and chunk II. Assune distribution:
is drawn from multivariate Gaussian
xv Ew
N
(1)
Tep Two
T Tow
)
(19)
T Tww
T Fwr aI + bN, I is identical matrix, N is normalized graph Laplacian, controls noise level and b controls the smoothness over the whole graph.
T ov
where
T wee
From Proposition the condition distribution of aw given H. = By can be written as
Fw Fy By (- wi Te P Twwl).
Condition on the data D,, the distribution of D2 is conditional multivariate Gaussian distribution with ww- Furthermore, multiplicative Gaussian distribution T ((D)) introduces
mean -rwir w and variance r-1
a Gaussian random noise into 17 According to Proposition (1) and (3) E satisfies (5 462 + 27, RRnni
Differential Privacy, where o² depends on r-1
ww decided by graph structure.
Meanwhile, although (18) is deterministic, we can manually add Gaussian noise (00 o²) such that F satisfies ---Rééníi Differential Privacy via Proposition (4). However, if we then let -> 0 to reproduce F, we have that 2 + 00, indicating that in fact F has no privacy guarantee.
]
B Experiment Details
B.1 Data descriptions
node features are the property of house, edges connect the neighbors, the task is predict the price of the house. County: cach node is a county and edges connoct two countics sharing a border, the lask is to predict the unemployment rate for county. VK: each node person and edges connect two people based the friendships, the task is to predict the age of each person. Avazu: each node is device and edges connect two devices if they appear on the same site with the same application, the target is the click- thick-through-rate of a node. For House, County, VK and Avazu datasets, Training/validatioon) ieliaatioottesttnggg besting are randomly split. with 6/2/2 ratio and all experiments results are averaged over 5 trails.
House:
OGB-Arxiv, OGB-Products are standard datascts from DGB-leaddrbsard and all training training/validaat walaavaaioontttttiiiaaieeoo splits follow the standard data splitting from OGB-leaderbaard. Reddit is standard datset from Deep Graph Library (DGL...
19

---

Published in Transactions on Machine Learning Research (2/2024)
Table Dataset Statistics in Number of Classes means regression task)
DATASET
NUMBER OF NODES
NUMBER OF EDGES
NUMBER OF CLASSES
HOUSE
20,640
182,146
COUNTY
3,217
12,684
VK
54,028
213,644
AVAZU
1,297
54,364
OGB-V
169,343
1,166,243
40
OGB-PRODUCTS
2,419,029
61,859, 140
47
Table 8: Hyperparameters
DATASET
INPUT FOR STACKING LAYER
HOUSe/CoUNTY/VKK AVAZU
(X, (F()
FO)
F
F(3)
F(4)
F
OGB-ARXIV
0.95
(X, (
(0)
F.ll
(33
FlS)
F!?
F
OGB-PRODUCTS
0.97
(x, (F
(0) F
F
F
(9)
Table 9: Hyperparameters for C&S
DATASET
KERNEL TYPE
A
KERNEL TYPE
NUM PROPAGATION
HOUSE/COUNTY /CO/////UUUAAAA
0.8
DA
0.5
DA
5
VK
0.8
DA
5
OGI-ARXIV
0.9
DA
0.1
AD
50
DGB-PRODUCTS
0.3
DAD
0.3
AD
50
2
3
4
Test. Acc. 0.710 + 0.001
0.708 + 0.002
0.712 + 0.001
Table 10: Ablation study for
B.2 Parameters for Graph-aware propagation
We do graph-aware propagation for the prediction to incorporate the graph structure. Table shows two hyperparameters considered in the propagation part: weight À and number of propagation step T. We also present the hyperparameters for Correct and Smooth in Table g
C Additional Ablations
The following tables show the performance with respect to k and L (refer to Table 10 and 11 (Experiments OGB-Arxiy with numerical embedding). From the following table, when L > 2, the outcome is not significantly affected by the value of L (owing to the incorporation of the graph signal from the second layer). Therefore, we lypically opt for L = 2. Typically, the Repeat bagging (N) is set to 1, which is the default value in the AutoGluon package. While a higher N might enhance performance, it also significantly raises computational cost. Notice here, for both L and N, the main paper, we present more generalizod framework for our approach, which explains why we employ a for loop. The base models are determined by the input data type (e-g, Bert for text node feature and Tree-base model for tabular data). In our study, the
20

---

Published in Transactions on Machine Learning Research (2/2024)
2
Test Acc.
0.550 + 0.01
0.712 + 0.02
0.711
0.03
0.707 + 0.01
Table 11: Ablation study for
1.0
0.9
0.8
0.7
Test Acc.
0.707 0.01
0.706 + 0.02
0.697
0.02
0.694 + 0.02
Table Ablation study
base models were designed in such way that they specific single modality, do not do ablation study on them. Table T2 shows the performance is not sensitive to À (Experiments on OGB-Arxiv with numerical embedding)..
21


Reviews:
[{"review_id": "QCsdRGZXHF", "reviewer": "Reviewer_QmtH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Innovative Integration: Combines graph-aware propagation with models designed for IID data, addressing GNNs' limitations with non-numerical data.\n* Mitigation of Label Leakage: Introduces techniques to effectively handle label leakage, a common issue in layer-wise training strategies.\n* Generalization Across Datasets: Demonstrates strong performance across a variety of graph datasets, showcasing the method's adaptability.\n* Theoretical Analysis: Provides a thorough theoretical analysis of the proposed methods, enhancing the paper's credibility.\n* Empirical Validation: Empirical results support the framework's effectiveness, with comparisons to state-of-the-art methods.\n\nWeaknesses:\n* Lack of Statistics of Datasets: The paper does not present the detailed statistics of the datasets used in the paper. Providing these dataset statistics, such as the number of nodes, edges, features, and classes in graph datasets, is crucial for readers to fully understand the context and scale at which the proposed methods are tested. This information is essential for replicating the study, evaluating the model's performance in various scenarios, and comparing it with other methods. Without these statistics, it's challenging to asses the robustness and adaptability of the proposed framework to different types of graph data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes BestowGNN, a novel framework that integrates graph-aware propagation with ensembles of arbitrary models intended for independent and identically distributed (IID) data. This approach addresses limitations of Graph Neural Networks (GNNs) for dealing with non-numerical node features like text or tabular data. By leveraging bagging and stacking strategies, BestowGNN effectively mitigates label leakage and overfitting, achieving strong generalization across various graph datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UC72lQ4dz3", "reviewer": "Reviewer_QGQx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths And Weaknesses\nStrengths:\n\nInnovative and Universal System Design: The paper introduces a framework capable of leveraging diverse collections of models for heterogeneous graph datasets with various feature types such as numerical, categorical, and text. This versatility is a significant advancement in the field.\n\nTheoretical Insights: The work makes a substantial theoretical contribution by exploring the label leakage issue in label-augmented Graph Neural Networks (GNNs), offering new perspectives and understanding.\n\nRobust Experimental Performance: The paper demonstrates strong performance through a series of comprehensive experiments, showcasing the practical effectiveness of the proposed system on graphs with numerical, categorical, and text features.\n\nWeaknesses:\n\nIncreased Complexity: The application of both bagging and stacking techniques introduces additional complexity, which could potentially hinder the practical deployment and usability of the system.\n\nManuscript Quality: The current state of the manuscript necessitates improvements. Notably, there are missing references, as indicated by \"?\" marks on page 9. Additionally, the heavy use of notation throughout the paper detracts from its overall readability, suggesting a need for clearer exposition and possibly simplification.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary Of Contributions\nThe paper designs an innovative framework , i.e. BestowGNN, that is capable of leverages bagging and stacking strategies to GNNs. It theoretically reduces label leakage problem and shows strong performance on multiple types of graphs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested Changes\nSee W2", "broader_impact_concerns": "Broader Impact Concerns\nNone", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "m0ZhzqxWo3", "reviewer": "Reviewer_nFRC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper's proposal is clearly explained, well-motivated, and timely. The experiments are, in principle, well-designed, and the model appears to perform competitively against relevant baselines.\n\nHowever, while the paper certainly has merit, and could be a useful TMLR publication, there are also several issues in the work's presentation and the depth of its experimentation, which could be misleading to uninitiated readers. Therefore, several changes would need to be implemented before I would be comfortable recommending acceptance. Such issues include:\n\n* Lack of appropriate positioning of BestowGNN with respect to prior art. This is all the more important, given that the method relies on a complex combination of existing ideas.\n* Unclear and somewhat 'cavalier' handling of the oversmoothing problem in the paper.\n* I have some doubts about possible cherry-picking of the baselines, as well as the statistical significance of certain claims the paper makes.\n* Lack of depth in the ablation studies.\n\nPlease see the \"Requested Changes\" cell for a summary of my suggestions to the authors.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces BestowGNN, a generic unified method of combining standard, off-the-shelf, point-wise classifiers with graph propagation, to enhance the predictions on graph data. The idea of enabling the combination of such ideas is clearly motivated by prior successes of methodologies that compose pointwise classifiers with graph smoothing operators, such as BGNN and C&S. The framework that the authors propose is as follows:\n\n* Repeatedly resample the training dataset into held-out splits using bagging.\n* Train all relevant base models on each split within each bagging run.\n* Combine the predictions of each base model on their held-out nodes, as well as the overall validation / testing nodes for the task.\n* Perform a few steps of graph propagation (using energy minimisation) to smooth out these predictions.\n* Re-run this procedure K times, at each step using the concatenated outputs of graph propagation at various scales as input features.\n\nTheoretical motivation for these design choices is provided, and favourable performance is demonstrated on relevant datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. BestowGNN is certainly not the first method to propose combining pointwise classifiers with graph operators. This is implied by the authors as well, by making references to works like BGNN and C&S. However, all of these references are made in passing, and certain variants of these models are then compared to in the experiments. The paper would strongly benefit from a detailed subsection elaborating the similarities and differences that BestowGNN has to both of these models, and ideally a discussion of possible tradeoffs between them.\n\n2. The paper seems to defend its choice of energy minimisation-based propagation by invoking the 'oversmoothing' problem. But, at least from my scan of the paper, I saw no clear indication (or proof) that the authors' approach makes oversmoothing less of an issue. This claim should be either toned down, or more explicitly and rigorously backed up by theoretical evidence. Further, in the very least, I would expect to see the following two empirical comparisons to solidify this claim:\n    * To confirm the lack of suitability of message-passing style propagation, it would be useful to provide a comparison against a variant of BestowGNN which uses a model like SGC (a nonparametric GNN) to propagate information. \n    * I have doubts that the tasks the authors considered would be appropriate to show clear impact of the oversmoothing effect in the first place, as at least some of these datasets are known to be homophilic. It would be highly beneficial to evaluate BestowGNN on heterophilic benchmarks---perhaps from the recent work of Platonov et al. (ICLR'23): https://openreview.net/forum?id=tJbbQfw-5wv.\n\n3. From the experiments presented, especially on the OGB benchmarks, I cannot draw clear conclusions about how does BestowGNN compare against the state-of-the-art. A simple look at the OGB leaderboards (https://ogb.stanford.edu/docs/leader_nodeprop/) shows that there are variants of GIANT that would outperform all numbers in Table 3. Further, in both Table 1 and 3, the authors often bold-face only the BestowGNN result, even though there are several other baselines that are clearly with highly overlapping error bars to BestowGNN, casting doubt at the statistical significance of its outperformance. I am not saying that the paper needs to set state-of-the-art to be publishable in TMLR---the argument the authors make about the uniformness of the method is certainly on point!---but the paper also needs to make a clear message about when the method is clearly outperforming its baselines, and when there exist better-performing methods. The correctness and clarity of this information would be invaluable to a GNN practitioner hoping to potentially make advantage of this method in their work.\n\n4. Last, but certainly not least, BestowGNN is a complex system consisting of several mechanisms stacked on top (each, roughly, exemplified by a for loop in Algorithm 1). Naturally, it is important to empirically defend the inclusion of these systems through careful ablation studies. Table 4 does this for the number of propagation steps, and the inclusion of bagging---both are highly appreciated. But there are several other axes as well: stacking ($L$), repeated bagging ($N$) and multiple models ($|\\mathcal{M}|$). It would be very useful if similar tables to Table 4 could be provided, showing the impact of various amounts of repeated bagging, various numbers of stacked layers, and the specific combinations of models used. As currently presented, unless I am missing something, I do not see any convincing evidence that either of these for-loops are necessary---and it would be great to have that evidence: both if you're a practitioner and if you are a researcher extending the method.\n\nI hope my suggestions are useful to the authors. I am willing to champion the paper if they are appropriately addressed.", "broader_impact_concerns": "No strong concerns to report; however, I did feel like the paper could have taken more rigour in describing the computational footprint of BestowGNN, especially considering the nested-ness of its forward pass. In the very least, I would have hoped to see a rigorous time complexity analysis, and supplementing Table 5 with representative baseline model (e.g. off-the-shelf GNN) timings, to empirically compare any slowdowns or speedups induced.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jMo1RFeJ75", "reviewer": "Reviewer_gZTB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* The ensemble methods, including stacking and bagging, are rarely studied with graph-structured data.\n* The idea of capturing the graph structures of raw text and tabular node features with graph-aware propagation is reasonable.\n* The authors provide a meaningful theoretical analysis that the bagging strategy in the proposed BestowGNN can alleviate the label leakage (i.e., over-fitting) issue by connecting it to the Renyi Differential Privacy.\n* The proposed BestowGNN outperforms relevant baselines on challenging datasets, such as OGB-arXiv and OGB-Products.\n\n### Weaknesses\n* The novelty of this paper is moderate. In terms of the stacking and bagging techniques, the authors directly use the existing methods. While the authors propose to consider the graph structures with graph-aware propagation during ensemble, this is a general way other studies also use when they deal with graph-structured data.\n* In Table 1, when considering the standard deviations, the performance improvement of the proposed BestowGNN is marginal against the baseline (AutoGluon + C&S), and the possible reason for this is not clearly discussed. In addition to this, it would be meaningful to discuss the difference between the AutoGluon + C&S and the proposed BestowGNN in terms of the methodology, since AutoGluon + C&S is the ensemble baseline that can also be aware of the graph structures similar to the proposed method.\n* In Table 3, the performance of the BestowGNN is far from the state-of-the-art results in the OGB leaderboards; therefore, it would be meaningful to discuss why the gaps between the proposed method and the state-of-the-art methods are significant and how we can bridge those gaps.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "To deal with heterogeneous node features (e.g., table and text) over graph-structured data, the main idea of this paper is to use bagging and stacking strategies with multiple models that are designed for such heterogeneous data. Also, to capture the graph structure (i.e., connections between nodes), the authors propose graph-aware feature propagation, which allows the model to consider the features of neighboring nodes of each node based on the energy function for graph-structured data. The authors evaluate the proposed BestowGNN on multiple benchmark datasets for node classifications and regressions with tabular, text, and numerical node features, demonstrating that the proposed BestowGNN outperforms other relevant ensemble baselines that usually cannot be aware of the graph structures.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "For major requested changes, please see the points in Weaknesses above. \n\nAdditionally,\n* When discussing the label leakage in Section 2.1, it may be beneficial to more clearly describe what is this issue and why it happens.\n* I am wondering why the authors do not use the same baselines across different datasets. For example, the authors compare the powerful baseline, namely AutoGluon + C&S, in Table 1; meanwhile, this baseline is not compared in Table 2.\n* In Table 5, when reporting the time efficiency of the proposed model, it is more beneficial to include the other baseline models (e.g., the model that does not use the ensemble technique), for understanding relative efficiencies in comparisons to others. \n* The paragraph (Base models) in Section 6 may appear earlier in the section. In particular, this paragraph explains the experimental setup of the proposed BestowGNN (e.g., which base models that they use for the ensemble), which may appear before showing the results.", "broader_impact_concerns": "While the authors do not clearly discuss the broader impact, I don't have any concerns on it.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper aims to expand joint training of Graph Neural Networks (GNN) with complex feature extraction models, which maynot be neural networks, such as decision trees, and not amenable to training by direct backpropogation. The paper presents BestowGNN, a framework that integrates graph-aware propagation with ensembles of models designed for independent and identically distributed (IID) data. In particular, the paper considers node features like numerical, text, and tabular data. In the initial draft the presentation and writing of the paper leaves more to be desired. We thank the authors and reviewers to actively engage in discussion for making the paper better. During discussion many of the reviewer concerns/questions were resolved like missing baselines and statistical significance of the result. There were some concerns left: 1) about oversmoothing claims, which can be made more rigorous more rigorous by more extensive experiments, 2) ablation for need of multiple heterogenous base models $\mathcal{M}$ vs using a transformer across modalities; 3) novelty, but it is not a criteria for TMLR.  Nevertheless, the proposed method and technique are correct, experimental result demonstrate competitive performance across various datasets using a single unified architecture with only the base model for processing input features varying for each dataset, and thus the approach will be of interest to the community, hence I propose to accept the paper with some minor modifications.

Please add all the reviewer suggestions in the final draft along with following two points:
- Revise the claim that C&S cannot use a graph-based base model or only in principle. The original C&S paper seems to do this.
- Consistency of bold-facing the top performing methods. Please bold-face all methods which have a statistical overlap as it is not feasible to decide which one is better given current statistics.
Decision: Accept (poster)