High dimensional, tabular deep learning with an auxiliary knowledge graph
Authors: Camilo Ruiz
Hongyu Ren
Kexin Huang
Jure Leskovec
Venue: NeurIPS 2023 poster (2023)
Keywords: Tabular Data
Deep Learning
Knowledge Graph
Regularization

Abstract: Machine learning models exhibit strong performance on datasets with abundant labeled samples. However, for tabular datasets with extremely high $d$-dimensional features but limited $n$ samples (i.e. $d \gg n$), machine learning models struggle to achieve strong performance due to the risk of overfitting. Here, our key insight is that there is often abundant, auxiliary domain information describing input features which can be structured as a heterogeneous knowledge graph (KG). We propose PLATO, a method that achieves strong performance on tabular data with $d \gg n$ by using an auxiliary KG describing input features to regularize a multilayer perceptron (MLP). In PLATO, each input feature corresponds to a node in the auxiliary KG. In the MLP’s first layer, each input feature also corresponds to a weight vector. PLATO is based on the inductive bias that two input features corresponding to similar nodes in the auxiliary KG should have similar weight vectors in the MLP's first layer. PLATO captures this inductive bias by inferring the weight vector for each input feature from its corresponding node in the KG via a trainable message-passing function. Across 6 $d \gg n$ datasets, PLATO outperforms 13 state-of-the-art baselines by up to 10.19%.

High dimensional, tabular deep learning with an auxiliary knowledge graph
Camilo Ruiz!
Hongyu Ren Kexin Huang', Jure Leskovec'
DDparrment of Computer Science, Stanford University
Department of Bioengineering, Stanford University "Equal contribution (caruiz, hyren, kexinh, jure) wtw.essstafrrdd..oo
Abstract
Machine learning models exhibit strong performance on datasets with abundant labeled samples. However, for tabular datasets with extremely high d-dimensional features but limited n. samples (i.e. d >> n1), machine learning models struggle to achieve strong performance due to the risk of overfitting. Here, our key insight is that there is often abundant, auxiliary domain information describing input features which can be structured as a hetcrogeneous knowledge graph (KG). We propose PLATO, method that achieves strong performance tabular data with d > by using an auxiliary KG describing input features to regularize a multilayer perceptron (MLP). In PLATO, each input feature corresponds to node in the auxiliary KG. In the MLP's first layer, each input feature also corresponds to weight vector. PLATO is based on the inductive bias that two input features corresponding to similar nodes in the auxiliary KG should have similar weight vectors in the MLP's first layer. PLATO captures this inductive bias by inferring the weight vector for each input feature from its corresponding node in the KG via a trainable pessage-passing function. Across 6 n datasets, PLATO outperforms 13 state-of-the-an baselines by up to 10.19%.
Introduction
Machine learning models have reached state-of-the-arr performance in domains with abundant labeled
data like computer vision [76] [10 and natural language processing 70 12 [2]] However, for tabular datasets in which the number d of features vastly exceeds the number of samples, machine learning models struggle to achieve strong performance (24] 41). Crucially, many tabular datasets from scientific domains [21] 30 79 16 33 have dimeeninnsii features but limited labeled samples due to the high time and labor costs of experiments. For these and other tabular datasets with d > n, the performance of machine learning models is currently limited.
The key challenge for machine learning models when > n is the risk of overfitting. Indeed, deep models can have large number of trainable weights, yet training is limited by the comparatively small number of labeled samples. As result, tabular deep learning approaches so far have focused on data-rich regimes with far more samples than features (n >> d) [19 [8 59). In the low-data regime with far more features than samples (d > n)), the dominant approaches for single tabular datasets still statistical methods (24). These statistical methods reduce the dimensionality of the input
space 1 41 66 67
select features [64] 14 46), impose regularization penalties on parameter
magnitudes
45
ensembles of weak reseb models [5]
42 5T].
Here, we present a novel problem setting and framework that enables tabular deep learning when d > (Figure Our key insight is that there is often abundant, auxiliary domain information
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a)
X E Rex
(c)
Semple i
A
A3
A7
:30
X
Featura
Qj R
Knowledge Graph G
(b)
(d)
B
layer of
@
E R^
X
X
F
Feature i
%
Figure I: PLATO is a method that uses auxiliary domain information describing input features to regularize multilayer perceptron (MLP) and achieve strong performance on tabular data with d > n. (a) In PLATO, each input feature j corresponds to a node in an auxiliary KG of domain information. (b) In the first layer of MLP with h hidden units, each input feature j corresponds to vector of weights ( E TR such that the weight vectors of all d features compose the weight matrix 1 E Rd similar nodes the auxiliary KG, they should have similar weight vectors O and θ in the MLP. (c,d) PLATO captures this inductive bias by inferring the weight vector for each input feature from its corresponding node in the KG. A trainable prsssage-passing function Q creates a wwwdiimnnional embedding Q3 Rs for each input feature j. neural network B that shared across all input features then infers the weight vector e/ corresponding to input feature j from Qÿ. Input features with similar embeddings produce similar weight vectors, regularizing the MI..
PLATO is based on the inductive bias that, if two input features j and k correspond to
describing input features which can be structured heterogeneous knowledge graph (KG). We propose novel problem setting in which each input feature of a tabular dataset corresponds to node in auxiliary KG [p). To represent diverse domain information describing the input features, the KG contains feature and non-feature nodes as well as multiple node and edge types. For example, consider tabular medical dataset which each row is cancer patient, each column is gene, and each value is the amount of gene in the patient's 's tumor. For this tabular dataset, there exists an auxiliary KG with each gene (i.e. input feature) other gene nodes (i.e. other feature nodes) with diverse edge types like "activates" or "inhibits." Each gene node also has edges to other nodes (i.e. non-feature nodes) representing the gene's function in the body like "heart rate". Finally, the function nodes (i.e. non-feature nodes) have edges to each other representing their anatomical relationships like "heart rate" "part of "cardiac system", Note that the KG does not capture the relationships between input data samples but instead captures the relationships between input features and other domain information.
node. Each gene node has edges to
Within our novel problem setting, we propose PI.ATO, a method that enables deep learning for tabular data with > by using an auxiliary KG describing input features (Figure IT. PLATO achieves strong performance by using the auxiliary KG to regularize a multilayer perceptron (MLP). In PLATO, each input feature corresponds to node in the auxiliary KG (Figure [pp) In the first layer of the MLP, each input feature also corresponds to a weight vector such that the weight vectors of all features collectively compose the weight matrix (Figure [p). PLATO is based on the inductive bias that two input features which correspond to similar nodes in the KG should have similar weight vectors in the first layer of MLP. PLATO captures this inductive bias by inferring the weight vector for feature from its corresponding node in the auxiliary KG with trainable message-passinng passing function (Figure [e.d). Inferring the weights in the MI.P's first layer also leads to a drastic reduction in the number of trainable weights, since most weights in MLP are usually in the first layer when d >

---

We exhibit PLATO's performance on 6 d > n tabular datasets with 13 state-of- sfate-of-the-as art baselines spanning dimensionality reduction, feature selection, statistical models, graph regularization, weight- inference, and tabular deep learning. Following, rigorous evaluation protocol from the tabular deep learning literature [19] 18.. PI.TTO outperforms the prior state-of-the-arr on all 6 datasets by up to 10. 10%.. Ablation studies demonstrate the importance of PLATO'S s trainable message-passing the importance of non-feature nodes in the KG, and PLATO's robustness to missing edges in the KG. Ultimately, PLATO enables deep learning for tabular data with d >>> by using auxiliary KG describing the input features.
2 Related Work
Tabular deep learning methods. In contrast to PLATO'S setting, tabular deep learning methods have primarily been developed for settings with far more samples than features (i.e. n. >> d). Indeed, recent tabular deep learning benchmarks ignore datasets with a large number of features and In the n > d setting, various categories of deep tabular models exist. First, decision tree models like NODE [50] make decision trees differentiable to enable gradient-based optimization 2 36 80). Second, multilayer perceptrons (MLPs) apply sequential, non-linear transformations input features [32 3T]. Third, tabular transformer architectures use an attention mechanism to select and learn interactions among features. Examples include TabNet [3) TabTransformer [29], FT-Trfsfformer [18, TabPFN (6),, SAINT [61), Non-Parrmetticc Transform- ers [37, and AutoInt (62). Finally, although PLATO focuses on single tabular datasets, transfer learning architectures can learn across multiple tabular datasets (39 84 73 Ultimately, we compare
small number of samples [19 18 59
PLATO to several benchmarked, state-of-the-art models for single, tabular datasets 19 18 59
d >>> n methods. For PLATO's setting in which d > n. various tabular machine learning approaches exist (24] First, dimensionality reduction techniques like PCA [I] aim to reduce the dimensionality
of the input data while preserving as much of the the variance in the data as possible [41 66, 67].
Second, feature selection approaches select a parsimonious set of features, leading to a smaller feature space. Feature selection approaches include LASSO [44] and its variants [8 14 46 For
feature selection with deep learning, Stochastic Gates 77 are among the best performing of many variants 4 43 Finally, tree-based models like XGBoost learn ensembles of weak decision trees models to make an overall prediction
34 51
Weight inference. Using one network infer the weights of another has been studied extensively [I] 58 []. For example, (22) infers the weights in all layers of a sequential model (i.e. RNN, LSTM) by using information about the weights' structure. Diet Networks [54) infer weights by hand-crafting prior information about the input features or using random projections. By contrast, PL.ATO infers the weights in a MIP from prior information describing the input features in an auxiliary KG. PLATO'S weight inference uniquely captures the inductive bias that two input features corresponding to similar nodes in KG should have similar corresponding weight vectors in the first layer of a MLP (Figure D
Graph regularization. Graph regularization approaches regularize the weights of linear model based on a simple graph between input features. The graph is typically constructed from the tabular data based on covariance relationships. Approaches then add regularization penalty to the loss function which forces the weights of the linear model to vary smoothly over the corresponding feature nodes in the graph. State-of-the-an methods include GraphNet 20 and Network-Constraine LASSO [4) which are based on I Laplacian regularization [60 2 as well as Network LASSO 23 which generalizes the Group LASSO [3 to network setting. PLATO differs from graph regulariza- tion approaches in two key ways. First, PLATO'S KG includes both feature and non-feature nodes and multiple edge types, thereby modeling diverse, prior domain information that is missing in graph regularization approaches. Second, PLATO infers the weights of deep non-linear model (i.e. MLP) rather than adding a regularization penalty to a loss, representing a distinct regularization mechanism.
Knowledge graph methods. Existing KG approaches are designed for tasks directly on the graph like link prediction or node classification 77
78 I3 By contrast, PI.ATO does not make any predictions on the KG. Instead, PIO makes predictions on a separate, tabular dataset by using the KG as a prior. Graph classification methods also do not apply (Appendix B).
65
72
3

---

3 PLATO
PLATO is machine learning method for tabular datasets with d > n and an auxiliary knowledge graph (KG) with input features as nodes (Section 3.T). PLATO's key insight is that there often exists abundant domain information describing input features which can be structured as an auxiliary KG G (FigureTi)) PLATO uses the auxiliary KG to regularize a multilayer perceptron (MIP) and achieve strong performance on tabular data when d > n.
3.1 Problem setting
Consider a tabular dataset X E Rexd with labels y E IR" and far more features than n samples such that d >> The goal is train a model F predict labels y from the input X. PLATO assumes the existence of an auxiliary knowledge graph G = (V, E) with [V] nodes and |E| edges such that each input feature corresponds to a node in G. Formally, Vjj E (I,..., d). 3v V s.t. + v, as shown in Figure G also contains additional nodes which represent broader knowledge describing the domain. The edges in G are (head node, relation type. tail node) triplets.
3.2 PLATO's inductive bias
In PLATO, each input feature j corresponds to node in the auxiliary KG (Figure Db. In the first layer of a MLP with hidden units, each input feature j also corresponds to weight vector e. !!] R such that the weight vectors of all features collectively compose the weight matrix ! E Rd (p). PLATO is based on the inductive bias that two input features j and which correspond to similar nodes in the KG should have similar weight vectors O and @,² in the first layer of the MLP. PLATO captures this inductive bias by inferring the weight vector for feature from its corresponding node in the auxiliary KG with trainable message message-passing function (Figure .d).
3.3 PLATO overview
PLATO has four key steps. First, PLATO uses a self-supervised objective on the auxiliary KG to pretrain an embedding for each input feature (Section 3.4 Second, PLATO updates each feature embedding with a trainable message- message-passing function that is trained on the supervised loss objective for the tabular data (Section 3.5 Tp) Third, PLATO infers the weights in the first layer of the MLP from the feature embeddings with a small neural network that is shared across input features (Section 3.6 Figure [t). Finally, the MLP predicts the label for the input sample.
3.4 Pretraining feature embeddings with self-supervision on the knowledge graph
First, PLATO learns general prior information about each input feature j from the auxiliary KG G. PLATO represents the general prior information about each input feature as a low-dimensional embedding M, E Re Since each input feature j corresponds to node in G. PI.ATO can learn M, by learning an embedding for the corresponding feature node in G. Any self-supervised node embedding method on G' can be used within PLATO's 's framework.
Formal notation. Formally, PLATO uses self-supervision on G to pretrain an embedding for each input feature according
M = H(G)..
(1)
M E dd is the matrix of all feature embeddings. H is self-supervised node embedding method. We refer Eq TT) as pretraining since only the auxiliary KG G' is used but the tabular data X, y is ignored. After pretraining, the feature embeddings M are fixed.
For H. we choose ComplEx as it is prominent and highly scalable KG node embedding method 655 ComplEx uses a self-supervised objective which learns an embedding for each node in by classi- fying whether proposed edge exists in G. ComplEX''s proposed edges include both feature nodes and other nodes in G. thereby integrating prior information about the input features and the broader domain. We also test KG embedding methods DistMult (78 and TransE [72) in Appendix C

---

3.5 Updating feature embeddings with a message-passing function trained on tabular data
PLATO next updates each feature embedding with trainable message message-passing function that is trained on the supervised loss for the tabular data (Figure[p) During message-passinng PLATO updates the embedding of each input feature to be a weighted aggregation of it's neighbors" embeddings.
Formal notation. Formally, PLATO uses message-passinng function Q on the KG to update each pre-trained feature embedding Mj E IR to feature embedding Q2 E R according to
= Q(M, G, II)
As input, the pressage-passingg function considers the pre-trained feature embeddings M, the knowl- edge graph G, and the sample value X, Q uses an attention mechanism which considers the sample value X, The only trainable weights in Q are in the attention mechanism and are II.
The message passing network Q. Let Q," be the embedding of input feature after round E (1.... RJ of message passing. For each input feature j. Q first initializes the updated feature embedding to the pretrained feature embedding.
0 MM,-
(2a)
Q then conducts R rounds of message passing. In each round of message passing, the feature embedding 0f/1 is updated from the feature embedding of each neighbor k in the prior round Q !r1 and its own feature embedding in the prior round |r-1] The "message" being passed is the embedding of each feature from the prior round.
Weighted messages from neighbors
q. o B KEN
Oiyh Q -l
+ (1 3) 1-11
(2b)
Weighted message self
a is an optional nonlinearity. Nj are the neighbors of feature node j in G.
During message-passing, Q uses two scalar values B E R and Oijk E R to control the weights of to control the weight of the messages aggregated from the feature node's neighbors vs. from the feature node itself. Second, Q calculates an attention coefficient azjz E R to allow distinct nodes the same neighborhood to have distinct weights. The coefficient Oijk specifies the weight of the message between feature and neighbor k for sample i.
messages. First, Q uses hyperparameter
After R rounds of message-passing, the updated feature embeddings Q, are set.
Q = QQ
(2c)
The attention coefficient. PLATO'S attention coefficient aifa is inspired by 668 in which node attributes are used to calculate the weight of a message between neighboring nodes. For a sample i in PLATO, the node attributes for features j and ic are their sample values X,, E R and X E R. PLATO thus uses the sample values X;, and X, to calculate the attention coefficient. The attention coefficient eijk indicates the importance of node to node k for sample i.
(2d) A is shallow neural network parameterized by II that is shared across samples and features. The number of trainable weights in II is small since the input of A is RR² and the output of A is a scalar TR.
Cifk A(X X II).
To make the attention coefficients comparable across different nodes, PLATO normalizes the attention coefficients with softmax function across the neighbors N, of node
exp (E(rx)
Oijk- = softMaxx (Eijk) EIEN, exp (eijt)
(2e)
5

---

Algorithm 1:
The PLATO Algorithm.
Input: A data sample X; E IR", knowledge graph G containing each input feature X as node, matrix of input feature embeddings M E _d pre-trained over G.
Output: A predicted label y; E IR.
Use a trainable message-
function Q to update the pre-trained feature embeddings: Q Q(M, G, x,; IT), Qj RR, Q Rdxe
2 Infer the weight vector in the first layer of a MLP that corresponds to an input feature j with a neural network B:
BQq x); @!! R
3 Repeat to infer the weight vectors corresponding to all input features by sharing the neural network B:
[ E Rdxh
Concatenate the first layer inferred weights with the trainable weights in the rest of the MLP
layers:
④ = ( J (
s Predict the label with MLP F that is parameterized by é
y. ; xx))). y; E
Trainable weights: II, +, O
914)
3.6 Inferring the first layer of weights in F from the updated feature embeddings
Finally, PIATTO infers the weights in the first layer of MLP F from the updated feature embeddings (Figure In the first layer MIP with h hidden units, each input feature j corresponds to a weight vector θ y [1] Rh (Figure [p). The weight matrix in the first layer of the MLP, O R is simply the concatenation of d weight vectors, one corresponding to each input feature. For each input feature j. PLATO infers the weight vector !!!!! IRA from the feature embedding Q2 R by using a shallow neural network shared across input features. Input features with similar feature embeddings will produce similar weight vectors. Thus, PLATO captures the inductive bias that input features corresponding to similar nodes in the KG should have similar corresponding weight vectors in the MLP's first layer.
Formal notation. PLATO infers the weight vector associated with each input feature in the first layer of JF with
6 = BX; +)
a shallow neural network with trainable weights C. Q; is the updated feature embedding of which is conditioned on the specific input sample X, since the input sample is used as an input in its 2). 4 are the weights of B. B and its weights d are shared across
calculation (Section 3.5 Equation
each feature (I....,d).
PLATO drastically reduces the number of trainable weights compared to standard MLP. The sharing B and & across all input features drastically reduces the number of trainable weights compared to standard MLP. For high-dimensional tabular dataset (i.e. > ni), standard MIPP T with h hidden units has large number of trainable weights in the first layer since O E dx A standard MLP T must learn all dh of these trainable weights by backpropagation. By contrast, B uses a shared set of trainable weights & to infer θ, from Q, for every j E [1...., d}. The number of trainable weights in & is small compared to dh since I need only transform every Q E R to @ R Thus, ( ch (assuming B is a single layer neural network). c, the dimensionality of the feature embedding, is much less than d the number of input features. As a result, | ch & dh and PLATO drastically reduces the number trainable weights the first layer of a MLP.
3.7 The PLATO algorithm
PLATO is outlined in Algorithm

---

Table 1: PLATO outperforms statistical and deep baselines when d > n. For every dataset, the best overall model is in bold and the second best model is underlined.
Dataset
MNSCLC
CM
PDAC
BRCA
CRC
CH
II of features
15,390
13,183
12,932
12,693
18,206
19,902
# of samples
295
286
321
476
562
924
d/n
52.2
46.1
40.3
28.2
22.6
19.7
Classic Stat MI.
Ridge
0. 053
0.3902000
0.34420000
0.538+00.0
0.37620000
0.546
Dim. Reduct.
PCA
0.156
0.070=0.0m
0.232+0.211
0.452+0.000
0.
0.237
Feat. Select.
LASSO
1,0000
40,000
0.346
0.470
0. 0.000
0.547
STG
0.132+0.100
0.366
0.258+0.05
0.485
0.301 #0.100
0.262
Decision Tree
XGBoost
-0.02.0.000
0.205
0.363
0.347
0.354
0.728
GraphNet
0. +0.0000000
0.277
0.249
0.350
0. 0.125:00061
0.6-16+nu55
Graph Reg.
NC LASSO
0.2 0.210
0.339=00044
0.007
0.458+0.88
0,220=0000
0.41.5a008
Network LASSO
0.21 220006
0.243
0. 06
0.348
0.171
0.212
Param. Infer.
Dict
-0.04:00.005
0.054
00009
0.213
0.087=0.112
0.14880000
MLP
0.128
0.322
0.289
0.240
0.305
0.044
NODE
0.003+0.000
0.150=000m
0.190=00000
0.512+00000
0.344=0..00
0.181
Tabular DL
TahTransformer
0,265+0.000
0.072#0000
0.029
0,202#0.00
0.23820000
0.020
TabNet
0.085
0.010=0066
0.08820047
0.055
0.01 020000
0.039
Ours
PLATO
0.272+0.130
0.435=a0.22
0.400=na21
0.583+a0.19
0.401
0.770
Table 2: PLATO'S performance depends on updat- Table 3: PLATO's performance depends ing feature embeddings with a trainable message- both feature nodes in G and other nodes representing broader domain information.
passing (MP) function.
Feature Info.
Trainable MP
Feature Info.
Broader Info.
Weight Infer. Input
PearsonR
Auxiliary
PearsonR
Updated feat. embed. Q General feat. embed M None
0.583++0019
Full KG
V
0.583=001
0.522
Feature-only KG
x
0.539
0.240
No KG
X
x
0.240_0057
Experiments
We evaluate PLATO against 13 baselines on 10 tabular datasets (6 with d >> n2, 4 with d ~ nt).
Datasets. We use tabular >> n datasets, d n datasets 16
79,, and a KG from prior (Appendix G H). The KG contains 108,447 nodes, 3,066,156 edges, 99 relation types. All datasets include features which map to a subset of knowledge graph nodes. Code, data, and the KG are available at htttt:::/ thtbb com/ swap-stanttaaaeae frrtannoor/pltooo
studies 44
35
38
56
63
74 75
Baselines. We compare PLATO to 13 state-of-the art statistical and deep baselines. We consider regularization with Ridge Regression 45), dimensionality reduction with PCA (I] followed by linear regression, feature selection with LASSO (64), deep feature selection with Stochastic Gates (77). and gradient boosted decision trees with XGBoost (7]. We consider tabular deep learning with standard MIP, self-attention-baste based methods with Tab TabTransformer 22 and TabNet [3]. differentiable
decision trees with NODE [50). and weight inference with Diet Networks 54 We also attempted FT-
Transformer [8] but it experienced out of memory issues on all datasets due to the large number of features. Finally, we consider graph regularization methods which also have access to the knowledge
graph including GraphNet (20), NC LASSO 40), and Network LASSO 23 (Appendix D)
Fair Comparison of PLATO with Baselines. To ensure fair comparison with baselines, we follow evaluation protocols in recent tabular benchmarks [19 18] We conduct random search with 500 configurations of every model (including PLATO) on every dataset across a broad range of hyperparameters (Appendix A). We split data with a 60/20/20 training, validation, test split. All results are computed across 3 data splits and 3 runs of each model in each data split. We report the mean and standard deviation of the Pearson correlation (PearsonR) between y and ý across runs and splits the test set. Each model is run on GeForee RTX 2080 TI GPU.
7

---

Table 4:
PLATO's performance with incom- Table PLATO'S MLP layers 2,... L with trainable weights are useful for performance.
plete knowledge graph.
Fraction of edges in KG
PearsonR
Model
Description
PearsonR
100%
0.583 + 0.019
PLATO
MLP with first layer
0.583 0.019
90%
0.570 + 0.017
weights inferred
70%
0.537 + 0.044
PLATO-LR
Linear regression
0.550 = 0.020
50%
0.412 + 0.011
with weights inferred
Table 6: PLATO's performance is competitive with baselines when d ~ For every dataset, the best overall model is in bold and the second best model is underlined.
Dataset
ME
BC
SCLC
NSCLC
# of features d
19,902
18,261
18,437
18,308
of samples n d/n
10,064
10,101
10,712
16,730
2.0
1.8
1.7
1.1
Classic Stat ML
Ridge
0.566-0008
0.483=0008
0.604+0u57
0.679
Dim. Reduct.
PCA
0.239_0310
0.233
0.284
0.645:20.000
LASSO
0.667=0.00
0.633=000m
0.669
0.637++0000
Feat. Select.
STG
0.676__0.000
0.643_.0.00
0.668
0. 0.646.00.00
Decision Tree
XGBoost
0.875=0.000
0.826=0.m
0.878+0000
0.843
GraphNet
0.675_00047
0.723 026
0.742.10039
0.627:10.022
Graph Reg.
LASSO
0.733__0.06
0.730_00.22
0.79310.009
0. 0.746.0.023
Network LASSO
0.401=0.084
0.451
0.417
0.465=#0.034
Param. Infer.
Diet
0.105=00000
0.037_00000
-0.050
0. 0.000000.000
MLP
0.487=0.331
0.508
0.537
0.573++0.05
NODE
0.870..00000
0,420
0.801 100102
0.487.10.19
Tabular DL
TabTrrnsffrmes TabNet
0.305=0.02
0.010=00000
0. 0.288
0.503
0.667=00002
0.624_0.00 001
0.657
0.647:10.000
Ours
PLATO
0.875_0.004
0.844_ 0,000
0.88310.002
0.83910.000
4.1
Results
PLATO outperforms statistical and deep baselines when d >> n. PLATO outperforms all baselines across all datasets with > n (Table PLATO achieves the largest improvement on the PDAC dataset, improving by 19% vs. XGBooost, the best baseline for PDAC (0.400 vs. 0.363). While PI.TTO achieves the strongest performance across all 6 datasets, the best performing baseline varies across datasets. Ridge Regression is the strongest baseline for BRCA, LASSO for CM and CRC, XGBoost for PDAC and CH, and TabTransformer for MNSCLC. The remaining baselines are not the strongest baseline for any dataset. We also find that the performance of specific baseline depends largely on the dataset. Taranssformrr, for example, the best baseline for the MNSCLC dataset but the worst baseline for the CH dataset. The rank order of all models on all datasets is Appendix D
PLATO'S performance depends on updating feature embeddings with trainable message- passing function. PLATO infers the weights @ in the first layer of a MLP FF by using feature embeddings which contain prior information about the input features. PLATO first pretrains general feature embeddings M E dd PL.AOO then updates the feature embeddings to Q Rdxe with trainable message-passing function. We test whether updating the feature embeddings based on the trainable message-passing function is necessary by evaluating PLATO'S performance on the BRCA dataset in three configurations (Table 2). The default configuration uses the updated feature embeddings Q generated by the message- passing function to infer 6 according to 6 ((Q X:). The second configuration uses the general feature embeddings M instead of Q to infer O according to è ! = B(M)). The third configuration does not use feature embeddings and thus ablates to standard MLP. Using general feature embeddings M improves over not using feature embeddings at all (0.522 vs. 0.240). Using feature embeddings Q that are generated by the trainable message-passing function further improves performance (0.583 vs. 0.522). Thus, updating the feature embeddings to Q based on the trainable message- passigg function is key to PLATO'S performance.
8

---

PLATO's performance depends on both feature nodes and broader knowledge nodes in the auxiliary KG. PLATO relies on an auxiliary KG G which contains information describing input features and the broader domain. Information describing input features is represented as feature nodes while information describing the broader domain is represented as other nodes in G (Methods [.T]. To test the relative importance of the feature information in G' vs. the broader domain information, we measured the performance of PLATO on the BRCA dataset in two KG configurations: PLATO with the full KG (i.e. both the feature nodes and the broader domain nodes) and PLATO with a "feature-only KG (i.e. an induced subgraph on only the feature nodes) (Table 3) We also compare to a "No KG" configuration in which PLATO does not have access to the KG. Without auxiliary information describing the imput features or the broader domain, PLATO is ablated to a standard MLP.
We find that both the feature nodes and the broader knowledge nodes are important for PLATO'S performance. Using the "feature-only KG" configuration of PLATO improves performance vs the "no KG" configuration (0.539 vs 0.240). Using the "full KG" configuration further improves performance vs the "feature-only KG" configuration (0.583 vs 0.539). PI.ATO's performance thus relies on both the feature information and the broader domain information in the KG.
PLATO's performance with an incomplete knowledge graph. All KGs are incomplete since there undiscovered knowledge. PLATO thus uses -iw-dimnnional embeddings from KG embedding approaches [72, 78 65] which are designed to account for missing information, enabling predictive performance even with missing edges. We conduct an ablation study to assess PLATO's robustness to missing edges in the KG. We randomly remove edges from the KG and measure PLATO'S performance on the BRCA dataset. We observe that with only 50% of the KG's edges, PIATTO still has 71% % of the performance as PTATOO with 100% of the KG's s edges (0.412 vs. 0.583) (Table D.
The importance of MLP layers 2,..., L, the layers with trainable weights, for PLATO. PLATO a MLP in which the weights in the first layer are inferred from the knowledge graph (KG) but the weights in the remaining layers 2.... L are trained normally. We conduct an ablation study to determine whether MLP layers 2...., L are necessary for PLATO'S performance or whether the first layer of inferred weights are sufficient. Note that single layer of inferred weights in PLATO is equivalent to a linear regression in which the weights are inferred from the KG. We thus compare PLATO to PLATO-LR, a linear regression in which the weights are inferred from the KG (Table 5). PLATO 's standard configuration outperforms PLATO-LR on the BRCA dataset (0.583 vs. 0.550). Therefore, layers 2, ... L of the MLP important for PLATO'S 's performance.
.....
For datasets with d ~ n. PLATO is competitive with baselines. Finally, we test PLATO'S perfor- mance for datasets with d ~ n. We test 4 datasets with d ~ n ranging from _ = 1.1 to 2.0 (Table 6). We find that on 4 datasets with d n, PLATO is competitive with the best baseline, XGBoost, but does not improve performance substantially. PLATO'S 's stronger performance for datasets with >> n than for datasets with d ~ n is justified. PLATO'S 's key idea is to include auxiliary information describing the input features. Auxiliary information is likely to help performance the most in settings with the least labeled data (i.e. d > n). When d ~ n, auxiliary information less helpful since the tabular dataset may already have enough information to train strong predictive model. We further find that XGBoost is the strongest baseline for all datasets with d n, in contrast to XGBoost's varied performance on the datasets with d >> n (Table
5
Discussion
PLATO achieves strong performance on tabular data when d >>>>>> n by using an auxiliary KG describing input features to regularize a multilayer perceptron (MLP). Across 6 datasets, PLATO outperforms 13 state-of-the-art baselines by up 10.19%. Ablations demonstrate the importance of PLATO'S trainable message-passingg function, of including nodes in the KG that don't 't represent input features but instead represent domain information, and of the layers in the MLP whose weights are trained directly rather than inferred. We also test PLATO'S robustness to missing information in the KG. PLATO has several limitations. First, PLATO matches but does not improve the performance of baselines for high-dimensional datasets with more samples (i.e. d ~ n)). Second, PL.TTO depends on the existence of an auxiliary KG of domain information. Overall, PLATO enables tabular deep learning when d > n by using auxiliary KG of domain information describing input features.
9

---

Acknowledgements
We thank Maria Brbic, Michael Moor, Kaidi Cao, Weihua Hu, Rajas Bansal, Qian Huang, and Hamed Nilforoshan for discussions and for providing feedback on our manuscript. C.R. is supposed by Siebel Fellowship. National Science Foundation Graduate Research Fellowship under Grant No. DGE-1656518 8, and Stanford Enhancing Diversity in Graduate Education (EDGE) Fellowship. H.R. supposed by the Masason Foundation, an Apple Fellowship, and a Baidu Scholarship. We also grate- fully acknowledge the support of DARPA under Nos. HR001 12190039 (TAMI), N66001 1924033 (MCS); ARO under Nos. W911NF-16 16- 1-0342 (MURT), W91 INF--6- 1-0171 (DURIP); NSF under Nos. OAC- DAC-1835525 1835598 (CINES), OAC-193453 (HDR), CCF-1918944 (Expeditions), NIH under No. 3U54HG010426-D4S (HuBMAP), Stanford Data Science Initiative, Wu Tsai Neurosciences Institute, Amazon, Docomo, GSK, Hitachi, Intel, JPMorgan Chase, Juniper Networks, KDDI, NEC, and Toshiba. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding entities.
References
[1] Hervé Abdi and Lynne J Williams. Principal Component Analysis. Wiley Interdisciplinary Reviews: Computational Statistics, 2(4):433-455) 2010.
[2] Rie Ando and Tong Zhang. Learning graph with Laplacian regularization. In Advances in Neural Information Processing Systems, volume 19, 2006.
[3] Sercan Ö Arik and Tomas Pfister. TabNet: Attentive interpretable tabular learning. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6679-6687, 2021.
[4] Muhammed Fatih Balın, Abubakar Abid, and James Zou, Concrete autoencoders: Differentiable feature selection and reconstruction. In International Conference on Machine Learning, pages 444-453. Proceedings of Machine Learning Research, 2019.
[5] Anna Bauer-Mehren, Michael Rautschka, Ferran Sanz, and Laura Furlong. DisGeNET: cy- toscape plugin to visualize, integrate, search and analyze gene gene-disease networks. Bioinformatics, 26(22):2924 2926, 2010.
[6] Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning synaptic learning rule. In International Joint Conference on Neural Networks, volume 2, pages 969-975. IEEE, 1991.
[7] Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the SIGKDD Conference on Knowledge Discovery and Data Mining, pages 785-794, 2016.
[8] Héctor Bimente-Gonzáäle Chloé-Agathe Azencott, Samuel Kaski, and Makoto Yamada. Block HSIC lasso: model-free biomarker detection for ultra-high dimensional data. Bioinformatics, 35(14)i447-i445, 2019.
[9] Gene Ontology Consortium. The Gene Ontology resource: 20 years and still GOing strong. Nucleic Acids Research, 47(D13:330--D338 2018
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 245-255. IEEE, 2009.
[11] Misha Denil, Babak Shakibi, Laurent Dinh, Marc' Aurelio Ranzato, and Nando De Freitas. Predicting parameters in deep learning. In Advances in Neural Information Processing Systems, volume 26, pages 2148-2156. 2013.
[12] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT:
Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
[13] Claudia d' Amato, Nicola Flavio Quatraro, and Nicola Fanizzi. Injecting background knowledge into embedding models for predictive tasks on knowledge graphs. In European Semantic Web Conference, pages 441-457, Springer, 2021.
[14] Tobias Freidling, Benjamin Poignard, Héctor Climente-Gonzáäel and Makoto Yamada. Post- selection inference with HSIC-Lasso. International Conference on Machine Learning, pages 3439-3444 PMLR, 2021.
10

---

[15] Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of Statistics, pages 1189-1232, 2001.
[16] Hui Gao, Joshua M Korn, Stéphane Ferretti, John E Monahan, Youzhen Wang, Mallika Singh, Chao Zhang, Christian Schnell, Guizhi Yang, Yun Zhang, et al. High-throughput screening using pdtient-derived tumor xenografts to predict clinical trial drug response. Nature Medicine, (1(11:1118--1355 2015.
[17] Mathew J Garnett, Elena J Edelman, Sonja Heidom, Chris D Greenman, Anahita Dastur, King Wai Lau, Patricia Greninger, Richard Thompson, Xi Luo, Jorge Soares, et al. Systematic identification of genomic markers of drug sensitivity in cancer cells. Nature, 483(7391):570-577 2012.
[18] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In Advances Neural Information Processing Systems, volume 34, pages -8932-18943, 2021.
[19] Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? Advances Neural Information Processing Systems, Datasets and Benchmarks Track, 2022.
[20] Logan Grosenick, Brad Klingenberg, Kiefer Katovich, Brian Knutson, and Jonathan E Taylor. Interpretable whole-brain prediction analysis with GraphNet. Neurolmage, 72:304-321, 2013.
[21] Isabelle Guyon, Steve Gunn, Asa Ben-Hur, and Gideon Dror. Result analysis of the NIPS 2003 feature selection challenge. In Advances in Neural Information Processing Systems, volume 17, pages 545-552, 2004.
[22] David Ha, Andrew Dai, and Quoc V Le. Hypernetorrss. In International Conference on Learning Representations, 2016.
[23] David Hallac, Jure Leskovec, and Stephen Boyd. Network LASSO: Clustering and optimization in large graphs. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 387-396, 2015.
[24] Trevor Hastie, Robert Tibshirani, Jerome Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
[25] Hussein Hazimeh, Natalia Ponomareva, Petros Mol, Zhenyu Tan, and Rahul Mazumder. The tree ensemble layer: Diffcrentiability meets conditional computation. In International Conference on Machine Learning, pages 4138 4148. PMLR, 2020.
[26] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in second. In International Conference on Learning Representations, 2023.
[27] Weihua Hu, Matthias Fey, Marinka Zitnik Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open Graph Benchmark: Datasets for machine learning on graphs. In Advances in Neural Information Processing Systems, volume 33, pages 221 18 11-11183,,, 2020.
[28] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference Learning Representations, 2020.
[29] Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. TabTransformer: Tabular data modeling using contextual embeddings. In Advances in Neural Information Processing Systems, volume 35, pages 2902-2915, 2022.
[30] Francesco Iorio, Theo Knijnenburg, Daniel J Vis, Graham Bignell, Michael Menden, Michael Schubert, Nanne Aben, Emanuel Gonçalves, Syd Barthorpe, Howard Lightfoot, et al. A landscape of pharmacogenomic interactions in cancer. Cell, 166(3):740-754, 2016.
[31] Alan Jeffares, Tennison Liu, Jonathan Crabbé, Fergus Imrie, and Mihaela van der Schaar. TANGOS: Regularizing tabular neural networks through gradient orthogonalization and special- ization. In International Conference on Learning Representations, 2023.
[32] Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. Advances in Neural Information Processing Systems, volume 34, pages 23928-2333 23941, 2021.

---

[33] Gregor Kasieczka, Benjamin Nachman, David Shih, Oz Amram, Anders Andreassen, Kees Benkendorder, Blaz Bortolato, Gustaaf Broojimans, Florencia Canelli, Jack Collins, et al. The LHC. Olympics 2020: a community challenge for anomaly detection in high energy physics. Reports on Progress in Physics, 84:1 24201, 2021.
[34]
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems, volume 30, pages 3146-3154 2017.
[35] Sebastian Köhler, Nicole A Vasilevsky, Mark Engelstad, Erin Foster, Julie McMurry, Ségolène Aymé, Gareth Baynam, Susan M Bello, Cornelius F Boerkoel, Kym M Boycott, et al. The Human Phenotype Ontology in 2017. Nucleic Acids Research, 45(D1):D865-D87, November 2016.
[36] Peter Kontschieder, Madalina Fiterau, Antonio Criminisi, and Samuel Rota Bulo. Deep neural decision forests. In Proceedings of the IEEE International Conference on Computer Vision, pages 1467-1475, 2015.
[37] Jannik Kossen, Neil Band, Clare Lyle, Aidan N Gomez, Thomas Rainforth, and Yarin Gal. Self-attention between datapoints: Going beyond individual input-output pairs in deep learning. In Advances in Neural Information Processing Systems, volume 34, pages 28742-287555 2021.
[38] Michael Kuhn, Ivica Letunic, Lars Juhl Jensen, and Peer Bork. The SIDER database of drugs and side effects. Nucleic Acids Research, 44(D1):D0755-D107) October 2015.
[39] Roman Levin, Valeriia Cherepanova, Avi Schwarzschild, Arpit Bansal, C Bayan Bruss, Tom Goldstein, Andrew Gordon Wilson, and Micah Goldblum. Transfer learning with deep tabular models. In International Conference on Learning Representations, 2020.
[40] Caiyan Li and Hongzhe Li. analysis of genomic data. Bioinformatics, 24(9):175--1122, 2008.
regularization and variable selection for
[41] Bo Liu, Ying Wei, Yu Zhang, and Qiang Yang. Deep neural networks for high dimension, low sample size data. In International Joint Conference on Artificial Intelligence Organization, pages 2287-2293 2017.
[42] Yin Lou and Mikhail Obukhov. BDT: Gradient boosted decision tables for high accuracy and scoring efficiency. In Proceedings of the SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1893-1901, 2017.
[43] Yang Lu, Yingying Fan, Jinchi Lv, and William Stafford Noble. DeepPINK: reproducible feature selection in deep neural networks. In Advances in Neural Information Processing Systems, volume 31, pages 8690-8700, 2018.
[44] Katja Luck, Dae-Kyum Kim, Luke Lambourne, Kerstin Spirohn, Bridget E Begg, Wenting Bian, Ruth Brignall, Tiziana Cafarelli, Francisco Campos-Laborie, Benoit Charloteaux, et al. A reference map of the human binary protein interactome. Nature, 580(78033 April 2020.
[45] Donald W Marquardt and Ronald D Snee. Ridge regression in practice. The American Statistician, 29(1):3-20, 1975.
[46] Lukas Meier, Sara Van De Geer, and Peter Bühlmann. The group LASSO for logistic regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 70(1):53-71, 2008
[47] Jörg Menche, Amitabh Sharma, Maksim Kitsak, Susan Dina Ghiassian, Mare Vidal, Joseph Loscalzo, and Albert-László Barabási. Uncovering disease-disease relationships through the incomplete interactome. Science, 347(6224):12576666 2015.
[48] Soufiane MC Mourragui, Marco Loog. Daniel Vis, Kat Moore, Anna G Manjon, Mark A van de Wiel, Marcel JT Reinders, and Lodewyk FA Wessels. Predicting patient response with models trained cell lines and patient-derived xenografts by nonlinear transfer learning. Proceedings of the National Academy of Sciences, 1 : 18, 2021.
[49] Rose Oughtred, Chris Stark, Bobby-Joe Breitkreutz, Jennifer Rust, Lorrie Boucher, Christie Chang, Nadine Kolas, Lara O' Donnell, Genie Leung, Rochelle McAdam, et al. The BioGRID interaction database: 2019 update. Nucleic Acids Research, 47(D1):559--D411 2019.
[50] Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. In International Conference Learning Representations, 2020. 2020.
12

---

[51] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev. Anna Veronika Dorogush, and Andrey Gulin. CatBoost: unbiased boosting with categorical features. In Advances Neural Information Processing Systems, volume 31, pages
2018.
[52] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv:2204.06/25, 2022.
[53] Thomas Rolland, Murat Taşan, Benoit Charloteaux, Samuel J Pevzner, Quan Zhong, Nidhi Sahni, Song Yi, Irma Lemmens, Celia Fontanillo, Roberto Mosca, et al. A proteome-scale map of the human interactome network. Cell. 159(5)1212--1228 2014.
[54] Adriana Romero, Pierre Luc Carrier, Akram Erraqabi, Tristan Sylvain, Alex Auvolat, Eti- enne Dejoie, Marc-Andrè Legault, Marie-Pierre Dubé, Julie G Hussin, and Yoshua Bengio. Diet networks: thin parameters for fat genomies. In International Conference on Learning Representations, 2017.
[55] Jean-François Rual, Kavitha Venkatesan, Tong Hao, Tomoko Kinorane-Ksshkkwa,, Amélie Dricot, Ning Li, Gabriel F Berriz, Francis D Gibbons, Matija Dreze, Nono Jyivi-Guedehoussou,, al. Towards a proteome-scale map of the human protein prottinnn interaction network. Nature, 2005.
(777706))117331111
[56] Camilo Ruiz, Marinka Zitnik, and Jure Leskovec. Identification of disease treatment mechanisms through the multiscale interactome. Nature Communications, 12(1):1-15, 2021.
[57] Lukasz Salwinski, Christopher S Miller, Adam Smith, Frank K Pettit, James U Bowie, and David Eisenberg. The Database of Interacting Proteins: 2004 update. Nucleic Acids Research, (Xsuppp__):D444-DD55 2004.
[58] Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139 1992.
[59] Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84 90, 2022.
[60] Alexander J Smola and Risi Kondor. Kernels and regularization on graphs. In Learning Theory and Kernel Machines, pages 144-158. Springer, 2003.
[61] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. SAINT: Improved neural networks for tabular data via row attention and contrastive pre-training. Table Representation Learning Workshop at NeurlPS, 2022.
[62] Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 1161-1170, 2019.
[63] Damian Szklarczyk, Annika L Gable, Katerina C Nastou, David Lyon, Rebecca Kirsch, Sampo Pyysalo. Nadezhda T Doncheva, Mare Legeay, Tao Fang. Peer Bork, Lars J Jensen, and Christian von Mering. The STRING database in 2021: customizable protein-protein networks, and functional characterization of user-uploaded gene/meessurement sets. Nucleic Acids Research, 49(D1))D005-D612 November 2020.
[64] Robert Tibshirani. Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288 1996.
[65] Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In International Conference on Machine Learning, pages 2071-2080. PMLR, 2016.
[66] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 911):2579-2660 2008
[67] Laurens Van Der Maaten, Eric Postma, Jaap Van den Herik, et al. Dimensionality reduction: comparative. Journal of Machine Learning Research, 10(66-71):13, 2009.
[68] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
13

---

[69] Kavitha Venkatesan, Jean-Francoii Rual, Alexei Vazquez, UIrich Stelzl, Irma Lemmens, Tomoko Atrozane-Kishikawe Tong Hao, Martina Zenkner, Xiaofeng Xin, Kwang- II Goh. et al. An empirical framework for binary interactome mapping. Nature Methods, 6(1):83-90,
2009.
[70] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019.
[71] Quan Wang. Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 29(12)2724-2743, 2017.
[72] Zhen Wang. Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating on hyperplanes. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 28, pages 11 1112-1119, 2014.
[73] Zifeng Wang and Jimeng Sun. TransTab: Learning transferable tabular transformers across tables. In Advances in Neural Information Processing Systems, volume 35, pages 2902-2915, 2022.
[74] David S Wishart, Yannick D Feunang, An C Guo, Elvis J Lo, Ana Marcu, Jason R Grant, Tanvir Sajed, Daniel Johnson, Carin Li, Zinat Sayeeda, Nazanin Assempour, Ithayavani Lynkkaran, Yifeng Liu, Adam Maciejewski, Nicola Gale, Alex Wilson, Lucy Chin, Ryan Cummings, Diana Le, Allison Pon, Craig Knox, and Michael Wilson. DrugBank 5.0: major update to the DrugBank database for 2018. Nucleic Acids Research, (6(DI):D1074-DD D1082, November 2017.
[75] David S Wishart, Yannick D Feunang, An C Guo, Elvis Lo, Ana Marcu, Jason R Grant, Tanvir Sajed, Daniel Johnson, Carin Li, Zinat Sayeeda, et al. DrugBank 5.0: major update to the 2017.
DrugBank database for 2018. Nucleic Acids Research, (6(D1):D1074-D1100
[76] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pages 23965-236909999 PMLR, 2022.
[77] Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, and Yuval Kluger. Feature selection using stochastic gates. In International Conference on Machine Learning, pages 10648-10659, PMLR, 2020.
[78) Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In International Conference on Learning Representations, 2015.
[79] Wanjuan Yang. Jorge Soares, Patricia Greninger, Elena J Edelman, Howard Lightfoot, Simon Forbes, Nidhi Bindal, Dave Beare, James A Smith, Richard Thompson, et al. Genomics of Drug Sensitivity in Cancer (GDSC): resource for therapeutic biomarker discovery in cancer cells. Nucleic Acids Research, 41 (1(DI):D955-DD DD61, 2012.
[80] Yongxin Yang, Irene Garcia Morillo, and Timothy M Hospedales. Deep neural decision trees. In International Conference on Machine Learning, Workshop Human Interpretability in Machine Learning (WHI), 2018
[81] Chengxuan Ying. Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform badly for graph representation? In Advances in Neural Information Processing Systems, volume 34, pages 28877-28888, 2021.
[82] Haiyuann Yu, Leah Tardivo, Stanley Tam, Evan Weiner, Fana Gebreab, Changyu Fan, Ne- nad Svrzikapa, Tomoko Hirozane- Kisaaknwa,,, Edward Rietman, Xinping Yang, et al. Next- generation sequencing generate interactome datasets. Nature Methods, 87))4488- 2011.
[83] Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1):49-67 2006.
[84] Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. XTab: Cross-table pretraining for tabular transformers. In International Conference on Machine Learning, pages 11-44004. PMLR, 2023.
14

---

A Evaluation protocol and hyperparameter ranges
To ensure fair comparison with baselines, we follow evaluation protocols outlined in tabular benchmarks [99 18 We conduct random search with 500 configurations of every model (including PLATO) on every dataset across broad range of hyperparameters. We base the hyperparameter ranges on the ranges used in prior tabular learning benchmarks [19 and the ranges mentioned in the original papers of the methods. Hyperparameter ranges for PLATO are given Table Hyperparameter ranges for baseline methods are given in Table g
Module in PLATO
Hyperparameter
Range
Learning rate
LogUniform(le-4, 5e-3)
General
Batch size
[16, 32, 64]
1.2
Log (1-5,, le-2)
Embedding dimension c
200
KG
Embedding model
ComplEx
# Rounds R
2
Message Passing (MP) Q
3
LogUniform(1ess le-1)
Hidden dimension in A
UniformInt(16, 512)
# Layers
Uniformint(2, 6)
Weight Inference B
Hidden dimension
UniformInt( 16, 512)
# Layers Hidden dimension
UniformInt(2, Uniformint( 16, 512)
Layers 2,.... L in MLP F
Table 7: Hyperparameter ranges used for PLATO.
15

---

Model
Hyperparameter
Range
LASSO
L1
LogUniform((E-4, 10)
Ridge
1.2
LogUniform(1E-4, 10)
n-estimators
Uniformlnt(1,20000
Max depth
UniformInt(3, 10)
Min weight
LogUniform(1E-8....
Subsample
Uniform(0.5, 1)
Learning rate
Deennfform(IE-555
Col sample by level
Uniform(0.5, 1)
XGBoost
Col sample by tree
Uniform(0.5, 1)
Gamma
0. LogUniform( 1E-8, 1E2)
Lambda
0. LogUniform 1E-8, 1E2)
Alpha
0. LogUniform/ 1E-8, 1E2) "ghtree"
Booster
Early-stopping-round
50
Iterations
100
PCA
Number of PCA Components
Uniformlnt(2,1000)
Hidden dimension
UniformInt( (0, 500) UniformInt(1.
Number of layers Activation
[Tanh, Relu, Sigmoid]
STG
Learning rate
LogUniform(1e-44 1e-1)
Sigma
Uniform(0.001, 2) LogUniform(1e-2) 10)
Lambda
Number of layers
Uniformint(1, 8)
Hidden dimension
UniformInt(1, 512)
Dropout
0, Uniform([O.0)
MLP
Learning rate
LogUniform( (1e-5, le-2)
0, LogUniform(le-6 le-3)
Decision Steps
Uniformint(3, 10)
Layer size
2, 4, 8, 16, 32, 64
Relaxation factor
Uniform[1, 2]
Sparsity loss weight
LogUniform[ (e-66 le-1]
TabNet
Decay rate
Uniform[0.4. 0.95]
Decay steps
100, 500, 2000
Learning rate
Uniform(le-3 le-2) 100
Iterations
Embedding dimension
4, 8, 16, 32, 64, 128
Number of heads
UniformInt(1. (, 10)
Number attention blocks Attention dropout rate
UniformInt(1 (1 12)
Uniform(O, 0.5)
Add norm dropout Transformation activation
Uniform(0. 0.5)
[Tanh, Relu, LeakyReLU] Log (1c-6, le-1)
TabTransforma
L2
Learning
LogUniform(le-6, le-3) Uniform(0, 0.5)
FF dropout
FF hidden multiplier
1, 3, 4, 5, 8, 9, 10 [Tanh, Relu, LeakyReLU] Uniform(O, 0.5)
Out FF activation
Out FF dropout
Learning rate
LogUniform( (1-5f 1)
Number of layers
UniformIn(1,, 10)
NODE
Number trees
UniformInt(2. 2048)
Depth
UniformInt(1, 10)
Embedding choice
X random
Diet Network
16

---

Number of layers Hidden dimension Dropout
Uniformint(1, 8)
Uniformint(1, 512)
0, Uniform((0.055)
Learning rate 1.2
LogUniform(le-5, le-2) 0, LogUniformm1e-6 1e-3)
Hidden dimension
Uniformint(1, 512)
Learning rate
LogUniform(le-5, le-2)
GraphNet
À
0, LogUniform(le-5, le2)
LI coefficient
0, LogUniform((e--h le2)
Hidden dimension
Uniformlnt(1, 512)
Learning rate
LogUniform(1e-5,, le-2)
NC Lasso
À
0, LogUniform(1e-5 1e2)
LI coefficient
0, LogUniform (Le-5, 1c2)
Hidden dimension
Uniformlnt(1, (1, 512)
Learning rate
LogUniform (1e55, le-2)
Network Lasso
À
0. LogUniform (1e-5, le2)
LI coefficient
0, LogUUnffrmmlle-3 le2)
Table 8: Hyperparameter range for all baselines.
17

---

B Graph classification approaches
Graph classification models are not appropriate for PLATO's setting. In graph classification models, every input sample is graph with node attributes, and model must make a prediction for that graph. The PLATO problem setting breaks key assumptions made by typical graph classification models. First, graph classification models assume that different samples correspond to different graphs [8 28 27].
However, PLATO every sample corresponds to the exact same graph. There is single background knowledge graph for all samples, and every sample has input features that correspond to the exact same nodes within the knowledge graph. Second, graph classification approaches typically assume that every node in input graph has node attribute [8] 28 27]. However, in PLATO only small subset the nodes in the knowledge graph have measured feature values. Finally, graph classification approaches typically assume small graphs: the largest graph classification task in the Open Graph Benchmark has only 244 nodes [27]. However PLATO, the knowledge graph contains 108,447 and the smallest dataset has 12,932 features corresponding nodes.
C PLATO'S performance across node embedding methods for pre-training the feature embeddings
We conduct an ablation study assess how PLATO'S performance depends on the node embedding method used to pre-train the feature embeddings (Methods3..4 We test three shallow node embedding methods for knowledge graphs which are scalable and prominent: TransE (72). DistMult (78), and ComplEx [65]. We find that PLATO's performance is similar across TransE, DistMult, and ComplEx (Table 9). More generally, PLATO makes no assumption about what type of self self- eeffssuprviiee node embedding method is used to pre-train the feature embeddings. The self eup-suueee embedding step simply module that pre-trains feature embeddings which are then passed the message passing and weight inference modules of PLATO.
KG Node Embedding Method
PearsonR (Test) BRCA Dataset
TransE
0.582 = 0.025
DistMult
0.575 = 0.011
ComplEx
0.583 0.019
Table 9: PLATO'S performance is consistent across knowledge graph node embedding methods.
D Rank ordering of methods for datasets with d > n
we show the rank order performance of all models on all d >>> n datasets. We find that PLATO exhibits consistent and strong performance while the performance of the baselines depends on the specific d > n dataset. For example, TabTransformer is the second best performing of all models on the MNSCLC dataset but the worst performing of all models on the PDAC and CH datasets. Similarly, XGBoost is the second best performing of all models on PDAC but only the tenth best performing of models on BRCA. The baselines with the most stable performance are LASSO and Ridge Regression which rank consistently between the second and eighth best of all models.
In Table 10
18

---

Table 10: For datasets with d > PLATO exhibits consistent and strong performance. By contrast, the performance of the baselines varies with each dataset. For every dataset, the rank order of performance from Table is shown. The best overall model is in bold and the second best model underlined.
Dataset
MNSCLC
CM
PDAC
BRCA
CRC
CH
of features d / of samples
52.2
46.1
40.3
28.2
22.6
19.7
Classic Stat ML
Ridge
2
5
Dim. Reduct.
PCA
7
12
10
7
10
8
LASSO
6
3
5
2
4
Feat, Select.
STG
Decision Tree
XGBoost
10
5
2
GraphNet
5
8
12
3
Graph Reg.
NC LASSO
Network LASSO
3
8
12
9
11
9
Param. Infer.
Diet
13
13
12
13
11
MLP
11
12
NODE
12
10
11
3
6
10
Tabular DL
TabTransformer
2
13
TabNet
11
14
13
14
14
13
Ours
PLATO
1
1
Table 11: For datasets with n, PLATO competitive with baselines. For XGBoost is consistently the strongest baseline. For every dataset, the rank order of performance from Table I1 shown. The best overall model is in bold and the second best model is underlined.
Dataset
ME
BC
SCLC
NSCLC
# of features d / # of samples n
2.0
1.8
1.7
1.1
Classic Stat MI.
Ridge
9
Dim. Reduct.
PCA
13
12
13
LASSO
7.5
6
6
8
Feat. Select.
STG
5
Decision Tree
XGBoost
1.5
2
2
GraphNet
6
4
9
Graph Reg.
NC LASSO
4
Network LASSO
11
10
11
13
Param. Infer.
Diet
14
13
14
14
MLP
10
10
10
NODE
3
11
12
Tabular DL
Tab TabTransformer
12
14
12
11
TabNet
7.5
5
Ours
PLATO
1.5
19

---

E Graph regularization baselines
We test the state-of-the-art graph regularization baselines GraphNet [20), Network-Constrranng
LASSO [40, and Network LASSO [23] The graph regularization baselines can only consider homogeneous graph with only features as nodes and a single edge type. For the graph regularization baselines, we thus induce subgraph between feature nodes from the knowledge graph and collapse all edge types between feature nodes into a single edge type. In this context, GraphNet, Network- constrained LASSO, and Network LASSO correspond to LASSO model with mean-squared error loss and a graph regularization penalty. Let be the graph regularization coefficient, j and be two imput features, let E be the set of edges in the graph, let & R& be the weights of the linear regression for d input features, and let D, be the degree of feature node j. The graph regularization penalty for GraphNet is J, (
)² the penalty for constraid LASSO is
A jEMER
+ VD,
VDL)² )2, and the penalty for Network LASSO is DjKer ( Ox|-
Number of trainable weights in PLATO vs. a multilayer perceptron
Table 12: PLATO drastically reduces the number of trainable weights compared to multilayer perceptron (MLP) across all of the datasets. The number of trainable weights in the best model from the hyperparameter sweep is shown for each dataset.
Model
MNSCLC
CM
PDAC
BRCA
CRC
CH
ME
BC
SCLC
NSCLC
MLP
429665
416961
820097
425217
200529
589761
586945
296113
298929
594209
PLATO
17154
42498
32066
17154
28386
61890
17154
28386
32066
17154
G Dataset Details
We use datasets with >> n and 4 datasets with n [16 17 30 79). In all datasets, a machine learning model must predict the response of a cell line or mouse tumor model to a drug. As input, the model considers tabular dataset. In the tabular dataset, every row corresponds to a specific cell line or mouse tumor model. Every column corresponds to gene name. Every value corresponds to the amount of that gene in the cell line or the mouse tumor (í.e. gene expression). the number of genes is large for all tasks and the number of cell lines or mouse tumor models is comparatively small (i.e. d >> n1.. For every row, the model also takes as input a fixed feature vector corresponding to the drug (i.e. 200-dimensional ComplEx [65] embedding of the drug node in the knowledge graph). The output label is the response the cell line (i.e. In-ic50) [30 79 17 or
practice,
mouse tumor model (i.e. minimum average percent tumor growth aaai---a-pccttumoorr "rommoorrrroh")))) to the drug [16.. Data is available at //ttps://githui httpggttttttg/..///////ggg. com/ /nap- stanford/ppl stanford/llato.
Gene expression datasets were pre-processed following standard process in 48
Briefly, gene
expression values underwent TMM normalization and log transformation (i.e. log( + 1)). Values were made to have zero mean and unit standard deviation. Dataset abbreviations are breast carcinoma
(BC) 33 17), breast carcinoma (BRCA) [66., chondrosarcoma (CH) 30
79 17,, colorectal
cancer (CRC) [16], cutaneous melanoma (CM) [16,, melanoma (ME)
33
79
7 non-small cell
lung carcinoma (MNSCLC) [6], non-small cell lung carcinoma (NSCLC)
30
79 pancreatic
ductal carcinoma (PDAC) [16], and small cell lung carcinoma (SCLC)
30
79
20

---

H Knowledge Graph Details
We compile general biomedical knowledge graph from prior studies 44
35
38
56
63
to use across all datasets. A schematic of the KG is in Supplementary Figure 2 A detailed
breakdown of relation types is in Supplementary Table 13 The knowledge graph is available at https://github ghub.. ctm/
The knowledge graph contains 108,447 total nodes, including 7,975 drugs, 18,370 diseases, 11,447 phenotypes, 22,319 genes, 11,153 molecular functions, 28,748 ,4, biological processes, and 4,184 184 cellular components. Every gene and every drug in every dataset is present as node in the knowledge graph. The knowledge graph also contains 3,066,156 edges with 99 distinct relation types. The remaining node types and their relationships serve broader domain knowledge.
Edges between drug nodes and gene/protein nodes were derived from Drugbank 75 Gao 16 and
the Genomics of Drug Sensitivity in Cancer (79 30 [7]. Edges between diseases and genes/proteins
were derived from DisGeNet [5]. Edges between diseases and phenotypes were derived from the Human Phenotype Ontology [35]. Edges between drugs and diseases were derived from the
Multiscale Interactome [56]. Edges between drugs and side effects were derived from SIDER [38
Edges between genes/proteins and other genes/proteins
derived from BioGRID
49
55 the
Database Interacting Proteins [57].
47
53
82 66), and STRING 63 Finally, edges
from genes/proteins to molecular functions, biological processes, and cellular components well as edges between molecular functions, biological processes, and cellular components were derived from
the Gene Ontology (
21

---

(c)
Legend
10,945
B
67,186
0,32
Node Type
Iuu Nodios)
Molecular
Function (11,153
Biological Precess
1/99
[1]
Component
(20,748)
Num stlons (uum rolation types]
35,897
44,397 [R]
86,801
[4]
Node Type 2
Gene/Protein (22.613)
inum Noceel
2,956,208 (40)
28, [4]
199,772 [14))
(b)
Vyndas 108,447
Drag (1915)
Disesse (18,570)
Phenotype/Side Effect (4-251)
Medaus 3,066.156
142.784
11
Figure 2: Knowledge graph unified knowledge backbone. We constructed knowledge graph as unified knowledge backbone across all datasets. (a) Legend. For each node type, the number of nodes is given in parentheses. Between node types, the number of edges and the number of relation types are given. (b) Number of total nodes and edges across entire knowledge graph. (c) Visual schematic of knowledge graph across each node type.
22

---

Head type
Relation
Tail type
edges
BiologicalProeess
EndsDuring
Biological
BiologicalProeess
HappensDuring
BiologicalProcess
BiologicalProcess
HasPart
BiologicalProcess
229
BiologicalProeess
IsA
BiolosicalProcess
53015
BiologicalProcess
RegguivelyRegulates
BiologicalProcess
2768
BiologicalProoess
PartOf
BiologicalProces.
5193
BiologicalProcess
RositivelyRegulltes
BiologicalProcess
2756
BiologicalProeess
Regulates
BiologicalProcess
3216
BiologicalProcess
OccursIn
CellularCom
149
BiologicalProcess
HasPart
MolecularFunction
173
BiologicalProcess
SeraaivelyRegulacce
MolecularFunction
269
BiologicalProcess
RositivelyRegulates
MolecularFunction
274
BiologicalProcess
Regulates
MolecularFunction
301
CellularComponeet
HasPart
CellularComponent
179
CellularComponent
IsA
CellularComponent
4863
CellularComponent
PartOf
CellularComponent
1990
Disease
AlteredExpression
Gene
7157
Disease
Biomarker
Gene
107160
Disease
ChromosomalRecraageeees
Gene
162
Disease
FusionGene
Gene
166
Disease
Variation
Gene
15076
Disease
GermlineCausallMuttatooo
Gene
4677
Disease
Modifying,Mutation
Gene
10
Disease
SomaticCausal
Gene
130
Disease
MaasepttibliityMutatioo
Gene
441
Disease
Therapeutic
Gene
1793
Disease
Has
Phenotype
195402
Drug
Treats
Disease
5926
Drug
Carries
Gene
866
Drug
Enzymes
Gene
5382
Drug
Targets
Gene
1981 17
Drug
Transports
Gene
3124
Drug
Has
Phenotype
140764
Gene
Associates
BiologicalProcess
43857
Gene
NotAssociates
BiologicalProcess
470
Gene
Associates
CellularComponent
35306
Gene
Colocalizes
CellularComponent
914
Gene
NotAssociates
CellularComponent
160
Gene
NotColocalizes
CellularComponent
11
Gene
Acetylation
Gene
Gene
Activation
Gene
58502
Gene
AdpRibosylation
Gene
Gene
Ampylation
Gene
Gene
Association
Gene
18
Gene
Binary
Gene
56565
Gene
Binding
Gene
287641
Gene
Catalysis
Gene
344801
Gene
Cleavage
Gene
22
Gene
Complexes
Gene
62552
Gene
CovalentBinding
Gene
52
Gene
Deacetylation
Gene
Gene
Demethylation
Gene
Gene
Dephosphorylation
Gene
26
Gene
Deubiquitination
Gene
23

---

Gene
DirectInteraction
Gene
2904
Gene
DisulfideBond
Gene
Gene
DosageGirowth Defect
Gene
9
Gene
Dosagel Dosagel.ethality
Gene
112
Gene
DosageRescue
Gene
63
Gene
Enzymatic
Gene
Gene
Expression
Gene
188
Gene
Geneticlnterference
Gene
32
Gene
Hydroxylation
Gene
26
Gene
Inhibition
Gene
20108
Gene
Kinase
Gene
11960
Gene
Literature
Gene
174162
Gene
Metabolic
Gene
10646
Gene
Methylation
Gene
25
Gene
NegativeGenntiic
Gene
3449
Gene
Gene
Gene
PhenooypicEnhaaneemen
Gene
209
Gene
PhenotyyicSuppression
Gene
214
Gene
Phosphorylation
Gene
166
Gene
Phosphotransfer
Gene
Gene
PhysicalAssociatiion
Gene
824164
Gene
PositiveGenetic
Gene
2331
Gene
PostTranslationaMMooficcaaaa
Gene
5306
Gene
ProteinCleavage
Gene
48
Gene
PutativeSelfhnerratton
Gene
Gene
Reaction
Gene
400658
Gene
Regulation
Gene
2650
Gene
Signaling
Gene
65412
Gene
SynthetieGrowthheefectt
Gene
407
Gene
SymtheticLethality
Gene
816
Gene
SyntheticRescue
Gene
91
Gene
Associates
MolecularFunction
35012
Gene
Contributes
MolecularFunction
596
Gene
NotAssociates
MolecularFunction
285
Gene
NotContributes
MolecularFunction
MolecularFunction
PartOf
BiologicalProcess
1068
MolecularFunction
Regulates
BiologicalProcess
MolecularFunction
OccursIn
CellularComponer
43
MolecularFunction
HasPart
MolecularFunction
204
MolecularFunction
IsA
MolecularFunction
13631
MolccularFunction
NegativelyRegulates
MolccularFunction
42
MolecularFunction
PartOf
MolecularFunction
11
MolecularFunction
RositivelyRegulates
MolecularFunctin
27
MolecularFunction
Regulates
MolecularFunctin
30
Phenotype
IsA
Phenotype
14650
Table 13: Knowledge graph relations between node types.
24


Reviews:
[{"review_id": "yvZhMMNrZe", "reviewer": "Reviewer_E2W5", "summary": "This paper introduces PLATO, an approach handles tabular deep learning challenges when the number of features (d) significantly outnumbers the available instances (n). The authors propose the integration of an auxiliary knowledge graph into the learning process to help the model better understand the relationships between features and hence make more accurate predictions. The author conducted extensive experiments to validate the effectiveness of their approach, involving scenarios with different data sparsity levels, diverse knowledge graph structures, and various types of tabular data. The results demonstrate the proposed method outperforms conventional deep learning approaches on tabular data when the number of features is much larger than the number of instances.", "questions": "Since PLATO utilizes auxiliary knowledge graph to pretrain an embedding for each input feature, will other pretrained methods achieve same competitive performance on the selected dataset?\n\nHow diverse or domain-specific is the auxiliary knowledge graph? Can the proposed method adapt to various domains or does it require specific types of knowledge graphs?\n\nCan the auxiliary knowledge graph combine with other advanced models rather than a simple MLP in the experiment's setting.", "limitations": "The reviewer does not see potential negative social impact of the work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper proposed an original approach to a longstanding issue in machine learning: the 'curse of dimensionality' in situations where the feature dimension (d) greatly exceeds the number of samples (n). Through combining several existing ideas in deep learning and knowledge representation, It creates a potentially transformative technique to address challenge of high-dimensionality in tabular data. \n\nThis paper provided empirical evidence of the method's advancement. The proposed method outperforms 13 state-of-the-art baselines across 6 d >> n datasets by up to 10.19%.\n\nThe paper is generally well written, with clear explanations of the main concepts and methodology.", "weaknesses": "There has been existing work combining knowledeg graph with deep learning on tabular dataset, and the distinctness of proposed method from previous work is not obviously illustrated.\n\nThe robustness and scalability analysis is lacked, how does performance vary with the size and the complexity of the knowledge graph? Does this method scale to large datasets? It would be useful to include a discussion of these issues.\n\nThe proposed method PLATO doesn't show evident advancement compared to selected baselines on 4 d~n datasets, especially for XGBoost, which performance seems to keep up with PLATO and has been proposed over five years, justifying the weakness of advancement for the selected baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": " \n> **Overall response**\n\nWe **thank** the reviewer! The reviewer highlights our “**original approach to** a longstanding issue in machine learning: **the ‘curse of dimensionality’.”** The reviewer says PLATO is a “**potentially transformative technique** to address [the] challenge of high-dimensionality in tabular data” and emphasizes the “empirical evidence of the method’s advancement.” Finally, the reviewer has helped us clarify PLATO’s differentiation from prior work combining tabular data with knowledge graphs (KGs). **While prior work integrating tabular data with a KG models input samples as nodes in the KG, PLATO models input features as nodes in the KG.** \n\n---\n\n> **How is PLATO different from prior work that combines a knowledge graph (KG) with deep learning on a tabular dataset?**\n\n**Prior work models input samples as nodes in the auxiliary KG while PLATO models input features as nodes in the auxiliary KG.** For traditional prediction problems combining an auxiliary KG with tabular data, each input sample is a node in the KG (i.e. users in a social network) and each node has a feature vector from the tabular data (i.e. a vector of the age, height, and weight of the user) [1]. However, PLATO operates in a distinct problem setting. In PLATO, every input feature (i.e. the column name) is a node in the KG. The KG, containing prior relationships between input features, is then used to regularize a separate MLP which predicts the label for each input sample.\n\nFor example, consider a tabular dataset in which each row is a cancer patient, each column is a gene, and each value is the level of each gene in a patient’s tumor. In PLATO, every gene (e.g. gene name) is a node in an auxiliary KG. The auxiliary KG is then used to regularize a MLP which, for each patient, takes in the vector of gene levels and outputs the patient’s survival. Knowledge of prior relationships between genes (e.g. input features) enables strong performance even when there are far more genes (e.g. tens-of-thousands) than patients (e.g. hundreds) in the tabular dataset.\n\n[1] Thomas et al. \"Graph neural networks designed for different graph types: A survey.\" TMLR (2022).\n\n---\n\n> **How does PLATO’s performance vary with the size of the KG?**\n\nWe conduct an **ablation study to assess how PLATO’s performance varies with the size of the KG (e.g. number of edges)**. We randomly remove edges from the KG and measure PLATO’s performance on the BRCA dataset. **With only 50% of the KG’s edges, PLATO still has 71% of the performance as PLATO with 100% of the KG’s edges** (0.412 vs 0.583). Generally, PLATO’s performance improves as the KG grows and there are fewer missing edges (e.g. less incomplete information) (Rebuttal Table 2).\n\n---\n\n> **Does PLATO scale to large datasets?**\n\nPLATO’s key benefit is the ability to learn from datasets with many features but few samples (e.g. $d \\gg n$). **PLATO thus scales to datasets that are large in the number of features** but small in the number of samples.\n\n---\n\n> **For datasets where the number of features is comparable to the number of samples, PLATO exhibits similar performance to XGBoost.**\n\n**XGBoost is currently state-of-the-art for tabular datasets with more samples than features** (e.g. $n \\gg d$) [1, 2]. We thus consider it a **strength**, not a weakness, **that PLATO matches XGBoost when the number of samples is comparable to the number of features** (e.g. $n \\sim d$). The key result, however, is that **PLATO outperforms XGBoost when the number of samples is far smaller than the number of features** (e.g. $d \\gg n$) (Table 1). In other words, PLATO continues to exhibit strong performance in lower-sample regimes where XGBoost’s relative performance falls substantially.\n\n[1] Gorishniy et al. “Revisiting deep learning models for tabular data.” NeurIPS (2021).\n\n[2] Grinsztajn et al. “Why do tree-based models still outperform deep learning on typical tabular data?” NeurIPS (2022).\n\n---\n\n> **PLATO uses the auxiliary KG to pretrain an embedding for each input feature. Will other pretrained methods achieve the same performance as PLATO?**\n\n**We are not familiar with other methods that can use PLATO’s pre-trained input feature embeddings or other pre-trained methods that apply.** First, other methods cannot use PLATO’s pre-trained feature embeddings because of PLATO’s unique problem setting. In PLATO, every input feature in the tabular dataset (e.g. column name) is a node in the auxiliary KG while in other approaches, every input sample (e.g. row name) is a node. PLATO’s input feature embeddings thus do not apply to other models. In fact, how to use these input feature embeddings is nontrivial, and PLATO’s use of them to infer the weights in the first layer of a MLP is itself an innovation. Second, alternative pre-training techniques don’t apply to PLATO’s problem setting because the information that PLATO uses in pre-training is an auxiliary KG rather than another tabular dataset.\n\n**Nonetheless, we still ensure that PLATO outperforms baselines which use the auxiliary KG more generally.** For example, we compare PLATO to three graph regularization methods that use the feature relationships in the KG to smooth the weights in a regression model. PLATO outperforms graph regularization baselines across all $d \\gg n$ datasets by >19% (Rebuttal Table 1, Table 1).\n\n---\n\n> **How diverse or domain-specific is the auxiliary KG? Can the proposed method adapt to various domains or does it require specific types of KG?**\n\n**PLATO can be applied to various domains and does not assume whether the KG used is domain-specific or general.** As long as the KG contains each input feature from the tabular data as a node and also contains prior information about the input features, PLATO can be applied.\n\n---\n\n> **PLATO’s framework applies the auxiliary KG to a MLP. Can PLATO apply the auxiliary KG to more advanced models?**\n\nExpanding PLATO beyond a MLP is an **exciting path for future work**!\n\n---"}, {"review_id": "bxFJmTfieR", "reviewer": "Reviewer_87H1", "summary": "The paper explores tabular ML tasks in scenarios where data has more features than samples and auxiliary structured data is available in the form of knowledge graphs. One of the central assumptions is that each feature corresponds to a node in the knowledge graph. A model called PLATO is proposed which combines a tabular MLP with the auxiliary knowledge graph, with the main idea that for features which correspond to similar nodes in the graph weight vectors in the first MLP layer should also be similar. The paper showcases performance gains of PLATO on 6 datasets compared to tabular models trained solely on the tabular data without leveraging the knowledge graph.", "questions": "1. I encourage the authors to compare against suitable graph neural network baselines for the problem of subgraph classification/regression. One possible baseline is GCN-based node embeddings combined into subgraph embeddings as weighted combinations with weights coming from feature values.\n2. How can categorical features be handled by PLATO?\n3. If PLATO can be used in cases where only some of the features correspond to nodes in the knowledge graph and other features don't, it would be great to see corresponding experiments.\n4. It is mentioned on page 3, line 101 that PLATO \"represents a new regularization mechanism\". If PLATO can be applied as a regularization technique to architectures other than MLP, it would be great to see corresponding experiments. \n5. Expanding on 1, as it is clear from Table 3 that the knowledge graph is a significant driver of performance gains as an additional information source, is it possible to compare against other tabular baselines which also leverage the KG information?", "limitations": "The authors did not address limitations or societal impact of the work.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "I would like to thank the authors for this paper. While I believe that it may not be quite ready for publication, I enjoyed reading the paper. The presentation is excellent and the ideas are explained well, it was definitely a fun read. Overall, I believe the paper has potential and I hope my feedback would help make the paper stronger. \n\nStrengths:\n- This paper explores an interesting and useful idea of leveraging auxiliary knowledge graphs to aid in tabular data tasks\n- The presentation is excellent, the ideas are explained well, with great figures\n- It is clear that the authors made a considerable effort to evaluate multiple tabular baselines\n- The work is reproducible with the code included in the submission", "weaknesses": "I believe that a number of weaknesses need to be addressed before the paper can be published.\n\n**Major:**\n\nGraph perspective\n- My main criticism is that it seems that PLATO is more of a method to learn graph embeddings which leverage auxiliary tabular data to solve subgraph classification or regression tasks rather than a tabular architecture regularized by a graph. This is a key distinction because it defines baselines that the model should be compared to. *That is, PLATO is more of a graph model and should be compared to graph models.* \n\n *Explanation for why PLATO is more a graph model than a tabular model*: As PLATO infers weight vectors in the first MLP layer which correspond to the input features from graph embeddings, it essentially means that that first MLP layer becomes simply a linear combination of graph-based feature embeddings with coefficients coming from feature value for a particular sample. This linear combination is then passed through a nonlinearity and into a downstream model (which in this case is all the other layers of MLP but it does not have to be just that). As the central assumption is that each feature has a corresponding graph node, this means that each sample can be represented as a subgraph with different weights on the subgraph nodes which correspond to feature value. So this becomes a subgraph classification/regression problem with the addition of those weights.\n\n- As such, I encourage the authors to compare against suitable graph neural network baselines for the problem of subgraph classification/regression. One possible baseline is GCN-based node embeddings combined into subgraph embeddings as weighted combinations with weights coming from feature values.\n\nTabular perspective\n- If we think of PLATO as a method for regularizing tabular architectures using a graph, then I see two major limitations: (1) PLATO is limited to MLP architectures only, (2) the central assumption is that *each* feature has a corresponding node in the graph is extremely limiting. If any of these assumptions can be relaxed, then it would be great to see corresponding experiments.\n- It seems likely that the performance gains observed in Table 1 come solely from leveraging the knowledge graph and not as much from the strength of PLATO as a tabular architecture. Other models in Table 1 do not leverage the knowledge graph in any way. Table 3 highlights that performance gains likely primarily come from the knowledge graph as Full KG gives significantly better performance than feature-only KG. That is, the performance gains do not come from regularizing and avoiding overfitting in short and fat datasets, but from the additional information source.\n- Ideally, PLATO should be compared against graph baselines, tabular baselines and some simple graph+tabular baselines to showcase the strength and justify the complexity of the PLATO architecture and design choices.\n\n**Less major:**\n- The paper claims to follow evaluation protocols from [1]. However, [1] used Bayesian HPO framework Optuna which is a stronger approach than random hyperparameter search. \n- As mentioned in the paper, it is true that default FT-Transformer fails with out of memory errors on datasets with many features. However, the RTDL package implements Linformer strategy for efficient attention to address cases with many features. To be able to use it as a baseline model, I encourage the authors to explore the kv_compression_ratio hyperparameter from the FT-Transformer documentation https://yura52.github.io/rtdl/stable/api/rtdl.FTTransformer.html?highlight=linformer\n\n**Related work issues:**\n- A significant portion of related work on tabular deep learning is missing. For example, tabular transofmer architectures [2,3,4] could be added to the section Tabular deep learning methods. Additionally, while the authors say that \"tabular deep learning methods have been developed for settings with far more samples than features\" and \"in the low-data regime with far more features than samples, the dominant approaches are still statistical\", there are in fact tabular deep learning works which specifically address low data regimes through transfer learning [5]. There are also other deep tabular transfer learning architectures [6,7].\n\n \nReferences:\n\n[1] Gorishniy, Y., Rubachev, I., Khrulkov, V. and Babenko, A., 2021. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34, pp.18932-18943.\n\n[2] Hollmann, N., Müller, S., Eggensperger, K. and Hutter, F., 2022. Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXiv:2207.01848.\n\n[3] Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C.B. and Goldstein, T., 2021. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342.\n\n[4] Kossen, J., Band, N., Lyle, C., Gomez, A.N., Rainforth, T. and Gal, Y., 2021. Self-attention between datapoints: Going beyond individual input-output pairs in deep learning. Advances in Neural Information Processing Systems, 34, pp.28742-28756.\n\n[5] Levin, R., Cherepanova, V., Schwarzschild, A., Bansal, A., Bruss, C.B., Goldstein, T., Wilson, A.G. and Goldblum, M., 2022. Transfer learning with deep tabular models. arXiv preprint arXiv:2206.15306.\n\n[6] Zhu, B., Shi, X., Erickson, N., Li, M., Karypis, G. and Shoaran, M., 2023. XTab: Cross-table Pretraining for Tabular Transformers. arXiv preprint arXiv:2305.06090.\n\n[7] Wang, Z. and Sun, J., 2022. Transtab: Learning transferable tabular transformers across tables. Advances in Neural Information Processing Systems, 35, pp.2902-2915.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Overall response**\n\nWe thank the reviewer for their **incredibly thoughtful review**! The reviewer says that the “**presentation is excellent**,” and the idea of “leveraging auxiliary KGs to aid in tabular data tasks” is “**interesting and useful.**” \n\n---\n\n> **The first MLP layer is a linear combination of graph-based feature embeddings with coefficients coming from feature values for a particular sample. The PLATO problem setting should thus be seen as a subgraph classification/regression problem where every input sample corresponds to a subgraph and every node has a one-dimensional node attribute which corresponds to the feature value for that node. A GCN graph classification/regression baseline would then apply.**\n\nThe reviewer makes an **excellent, insightful observation** about PLATO’s first MLP layer! PLATO’s problem setting can indeed be reformulated as graph classification/regression. However, **a graph classification/regression formulation does not work for three key reasons.**\n1. *Every sample would correspond to a subgraph that is structurally equivalent.* Every sample’s subgraph would have the exact same nodes and edges while only the node attributes vary. However, graph classification/regression models assume that different samples correspond to structurally distinct subgraphs [68, 26, 25].\n2. *Every subgraph would be too large for typical graph classification/regression approaches.* For example, the smallest subgraph in PLATO’s datasets would have 12,639 nodes. However, the largest subgraph in the Open Graph Benchmark’s (OGB) graph classification challenge has only 244 nodes [25].\n3. *There would be too few samples (e.g. subgraphs) to learn an effective graph classification/regression model.* The PLATO $d \\gg n$ dataset with the largest number of samples would only have 924 subgraphs. However, the smallest number of subgraphs in the Open Graph Benchmark’s graph classification task has 41,127 graphs [25].\n\n**We attempted GCN-based graph classification/regression models for the PLATO problem setting but they did not work for the reasons described above.** We thank the reviewer for their incredibly insightful comment and will add this discussion to the revised manuscript!\n\n---\n\n> **The authors should compare to graph, tabular, and graph + tabular baselines.** \n\nGraph classification/regression baselines do not apply as described above. For tabular deep learning baselines, we already compare to methods from recent benchmarks [1, 2]. For tabular + graph baselines, we already compare to three graph regularization methods that use the KG. PLATO outperforms all baselines on all $d \\gg n$ datasets (Table 1).\n\n---\n\n> **If PLATO is a tabular deep learning method, there are two limitations: (1) PLATO is limited to a MLP,  (2) each feature has a corresponding node in the graph.** \n\n**Expanding PLATO to architectures beyond a MLP and to datasets where some input features are not in the KG is exciting future work! However, these are not fundamental limitations.** First, we do not claim PLATO is a general regularization method for tabular deep learning. Instead, PLATO is a specific architecture in which the first layer of a MLP’s weights are regularized by an auxiliary KG. Second, future work could expand PLATO to problem settings where only some input features are nodes in the KG! With a MLP in its current problem setting, PLATO already outperforms 10 state-of-the-art baselines on 6 $d \\gg n$ datasets by up to +10.19%.\n\n---\n\n> **It seems that performance gains come solely from using the auxiliary KG rather than regularization. Do any tabular baselines also leverage the KG?** \n\n**Three tabular baselines use the auxiliary KG.** GraphNet, Network-Constrained LASSO (NC LASSO), and Network LASSO are graph regularization methods that use the KG to smooth the weights in a regression model. **PLATO outperforms these baselines by > 19%** (Rebuttal Table 1, Table 1).\n\nWe also do not claim that PLATO’s performance comes solely from “regularization.” We instead emphasize that PLATO’s ability to incorporate prior information is key to performance. Indeed, Table 3 shows that having more information in the KG (i.e. the full KG) leads to better performance than having less information (i.e. the feature-only KG), supporting our main claim.\n\n---\n\n> **The paper claims to follow evaluation protocols from [1]. However, [1] used Bayesian HPO framework Optuna.**\n\n**[1] indeed uses Optuna.** However, we based PLATO on rigorous evaluation protocols in both [1] and [2], two recent tabular learning benchmarks. **[2] uses a random hyperparameter search like PLATO’s.** We reference [1] and [2] since both include hyperparameter ranges for baseline models.\n\n---\n\n> **The authors should add tabular deep learning and transfer learning methods to related work.**\n\nWe will add all suggested references, include tabular transfer learning as an alternative, and revise the statements suggested by the reviewer!\n\n---\n\n> **Can PLATO handle categorical features?**\n\n**Yes, PLATO can handle categorical features.** If the input features are categorical rather than numerical, PLATO would simply learn different weights in the first and later layers of the MLP. \n\n---\n\n> **Line 101 says PLATO \"represents a new regularization mechanism\".**\n\n**We only mean to claim that PLATO regularizes a MLP.** We do not mean to claim that PLATO can regularize any tabular model! We will remove the phrase that PLATO “represents a new regularization mechanism” in Related Work.\n\n---\n\n> **References**\n\n[1] Gorishniy et al. “Revisiting deep learning models for tabular data.” NeurIPS (2021).\n\n[2] Grinsztajn et al. “Why do tree-based models still outperform deep learning on typical tabular data?” NeurIPS (2022).\n\n[68] Yang et al. “Do transformers really perform badly for graph representation?” NeurIPS (2021).\n\n[26] Hu et al. “Strategies for pre-training graph neural networks.” ICLR (2020).\n\n[25] Hu et al. “Open Graph Benchmark.” NeurIPS (2020).\n\n---"}, {"review_id": "LK7uKZsdw6", "reviewer": "Reviewer_p8DV", "summary": "The paper introduces PLATO, a novel machine learning approach for high-dimensional tabular data, where the number of features far exceeds the number of samples (d>>n). This scenario is often encountered in fields like physical sciences and biology. PLATO uses a knowledge graph (KG) to train the first layer of a multilayer perceptron (MLP), creating feature embeddings based on the information in the KG and a message-passing algorithm. This approach leverages auxiliary domain information to regularize the MLP, reducing the risk of overfitting. Benchmarking PLATO on six biomedical datasets revealed it outperformed 13 state-of-the-art baselines by up to 10.19%, demonstrating its effectiveness for the d>>n scenario.", "questions": "Missing / Hard to understand where knowledge graphs come from. Were they created using the full data? Were they created using completely separate experiments? What was the method to create each? the supplement lists only the studies with no further details on how the graphs were obtained.", "limitations": "They adress some limitations. I would add that while PLATO uses domain knowledge, it only models one aspect of the domain knowledge provided by the graph: the similarity of nodes in the graph (Hope I correctly understand that?).", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The mechanism to induce KG regularization is novel. Studying KG integration into predictions seems understudied, while it has been done in specific contexts (e.g. gene interaction networks https://www.nature.com/articles/nmeth.4627, https://www.nature.com/articles/s42003-021-02622-z). The studies setting matters especially in a scientific context. A general method for knowledge graph integration could be very valuable to numerous scientific fields. The work is clearly written, with minor writing issues (especially in conclusion). It seems to provide most experimental details necessary. \n\nAn ablation of varying the fraction of the KG graph used clearly shows that the method is able to learn from increasing domain knowledge.\n", "weaknesses": "Baselines may not be as well tuned, e.g. use combinations of PCA + XGB. Hyperparameters might not be appropriate for the high dimensional setting, e.g. for trees.\nA cross-validation procedure, e.g. 5-fold CV to find optimal hyperparameters on train and valid would be more optimal. Baselines with many hyperparameters to tune would profit more from this stronger evaluation. This is also what data-scientists would perform on such data, especially given the low N.\nIts not an apples to apples comparison for all methods since domain knowledge is used for this approach. A simple baseline using domain knowledge + Baselines would be interesting, e.g. keeping only relevant genes in BRCA dataset based on prior knowledge. I believe significance values for genetic locations from prior studies are much more commonly known than graph connectivity. I understand this may still be hard to evaluate.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n> **Overall response**\n\nWe thank the reviewer for pointing out that **“the mechanism to induce KG regularization is novel.”** The reviewer further notes that our study’s **“setting matters especially in a scientific context”** and that our **“general method for knowledge graph integration could be very valuable to numerous scientific fields.”** We have made multiple clarifications and strengthened the manuscript based on the reviewer’s comments. Thank you!\n\n---\n\n> **How did the authors select the hyperparameter ranges for the baseline methods and the overall protocol to tune hyperparameters and evaluate PLATO compared to the baselines?**\n\n**We selected hyperparameter ranges for the baseline methods and the overall protocol to tune hyperparameters from recent tabular deep learning benchmarks [1, 2].** For each model, we select the hyperparameter range based on the range used in prior tabular learning benchmarks [1, 2] or the original papers. We then hyperparameter tune every model by conducting a random search with 500 configurations of every model on every dataset. We use the **same evaluation and tuning protocol on PLATO and on baselines**, thus ensuring a **fair comparison.** \n\nUltimately, PLATO outperforms all baselines on 6 $ d \\gg n$ datasets by up to 10.19% (Table 1). The evaluation protocol and hyperparameter ranges are further detailed in Appendix A, Table 7, and Table 8.\n\n[1] Grinsztajn et al. \"Why do tree-based models still outperform deep learning on typical tabular data?.\" NeurIPS (2022).\n\n\n[2] Gorishniy et al. Revisiting deep learning models for tabular data. NeurIPS (2021).\n\n---\n\n> **PLATO uses domain knowledge in addition to the tabular baselines to achieve its strong performance. Can the authors compare PLATO to a baseline method that also uses domain knowledge?** \n\n**We compare PLATO to three baseline methods that use domain knowledge.** GraphNet [1], Network-Constrained LASSO (NC LASSO) [2], and Network LASSO [3] are three graph regularization approaches that use the auxiliary knowledge graph in addition to the tabular dataset. In these graph regularization approaches, the feature relationships in the KG are used to smooth the weights in a regression model, thus using domain knowledge as suggested by the reviewer. **PLATO outperforms all graph regularization baselines by >19% (Rebuttal Table 1, Table 1).** We believe these graph regularization baselines are the most relevant comparisons as they use the same systematic prior knowledge as PLATO (e.g. the knowledge graph) rather than a separate prior knowledge source (e.g. hand-engineered features for a particular dataset).\n\n---\n\n> **Where does the knowledge graph come from? Was it constructed from the tabular data or orthogonal information?**\n\n**PLATO’s knowledge graph comes from 6 prior databases which use information that is orthogonal to the tabular data.** To repeat, PLATO’s tabular datasets were not used to construct the auxiliary knowledge graph. Each of the 6 databases that we use to construct the overall knowledge compiles its edges from either relationships previously described in the scientific literature via literature curation (e.g. disease-symptom relationships) or through orthogonal scientific experiments which directly measure edges in the graph (i.e. protein-protein interactions via high-throughput experiments). We detail every database below, the source of it’s relationships, and the reference (numbers from article).\n\n| Database                               | Source                            | Reference in manuscript |\n|----------------------------------------|-----------------------------------|-----------|\n| Protein-protein interactions (Luck et al.)  | Orthogonal Scientific Experiments | [39]      |\n| Human Phenotype Ontology               | Literature Curation               | [31]       |\n| SIDER                                  | Literature Curation               | [34]       |\n| Multiscale Interactome                 | Literature Curation               | [46]       |\n| STRING                 | Literature Curation, Orthogonal Scientific Experiments   | [52]       |\n| DrugBank                               | Literature Curation               | [61, 62]       |\n\n---\n\n> **As a limitation, the authors may consider including that PLATO only models one aspect of the domain knowledge provided by the graph: the similarity between nodes.**\n\nWe thank the reviewer for their suggestion and will include this as a limitation in the revised version of the manuscript! \n\n---\n"}, {"review_id": "WarD7Rri9R", "reviewer": "Reviewer_dZQm", "summary": "The authors present a method to leverage prior knowledge in the form of a knowledge graph for tabular deep learning, illustrating performance on a number of biological prediction problems in which features correspond to gene-level measurements. In this case, the knowledge graph is a biomedical knowledge graph which contains a node for each gene as well as nodes representing related biomedical concepts. A set of initial node embeddings are generated using a standard KG embedding approach. Once these are obtained, a set of node vectors for each sample is generated via message passing on the knowledge graph, making use of the sample values for each feature in the message passing step. These weights are then used as the first layer in a MLP. \n\nThe authors show that their method consistent achieves the best performance across a set of 6 data-sets where the number of features is significantly larger than the number of samples and is consistent with XGBoost when the number of features is comparable to the number of dimensions, suggesting the method could be a good default choice in cases where a relevant knowledge graph is available\n\nThe authors conduct a comprehensive ablation study to demonstrate the necessity of each component of their model in achieving the best prediction performance.", "questions": "Can you include data for missing edges in the other 6 data-sets? Looking at Table 4, I am left wondering if the model will indeed outpeform all baselines when using 70% or 75% of the data\n\nAn interesting extension, or an opportunity for future work, could evaluate whether adding high-confidence predictions from ComplEx was sufficient to recover the performance drop at 50%.", "limitations": "The authors have addressed the key technical limitation: the availability of a good quality and comprehensive knowledge graph. The data presented in Table 4 could be reframed as an additional statement of this limitation - in the presence of an incomplete knowledge graph the performance is worse than simple baselines like Ridge.\n\nI do not believe the model has potential negative societal impact", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The approach is non-obvious and effective, while also being intuitive and clearly explained.The authors have combined ideas from knowledge graph embedding and graph-based regularisation to provide a state-of-the-art model for tabular problems in which a relevant knowledge graph is available. Given the abundance of prediction problems in the biomedical space and the accessibility of high quality knowledge graphs, I expect this method will see many applications there.", "weaknesses": "The authors claim PLATO is robust to missing edges in the knowledge graph. However, when 50% of edges have been removed the model is outperformed by multiple approaches on BRCA, the data-set on which this comparison was carried out. To make this claim, I think the authors need to show that the SOTA performance is maintained across all data-sets in the presence of missing edges. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n> **Overall response** \n\nWe **thank the reviewer** for their excellent review! The reviewer states that PLATO is **“non-obvious and effective”** while also being **“intuitive and clearly explained.”** The reviewer highlights PLATO’s novelty in “[combining] ideas from knowledge graph embedding and graph-based regularization to provide a **state-of-the-art model for tabular problems in which a relevant knowledge graph is available**.” Finally, the reviewer emphasizes that PLATO will **\"see many applications\" in the \"biomedical space\"** given “the **abundance of prediction problems**\" and the \"**accessibility of high quality knowledge graphs**.” We thank the reviewer for their comments which have strengthened the manuscript. \n\n---\n\n> **The authors claim that PLATO is robust to missing edges in the knowledge graph. However, when 50% of the edges have been removed, the model is outperformed by multiple approaches on the BRCA dataset. To make this claim, I think the authors need to show that the SOTA performance is maintained across all data-sets in the presence of missing edges.**\n\n**We thank the reviewer for their comment and will revise the phrase “PLATO is robust to missing edges in the knowledge graph.”** We do not claim that PLATO is state-of-the-art on any of the datasets (BRCA or others) with only 50% or 75% of the edges in the knowledge graph. We only claim that PLATO outperforms the baselines on the 6 $ d \\gg n$ datasets when using 100% of the edges which is shown in Table 1.  \n\nThe ablation analysis shown in Table 4 is meant to demonstrate how PLATO’s performance is affected by the removal of edges in the knowledge graph for the BRCA dataset. We agree that our initial phrase of “PLATO is robust to missing edges in the knowledge graph” should be clarified. We will revise the phrase to read “with 50% of the edges, PLATO retains 71% of the performance as PLATO with 100% of the edges on the BRCA dataset (Table 4).” We thank the reviewer for their comment which strengthens the manuscript! \n\n---\n\n> **The authors test PLATO’s performance when 50% of the edges in the knowledge graph are removed. An interesting direction for future work would be to use a link prediction method to predict some of the missing edges and subsequently apply PLATO to improve performance.**\n\nWe thank the reviewer for their question! When 50% of the edges in the knowledge graph are removed, it may be possible to first use a link prediction methodology to recover some of the missing edges and subsequently apply PLATO. We view this as an **exciting direction for future work** and will add it to the discussion section of our manuscript!\n\n---\n\n> **The authors have addressed the key technical limitation: the availability of a good quality and comprehensive knowledge graph. The data presented in Table 4 could be reframed as an additional statement of this limitation.**\n\nWe thank the reviewer for pointing out that we have already addressed the key technical limitation! Per the reviewer’s suggestion, **we will re-state the importance of knowledge graph in the ablation study in Table 4.**\n\n---\n"}]

Author rebuttal:
> **Overall response**


We are extremely grateful for the highly detailed and constructive feedback on our manuscript! The reviewers describe PLATO as a **“potentially transformative technique to address [the] challenge of high-dimensionality in tabular data”** (E2W5), say that **“the mechanism to induce KG regularization is novel”** (p8DV), emphasize that PLATO “could be **very valuable to numerous scientific fields**” (p8DV) including an “**abundance of prediction problems in the biomedical space**” (dZQm), and say that “the approach is non-obvious and effective, while also being intuitive and clearly explained” (dZQm). 


PLATO is a deep learning method for tabular datasets with orders of magnitude more features than samples. PLATO tackles a **unique problem setting in which every input feature (e.g. column name) in a tabular dataset is a node in an auxiliary knowledge graph (KG)**. By contrast, prior work combining tabular data with KGs models every input sample (e.g. row name) as a node. Since PLATO leverages prior information about the input features, PLATO can achieve strong predictive performance, even when there are orders of magnitude more input features than samples (e.g. $d \gg n$). On **6 datasets with $d \gg n$**, we demonstrate that **PLATO outperforms 10 state-of-the-art baselines by up to +10.19%.**

In our rebuttal, we address the comments of the review team. This has led to multiple improvements including: illustrating that PLATO outperforms baselines that also use an auxiliary KG (E2W5, 87H1, p8DV), clarifying PLATO’s contributions (p8DV, 87H1), explaining PLATO’s limitations (E2W5, 87H1, dZQm), and enhancing the related work (87H1). We answer all reviewer comments below and thank them for the time they put into reading, understanding, and evaluating our paper! Tables for the rebuttal (e.g. Rebuttal Table 1, Rebuttal Table 2) are in the attached PDF.

---


Meta-review: This paper introduces a creative and elegant way to use prior knowledge when training NN on tabular data. The idea is to create a graph such that a feature in the data corresponds with a vertex in the graph. The graph is used in the training process to create a model. The idea is very appealing, and the empirical results are promising. However, the specific implementation can be improved by considering it as a graph learning problem and using GNNs as proposed by one of the reviewers.
Decision: Accept (poster)