Tabular Deep Learning when $d \gg n$ by Using an Auxiliary Knowledge Graph
Authors: Camilo Ruiz
Hongyu Ren
Kexin Huang
Jure Leskovec
Venue: Submitted to ICLR 2023 (2023)
Keywords: Tabular Deep Learning
Knowledge Graph
High Dimensional
Low Sample

Abstract: Machine learning models exhibit strong performance on datasets with abundant labeled samples. However, for tabular datasets with extremely high $d$-dimensional features but limited $n$ samples (i.e. $d \gg n$), machine learning models struggle to achieve strong performance. Here, our key insight is that even in tabular datasets with limited labeled data, input features often represent real-world entities about which there is abundant prior information which can be structured as an auxiliary knowledge graph (KG). For example, in a tabular medical dataset where every input feature is the amount of a gene in a patient's tumor and the label is the patient's survival, there is an auxiliary knowledge graph connecting gene names with drug, disease, and human anatomy nodes. We therefore propose PLATO, a machine learning model for tabular data with $d \gg n$ and an auxiliary KG with input features as nodes. PLATO uses a multilayer perceptron (MLP) to predict the output labels from the tabular data and the auxiliary KG with two methodological components. First, PLATO predicts the parameters in the first layer of the MLP from the auxiliary KG. PLATO thereby reduces the number of trainable parameters in the MLP and integrates auxiliary information about the input features. Second, PLATO predicts different parameters in the first layer of the MLP for every input sample, thereby increasing the MLP’s representational capacity by allowing it to use different prior information for every input sample. Across 10 state-of-the-art baselines and 6 $d \gg n$ datasets, PLATO exceeds or matches the prior state-of-the-art, achieving performance improvements of up to 10.19%. Overall, PLATO uses an auxiliary KG about input features to enable tabular deep learning prediction when $d \gg n$.

Under review as a conference paper at ICLR 2023
TABULAR DEEP LEARNING WHEN d > n BY USING AN AUXILIARY KNOWLEDGE GRAPH
Anonymous authors
Paper under double-blind review
ABSTRACT
Machine learning models exhibit strong performance on datasets with abundant labeled samples. However, for tabular datasets with extremely high d-dimensional features but limited n samples (i.e. d >> n), machine learning models struggle to achieve strong performance. Here, our key insight is that even in tabular datasets with limited labeled data, input features often represent real-world entities about which there is abundant prior information which can be structured an auxiliary knowledge graph (KG). For example, in tabular medical dataset where every input feature is the amount of a gene in patient's tumor and the label is the patient's s survival, there is an auxiliary knowledge graph connecting gene names with drug, disease, and human anatomy nodes. We therefore propose PLATO,, machine learning model for tabular data with d >> n and an auxiliary KG with input features as nodes. PLATO uses a multilayer perceptron (MLP) predict the output labels from the tabular data and the auxiliary KG with two methodological components. First, PLATO predicts the parameters in the first layer of the MLP from the auxiliary KG. PLATO thereby reduces the number of trainable parameters in the MLP and integrates auxiliary information about the input features. Second, PLATO predicts different parameters in the first layer of the MLP for every input sample, thereby increasing the MLP's representational capacity by allowing it to use different prior information for every input sample. Across 10 state-of-ther of-the-rrt baselines and 6 d >> n datasets, PLATO exceeds or matches the prior wate-of-the-ar performance improvements of up to 10.19%. Overall, PLATO uses an auxiliary KG about input features to enable tabular deep learning prediction when >> n.
achieving
INTRODUCTION
Machine learning models have reached state-offthe-art performance in domains with abundant labeled data like computer vision (Wortsman et al.) [2022] Deng et al. 2009) and natural language processing (Wang et al. 2019 Devlin et al.) [201 Ramesh et al. 2022). However, for tabular datasets in which the number of features vastly exceeds the number n of samples, machine learning models struggle to achieve strong performance (Hastie al. 2009 Liu et al. 2017). Unfortunately. many high impact domains like chemistry (Guyon et al. 2004). biology (lorio et al. 2016)] Yang et al. 2012) Garnett al. 2012 Gao et al. 2015). and physics (Kasieczka al. [022]) produce datasets with high-dimensional features but limited labeled samples due to the high time and labor costs associated with experiments.
chemistry, for example, mass spectrometry datasets can have tens of thousands of features but only tens or hundreds of samples (Guyon et al. 22004). For these and other tabular datasets with d >> n, the performance of machine learning systems currently limited.
To date, deep learning approaches for tabular data have focused on data regimes with far more samples than features (n >> d) (Grinsztajn et al.) 2002, Gorishniy et al.] 2021 Shwartz-Ziv & Armon! 2022). In the low-data regime with far more features than samples (d >> n)), the dominant approaches are classical statistical methods (Hastic et al. 2009). These statistical methods reduce the dimensionality
of the input space(Abdl & Williams) 2010 Liu et al.) 2017]] Van der Maaten & Hinton 2008 Van
Der Maaten et al. 2009). select features (Tibshirani 1996] Climente-González et al. 2019 Freidling et al.[[2021| Meier et al. 2008), impose regularization penalties on parameter magnitudes (Marquardt & Snee 1975). or use ensembles of weak tree-based models (Friedman, 2001 Chen & Guestrin
2016 Ke et al. 2017 Lou & Obukhov 2017 Prokhorenkova al. 2018)

---

Under review as a conference paper at ICLR 2023
Here, we present novel problem setting and framework for tabular deep learning when d' > (Figure D Our key insight is that even in tabular settings with limited labeled data, input features often represent real-world entities about which there is abundant prior information which can be structured as an auxiliary knowledge graph (KG). We propose a novel problem setting in which every input feature of tabular dataset corresponds to a node in auxiliary KG (Figure[p) For example, consider tabular medical dataset in which every row is a cancer patient, every column is gene, every value is the amount of that gene in the patient's is tumor, and the task is to predict the patient's ' survival. For this tabular dataset, there exists an auxiliary KG which consists of each gene's s function, the relationships between genes, how a gene affects a part of human anatomy, and how human anatomy itself is structured. Note that the KG does not capture the relationships between input data samples but instead captures the relationships between inpul features.
Within our novel problem setting, we propose PLATO,, a deep learning method for tabular data with + n and an auxiliary KG with input features as nodes (Figure FFigur[[b)-(ee [b)- PLATO uses a modified multilayer perceptron (MLP) to predict the output labels from the input samples and the auxiliary KG with two methodological components. First, the parameters in the first layer of the MLP are predicted from the auxiliary KG and the input sample rather than learned from just the tabular data. PLATO thereby integrates prior information about the input features from the auxiliary KG and drastically reduces the number of trainable parameters in the MLP. Second, the parameters in the first layer of the MI. are predicted differently for every sample by using the auxiliary KG and the sample values. PLATO thereby increases the representational capacity of the MLP and enables effective predictions.
We exhibit PLATO'S performance on 6 datasets. We choose computational biology as it is rich domain for d >> n in which we can construct a single knowledge graph to serve as unified backbone for many distinct tabular datasets with distinct input features. We compare PLATO to 10 state-of-the-as baselines spanning dimensionality reduction, feature selection, classic statistical models, deep tabular learning methods, and parameter- prediction methods. Following a rigorous evaluation protocol from the tabular deep learning literature (Grinsztajn et al., 2022 Gorishniy et al. 202TD. D. PLATO achieves or matches the prior state-of-the-art on all 6 datasets, achieving performance improvements of up to 10.19%. Ablation studies further demonstrate the necessity of each methodological component of PLATO. Ultimately, PLATO uses an auxiliary KG about input features to enable tabular deep learning prediction when >> n.
purametee-prediction
2 RELATED WORK
Tabular deep learning methods. In contrast to PLATO'S setting, tabular deep learning methods have been developed for settings with far more samples than features (i.e. n >> d). Recent tabular deep learning benchmarks ignore datasets with high numbers of features and low numbers of samples (Grinsztajn et al. 2022] Gorishniy et al., 2021, Shwartz-Ziv & Armon 2022). In the n >> d setting, various categories of deep tabular models have been benchmarked. We select the state-of-the-art models to compare against PLATO. First, decision tree models like NODE (Popov) ct al. 2020) make decision trees diffcrentiable to enable gradient- based optimization (Hazimeh et al. 2020 Kontschieder al., 2015) Yang et al. 20188). Second, tabular transformer architectures
an attention mechanism to select and learn interactions among features. These include TabNet (Arik & Plister 2021). Tranrfosfrrmrr (Huang et al. 2020)) FT-Transformer (Gorishniy al. 202 ) and
others
(ong et al. 2019 Somepalli et al. 2021 Kossen et al. 2021).
d d>> n. methods. For PLATO'S setting in which d >> nt, various tabular machine learning approaches exist (Hastie et al., 2009). First, dimensionality reduction techniques like PCA (Abdi & Williams/ 2010) aim to reduce the dimensionality of the input data while preserving as much of the the variance in the data as possible (Liu et af., 2017: Van der Maaten & Hinton, 2008: Van Der Maaten et al.. 2009). Second, feature selection approaches select parsimonious set of features, leading to smaller feature space. Classical feature selection approaches include LASSO (Tibshirani] 1996) and its variants Climente-Gonzáüee
et al. 2019 Freidling et al. 2021 Meier et al. 2008). For feature selection with deep learning. Stochastic Gates Yamadaa et al. 20200) are among the best performing of many variants (Baln et al. [2019] Lu et al. 2018). Finally, classical tree-based models like XGBoost
learn ensembles of weak decision trees models to make an overall prediction (Friedman
& Guestrin 2016 Ke et al. 2017 Prokhorenkova et al. 2018).
2

---

Under review as a conference paper at ICLR 2023
(e)
Densa
(b)
H
(6)
Sampie
Festure
M
XX
Knowledge Graph G
P
())
F
X,1
X
X
Figure 1: PLATO is machine learning model for tabular data with d > and auxiliary knowledge graph with input features as nodes. Machine learning models struggle achieve strong performance on tabular datasets with far more features than n samples (i.e. d > n). (a) The key insight of PLATO is that even in settings with limited labeled samples, input features often represent real-world entities about which there is abundant prior knowledge. We proposc corresponds node an auxiliary knowledge graph (KG) G. (b-e) PLATO uses G to predict the parameters the first layer of a modified MLP F. (b) First, PLATO pretrains an embedding M, E R every feature node in G using H. self-supervised KG node embedding approach. (c) Second, PLATO updates each feature embedding to focus the feature information that is most relevant to input sample X, PLATO uses message passing network sample X,, Q2 thus depends on the input sample (i.e. Q X.). (d, e) Finally, PLATO uses small neural network P to predict the parameters in the first layer of MLP F from Q The parameters in the first layer of FF vary for every input sample X..
problem setting in which every feature in the input matrix
produce Q₂ R". uses attention mechanism which considers the input
Knowledge graph approaches. Existing knowledge graph approaches are designed for tasks directly on the graph such as link prediction (Wang al. 2017 Trouillon et al.] 2016,) Wang et al., 2014] Yang et al. 2015 d' A'mato et al.) 2021). By contrast, PLATO does not make any predictions the knowledge graph. Instead, PLATO makes predictions on a separate, tabular dataset by using the knowledge graph as prior information about the features and domain.
Graph classification approaches. In graph classification models, every input sample is graph with node attributes, and a model must make a prediction for that graph. Graph classification models are relevant for PLATOOS problem setting. Graph classification models assume that different samples correspond to different graphs (Ying et al. 2021 Hu et al., 2020b)a) a). However, in PI.ATO every input sample corresponds to the exact same graph. Additional comments
Appendix B
Parameter prediction. Using one network predict the parameters of another has been extensively studied (Denil et al., 2013) Schmidhuber 1992 Bengio et al. 1991). For example, Ha et al. (0016) predicts the weights all layers of sequential model (i.e. RNN, LSTM) by using information about the structure of the weights. Another approach, Diet Networks (Romero et al.) 2017) predicts parameters by hand-crafting prior information about the input features or using random projections. By contrast, PI.ATO predicts parameters in network by leveraging prior information about the input features in an auxiliary KG. PLATO systematically constructs an embedding for each input feature which contains the prior information about the feature that is most relevant to given sample.
PLATO
PLATO is a machine learning method for tabular datasets with > n and an auxiliary KG with input features nodes (Section[.11) The key insight of PLATO is that even in tabular datasets with limited labeled samples, input features often represent real world entities about which there is abundant prior information which can be structured auxiliary knowledge graph (KG) G (Figure a)).
3

---

Under review as a conference paper at ICLR 2023
PLATO uses modified MLP to predict the output labels from the input samples and the auxiliary KG. PLATO's modified MLP has two key methodological components. First, the parameters in the first layer of the MLP are predicted from the auxiliary KG and the input sample rather than learned from just the tabular data (Figure [b-)). PLATO thereby integrates prior information about the input features from the auxiliary KG and drastically reduces the number of trainable parameters in the MLP. Second, the parameters in the first layer of the MLP are predicted differently for every sample by using the auxiliary KG and the sample values. PLATO thereby increases the representational capacity of the MLP and enables effective predictions. The full PLATO Algorithm is given in Algorithm
3.1 PROBLEM SETTING
Consider a tabular dataset X E Ruxd with labels y E R and far more features than samples such that d >> The goal is to train machine learning model F to predict labels & from the input X. PLATO assumes the existence of an auxiliary knowledge graph G = (V, E) with [V nodes and EE] edges such that every input feature j corresponds to a node in G. Formally, Vi E {l...., d}. Bu E V s.t. + as shown in Figure a). G also contains additional nodes which represent broader knowledge about the domain. The edges in G are (head node, relation type, tail node) triplets.
3.2 THE PLATO MLP F
Consider a standard MLP y = ((x; Q) with L layers, h hidden units in the first layer, and trainable parameters θ = [ A|! θ I.). The PLATO MLP F differs from T in two key ways.
First, the parameters in the first layer of PLATO'S MLP F are predicted from prior information rather than learned only from the tabular data. We observe that every parameter in the first layer of T is associated with an input feature j. In particular, @ E Rdxh such that O [4]] E Rh is a vector of parameters connecting input feature j to every hidden unit in the first layer of the MLP (Figure (e). Typically. T learns the parameters @!!l and @x associated with two features j and independently by gradient backpropagation. In PLATO, we propose that if two input features j and represent real-world entities that are related, then their corresponding parameters @!!l and ! should be related too. To capture the intuition that related input features j and & should have related parameters, PLATO's MLP F predicts O and θ
j
[1] from prior information known about j and k. If
input features j and & are related, then the parameter prediction module produces related 6.") and 6 Parameter prediction details are in Section 3.3 For now, note that the parameters in the first layer of PLATO's MLP F are predicted such that O! e Rdx The parameters O @ ( in the remaining layers of PLATO'S MI.P F are learned normally.
Second, the parameters in the first layer of PLATO'S MI.P are allowed to vary for every input sample. In the standard MLP T, all parameters !!!) ...., ( the same for every input sample X, In the first layer of PLATO'S MLP J, however, the parameters Gll are being predicted from prior information about the input features. We observe that for each input sample X,, the most relevant prior information about each input feature j might differ. Therefore for each sample X,, PLATO uses different prior information about each input feature j to predict the parameters 6 As a result, the parameters (!) in the first layer of F vary with each input sample X,, increasing the representational capacity of F. How PLATO uses different prior information for parameter prediction left Section 3.3 For now, note that PLATO predicts ).") from prior information about feature j.
The prior information about feature that is used depends on the input sample X, As result, 6 is conditional on X, according to @!I|| [X,
Formal Notation. Overall, PLATO'S MLP F takes the form
(1) F has parameters é (611) X U ( e where T. is the number of layers in F. The
y: ê(x)).
parameters 6 X; in the first layer of F are predicted from the input sample X; via message-passing
on the KG according to Section 3.3 For every sample new 61'l is predicted such that @!1]
4

---

Under review as a conference paper at ICLR 2023
is conditional on X, at both training and inference time. The dimensionality of 6 R is the same as in a normal MLP where is the number of hidden units in the first layer of F. The parameters in the remaining layers of PLATO'S MLP @ ..... @14) are the same as in standard MLP: they are learned normally, are the same for every sample at both training and inference time, and are thus not conditional on X.
3.3 PREDICTING THE PARAMETERS IN THE FIRST LAYER OF PLATO'S MLP F
PLATO uses prior information about the input features to predict the parameters in the first layer of PLATO'S MLP F.. PLATO predicts these parameters in three steps. First, PLATO uses superviirvi on the auxiliary KG to pretrain an embedding for every input feature (Section 3.3. Figure (b). Second, since different input samples might rely different prior information about each input feature, PLATO updates each feature embedding to contain the most relevant prior information about
the input feature for the given input sample (Section 3.3.2 Figure [c)). Finally, PLATO predicts the parameters in the first layer of F from the updated feature embeddings with small neural network that is shared across input features (Section 3.3.3 Figure (d)(e)..
3.3.1 PRETRAINING FEATURE EMBEDDINGS WITH SELF- SUPERVISOO
First, PLATO learns general prior information about each input feature from the auxiliary KG G' (Figure [)). PLATO represents the general prior information about each input feature j low-dimensional embedding M2 E IR.. Since every input feature j corresponds to a specific node in
G. PLATO can learn M, by learning an embedding for the corresponding feature node in G. Any self-supervised node embedding method on G can be used within PLATO's 's framework.
Formal notation. Formally, PLATO uses self- on G to pretrain an embedding for every input feature according
M H(G)).
(2)
M E Rd is the matrix of all feature embeddings. H. is a self-supervised node embedding method.
We refer Eq. ( as pretraining since only the auxiliary KG G' is used but the tabular data X, is ignored. After pretraining, the feature embeddings M are fixed.
Implementation. For H. we choose ComplEx as prominent and highly scalable KG node embedding method (Trouillon et al. 2016). ComplEx uses ceff-supervised objective which learn an embedding for every node in G by classifying whether a proposed edge exists in G. ComplEx's 's proposed edges include both feature nodes and other nodes in G, thereby integrating prior information about the input features and the broader domain.
3.3.2 UPDATING FEATURE EMBEDDINGS TO CONTAIN THE MOST RELEVANT INFORMATION FOR AN INPUT SAMPLE
Since different input samples might rely on different prior information about each input feature, PLATO next updates each feature embedding M, Rdxc to Q, E Rd a feature embedding which contain the most relevant prior information about feature j for a given input sample X; (Figure (F)). PLATO uses pesssge-passing network Q on the KG update the feature embeddings in a way that minimizes the number of additional trainable parameters.
Q(X; M, G)
The message-passing network in @ uses an attention mechanism which considers the sample values X, update the feature embeddings. The attention mechanism has small number of trainable parameters II.
Let 0²"² be the embedding of input feature j after round r E (1,..., R} of message passing. For every input feature j. Q first initializes the updated feature embedding to the pretrained feature embedding.
The message passing network Q.
2 00]
M
(3a)
5

---

Under review as a conference paper at ICLR 2023
Q then conducts R rounds of message passing. In each round of message passing, the feature embedding Q.11 is updated from the feature embedding of each neighbor in the prior round Q!-"" and its own feature embedding in the prior round Q Q!!-!!_ The "message" being passed is the embedding of each feature from the prior round.
Weighted messages from neighbors
Q [r]
3 KEN,
A Qk 1
+ (1- (1
1-
(3b)
Weighted message from self
Q2 = = Σ
Ma) (1-
(3c)
KEN;
a is an optional nonlinearity. Nj are the neighbors of feature node j in G.
During message-passing, Q uses two scalar values 3 E R and Aijk E IR to control the weights of messages. First, Q uses hyperparameter to control the weight of the messages aggregated from all neighbors vs. the message from the feature node j itself. Second, Q calculates Aijk to control the weight of the specific message between feature j and neighbor k. The attention score is different for every sample i and is calculated by shallow neural network A with small number of trainable parameters II. The attention score Aija thus enables Q to update the information in the feature embedding way that is most relevant for the input sample i. Formally:
attention score
exp (A(Xi) Xiki IT) Aijk ΣLEN, exp (A(Xi) X II)
(3d)
The number of trainable parameters in is small since the input of A is TR. and the output of A is scalar IR. A and its parameters II are shared for all samples and features.
Finally, the updated feature embeddings Q, are set after R rounds of message-passingg
Q2 = -
(3e)
3.3.3 PREDICTING THE FIRST LAYER OF PARAMETERS IN F FROM THE UPDATED FEATURE EMBEDDINGS
Finally, PLATO predicts the parameters in the first layer of J. from each updated feature embedding (Figure (dcuure[)(e)
[d))). Every parameter in the first layer of F is associated with a feature PLATO thus predicts , the parameters associated with the feature j. from Q,, the prior information about j.
Formal notation. PLATO predicts the parameters associated with every input feature in the first layer of J according to
6.' = P(Q, X,;
P is shallow neural network parameterized by 4. Q is the updated feature embedding of j which is conditional on the specific input sample X; 4 are the parameters of P. P and its parameters $b are shared for every feature j E (1.....d).
PLATO drastically reduces the number of trainable parameters compared standard MLP. The sharing of 'P and & across all input features enables drastic reduction in the number of trainable parameters compared to standard MLP. For high-dimensional tabular dataset (i.e. d > n), standard MLP T with h hidden units has large number of trainable parameters in the first layer since [I]] E ddx A standard MLP T must learn all dh of these trainable parameters independently. By contrast, P uses shared set of trainable parameters 4 to predict θ from Qj for every E (1...., dd.. The number of trainable parameters + is small compared to dh since P need only transform every Q, E R to @ E Rh. Thus, | = ch (assuming that P is a single layer neural network). c, the dimensionality of the feature embedding, is much less than d the number of input features. As a result, ( = ch & dh and PLATO drastically reduces the number of trainable parameters in the first layer of ML.P.

---

Under review as a conference paper at ICLR 2023
Algorithm 1: The PLATO Algorithm.
Input:
a data sample X; RRd knowledge graph (KG) G (V,
predicted label y; Pretrain KG embedding for every feature: - H())
Output:
Initialize feature embedding for feature j: 0, = Mj
Compute sample i-specific attention weight: Aijl: = exp (A(X, II)
Eiex, (A(X, X V features j, k,
where A is NN parameterized by II
ro r S R do
Update feature embedding with message passing neural network at layer r:
0 |rl B(EKEN, Aqja Q + 3) -1
end
Obtain feature j embedding from GNN last layer R: Q, = 0,
Predict the parameter of first layer of MLP from the feature embedding: 6 P(Q Ø).
where P is NN parameterized by +b
Concatenate the first layer predicted parameters with the parameters from the rest of layers:
= [ U [
014]
10 Predict label: y; F(X; X.), where F is an MLP parameterized by b
Table 1: PLATO outperforms statistical and deep baselines when d >> n. For every dataset, best overall model is in bold and the second best model is underlined.
Dataset
MNSCLC
CM
PDAC
BRCA
CRC
CH
15,390
13,183
12,932
12,693
18,206
19,902
N
295
286
321
476
562
924
DIN
52.2
46.1
40.3
28.2
22,6
19.7
Classic Stat ML
Ridge
0.153
0.390=0000
0.344=000
0.538
0,376:0000
0,546=0000
Dim. Reduct.
PCA
0.15630:11
0.070
0.23222012
0.452
0.193
0,237
Feat, Select.
LASSO
0.168
0.431=00000
0.346=0000
0,470=a000
0.400=0.00
0.547=0.00
STG
0.132:6410
0.36611003
0.258:a08s
0.485
0.301 :000
0.262:00m6
Decision Tree
XGBoost
00.02=0000
0.225
0.363-0000
0.347:0000
0.354:2000
0.728
Param.
-0.04_0309
0.054 149
0.309_nmm
0.21 110000
0.087
MI.P
1.128
0.322.1004
0,289_00m
0.355 0.2401
0.044.9099
Tabular DL
NODE
a.003
0.190=mmm
0.512
0.344:0000
0. 1118 20000
Tab Transformer
0.265
0.072::0.00
0.029=0000
0,202#0:00
0. 0,238=0000
0.020=00007
TabNet
0,085 10022
0.010:006s
0.088=0007
0.055
0.018
0.039
Ours
PIATOO
0.272
0.435
0.400
0.5
:oue
0.770
4 EXPERIMENTS
We evaluate PLATO against 10 statistical and deep baselines on 6 tabular datasets with d >>>>>> n.
Datasets. We use 6 tabular d >> n datasets in biomedicine compiled from prior studies (Gao et al. 2015 Garnett al. 2012: Torio et al., 2016, Yang et al.) 2012). We focus biomedicine because it is a rich domain for d >> in which single KG can be used as a unified knowledge backbone across many datasets. Additional data descriptions are in Appendix[E
Auxiliary Knowledge Graph. As a unified knowledge backbone for the datasets, we compile a general biomedical knowledge graph from prior studies (et al., 2020; 2016 Kuhn et al.) 2015) Ruiz jet al., 2021 Szklarczyk et al., 2020 Wishart et al., 2017aa b). Our knowledge graph contains 108,447 nodes, 3,066, 156 edges, and 99 relation types. All datasets include features which map to subset of nodes in the knowledge graph. The remaining nodes serve as broader domain knowledge. The same KG is used across all datasets even though all datasets have distinct feature sets with distinct cardinalities. PLATO thus allows a single KG to serve as a unified knowledge backbone for different datasets in domain. Additional KG details are in Appendix F
7

---

Under review as a conference paper at ICLR 2023
Table 2: PLATO's performance depends on updat- ing feature embeddings to contain information that is specific to given sample.
Table 3: PLATO's performance depends both feature nodes in G and other nodes which represent broader domain knowledge.
Parameter Pred. Input
Feature Information
Sample Specilic
PearsonR
Auxiliary KG
Feature-only Knowledge
Broader Knowledge
PearsonR
Updated feat, embed. Q General feat, embed M None
0
Full KG
0.583
0.522=0000
Feature-only KG
X
x
0,240=0067
No
0.240=000m
Table 4: PLATO's performance is competitive with baselines when d ~ Tr, For every dataset, the best overall model in bold and the second best model underlined.
Datasct
ME
BC
SCLC
NSCLC
19,902
10.101
10.712
16,730
19,902
18,261
18,437
18,308
DIN
2.0
1.8
1.7
1.1
Classic Stat ML
Ridge
0.566110.0s
0.483.10008
0.60410057
0.679
Dim. Reduct.
PCA
0.239
0.233
0.284
0.645
LASSO STG
0.667=00000
0.633gn000
0.66920000
0.637
Feat. Select.
0.676200000
0. 2.6432000
0. 0.66810000
0.64620000
Decision Tree
XGBoost
0.875
0.826:00000
0.87810000
0.843:aum
Param. Pred.
Diet
0.105
0.037
=0.05
0.002
MI.P
0.487=0.131
0.508=200l
0.53720061
0.573
Tabular DL
NODE
0.870200000
0.420:2.106
0.487 1:001 100
197
TabTransformer
0.305
0.010
0,2881020
0.503
TabNet
0.667
0.624:0000
0.657:0004
0.647
Ours
PLATO
0.875=:0.04
0.844:000
0.883
0.839#0000
Baselines. We compare PI.ATO to 10 state-of-the art statistical and deep baselines. We consider
classic regularization with Ridge Regression (Marquardt & Snee (975), dimensionality reduction with PCA (Abdi & Williams 2010), feature selection with LASSO (Tibshirani] 1996) deep feature selection with Stochastic Gates Yamada al. 2020), and gradient boosted decision trees with XGBoost (Chen & Guestrin 2016). We also consider deep tabular learning methods including standard MLP. self-attention- based tabular methods with TabTransformer (Huang et al. 2020) and TabNet (Arik
Phister 202T) differentiable decision trees with NODE (Popov et al. 2020)) and parameter-predict with Diet Networks (Romero et al., 2017). We also attempted FT Fi-Transformer (Gorishniy al..
2021) but it experienced out of memory issues all datasets due to the large number of features.
Fair Comparison of PLATO with Baselines. To ensure fair comparison with baselines, we follow evaluation protocols tabular benchmarks (Grinsztajn et al.) 2022, Gorishniy et al., 2021). ). We conduct a random search with 500 configurations of every model (including PLATO) every dataset across a broad range of hyperparameters (Appendix A}. We split data with a 60/20/20 training, validation, test split. All results are computed across 3 data splits and 3 runs of each model in each data split. We report the mean and standard deviation the Pearson correlation (PearsonR) between y and across runs and splits on the test set. Each model is run on a GeForce RTX 2080 TI GPU.
4.1 RESULTS
PLATO outperforms statistical and deep baselines when d > n. PLATO outperforms all baselines across all 6 datasets with d > n (Table D.. PLATO achieves the largest relative improvement on the PDAC dataset, improving by 10. vs. XGBoost, the best baseline for PDAC (0.400 vs. 0.363). While PI.ATO achieves the strongest performance across all 6 datasets, the best performing baseline varies with different datasets. Ridge Regression is the strongest baseline for BRCA, LASSO for CM and CRC, XGBoost for PDAC and CH, and TabTransformer for MNSCLC. The remaining baselines (PCA, STG, Diet Networks, MLP, NODE, and TabNet) are not the strongest baseline for any dataset. We also find that the performance of specific baseline depends largely the specific dataset. Tranrrrsformer, for example, is the best baseline for the MNSCLC dataset but the worst baseline for the CH dataset. The rank order of all models on all datasets is Appendix C
8

---

Under review as a conference paper at ICLR 2023
PLATO's performance depends updating feature embeddings to contain information relevant to sample. PLATO predicts the parameters 6 in the first layer of a modified MLP FF by using feature embeddings which contain prior information about the input features. PLATO first pretrains general feature embeddings M Rd PLATO then updates the feature embeddings Q Raxe which contains information about the input features that is most relevant to a given sample X, We given X; is necessary by evaluating PLATO on the BRCA dataset in three configurations (Table 2). The default configuration uses the updated feature embeddings Q to predict O according to O _ = P(Q The second configuration uses the general feature embeddings M instead of Q to predict 6l") according to G =P(M). The third configuration does not use feature embeddings and thus reduces to a standard MLP.
test whether updating the feature embeddings based
We compare PLATO'S performance when it uses feature embeddings which contain the relevant information for given sample X; (i.e. 611) P(Q vs. the pretrained feature embeddings M which contain general information about the input features that is not specific to a given sample (i.e. 6 = P(M)). Using general feature embeddings M improves over not using feature embeddings at all (0.522 vs. 0.240). Using feature embeddings Q that are specific to given input sample further improves performance (0.583 vs. 0.522). Therefore, updating the feature embeddings to Q such that they contain the information specific given sample is critical to PLATO's performance.
PLATO'S performance depends on both feature nodes and broader knowledge nodes in the auxiliary KG. PLATO relies on an auxiliary KG G which contains information about input features and information about the broader domain. Information about input features is represented feature nodes while information about the broader domain is represented as other nodes in G' (Methodss3.I). To test the relative importance of the feature information G' vs. the broader domain information, we measured the performance of PTATOO on the BRCA dataset in two KG configurations: PI.ATO with the full KG (i.e. both the feature nodes and the broader domain nodes) and PLATTO with a "feature-only KG (i.e. an induced subgraph on only the feature nodes) (Table(3). We also compare to a "No KG" configuration which PLATO does not have access to the KG. Without auxiliary information about the input features or the broader domain, PLATO is ablated to become standard MLP.
We find that both the feature nodes and the broader knowledge nodes are important for PLATO'S performance. Using the "feature-only KG" configuration of PLATO improves performance vs the "no KG" configuration of PLATO (0.539 vs 0.240). Using the "full KG" configuration further improves performance vs the "feature-only KG" configuration (0.583 vs 0.539). PI.ATO's performance thus relies on both the feature information and the broader domain information in the KG.
For datasets with d ~ nI, PLATO is competitive with baselines. Finally, we test PLATO'S perfor- mance for datasets with d n. We test datasets with d ~ n ranging from & 1.1 to 2.0 (Table We find that on 4 datasets with d ~ n, PLATO is competitive with the best performing baseline, XGBoost, but does not improve performance substantially. PLATO's 's stronger performance for datasets with d >> than for datasets with d ~ n is justified. PLATO's key idea is to include auxiliary information about the input features. Auxiliary information is likely to help performance the most in settings with the least labeled data (i.e. d > nt). When d ~ n, auxiliary information is less helpful since the tabular dataset may already have enough information to train a strong predictive model. We further find that XGBoost is consistently the strongest baseline for datasets with d' ~ n, in contrast to the varied performance of XGBoost on the datasets with d > n (Table
DISCUSSION
PLATO is a machine learning model for tabular data with d >> n and an auxiliary KG with input features as nodes. Across 6 datasets and 10 baselines, PLATO achieves state-of-the-art performance, including relative performance improvements of up to 10.19%. Ablation studies also confirm the importance of each component of PLATO. PLATO has several limitations. First, PLATO matches but does not substantially improve the performance of baselines for high-dimensional datasets with more samples (i.e. d ~ n).. Second, PLATO relies on the coverage of prior information. Datasets with input features that have little prior information in the KG are less likely to benefit from PLATO. Overall, PLATO uses an auxiliary KG about input features to enable tabular deep learning when d >>> n.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Hervé Abdi and Lynne J Williams. Principal Component Analysis. Wiley Interdisciplinary Reviews:
Computational Statistics, 2(4))433-459) 2010.
Sercan Ô Arik and Tomas Pfister. TabNet: Attentive interpretable tabular learning. In Proceedings of
the AAAI Conference on Artificial Intelligence, volume 35, pp. 6679-6687, 2021.
Muhammed Fatih Balm, Abubakar Abid, and James Zou. Concrete autoencoders: Differentiable feature selection and reconstruction. In International Conference on Machine Learning, pP-
444 453. Proceedings of Machine Learning Research, 2019.
Anna Bauer-Mehren, Michael Rautschka, Ferran Sanz, and Laura Furlong. DisGeNET: cytoscape plugin to visualize, integrate, search and analyze gene-disease networks. Bioinformatics, 26(22): 2924 2926, 2010.
Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning a synaptic learning rule. In Interna- tional Joint Conference on Neural Networks, volume 2, pp. 969-975. IEEE, 1991.
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 785-794, 2016.
Héctor Climente-Gonzáäee Chloé-Agathe Azencott, Samuel Kaski, and Makoto Yamada. Block HSIC lasso: model-free free biomarker detection for ultra-high dimensional data. Bioinformatics, 35
(14):4477-i435, 2019.
Gene Ontology Consortium. The Gene Ontology resource: 20 years and still GOing strong, Nucleic Acids Research, 47(DD)):D330-D338 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale scale hierarchical image database. In IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255. IEEE, 2009.
Misha Denil, Babak Shakibi, Laurent Dinh, Marc' Aurelio Ranzato, and Nando De Freitas. Predicting parameters in deep learning. In Advances Neural Information Processing Systems, volume 26, pp. -156,, 2013.
Jacob Devlin, Ming-Wet Chang, Kenton Lee, and Kristina Toutanova. BERT:
Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
Claudia d' A'mato,, Nicola Flavio Quatraro, and Nicola Fanizzi. Injecting background knowledge into embedding models for predictive tasks on knowledge graphs. In European Semantic Web Conference, pp. 441-457. Springer, 2021.
Katja Luck al. reference map of the human binary protein interactome. Nature, 580(7803): 402 408, April 2020. doi: 10.1038/s4 0288-000-2188-- 141586-020-12
Sebastian Köhler et al. The Human Phenotype Ontology in 2017. Nucleic Acids Research, 45 (DI):D865-D87,, November 2016. doi: 10. 10.10933/ar/gkw10039 URL bttps://6oi.org/10
1093/nar/qkw1033
Tobias Freidling, Benjamin Poignard, Héctor Climente-Gonzálle, and Makoto Yamada. Post-selection inference with HSIC-Lasso. In International Conference on Machine Learning, pp. 3439-3448.
PMLR, 2021.
Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of Statistics, pp. 1189-1232, 2001.
Hui Gao. Joshua Korn, Stéphane Ferretti, John E Monahan, Youzhen Wang, Mallika Singh, Chao Zhang, Christian Schnell, Guizhi Yang, Yun Zhang, et al. tigh-throughput screening using patient-derived tumor xenografts to predict clinical trial drug response. Nature Medicine, 21(11): 1318-1325, 2015.
10

---

Under review as a conference paper at ICLR 2023
Mathew J Garnett, Elena J Edelman, Sonja Heidorn, Chris D Greenman, Anahita Dastur, King Wai Lau, Patricia Greninger, Richard Thompson, Xi Luo, Jorge Soares, et al. Systematic identification of genomic markers of drug sensitivity in cancer cells. Nature, 483(7391):5700-7777 2012.
Yury Gorishniy. Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In Advances in Neural Information Processing Systems, volume 34, pp. 18932-18943, 2021.
Lco Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2022.
Isabelle Guyon, Steve Gunn, Asa Ben-Hur, and Gideon Dror. Result analysis of the NIPS 2003 feature selection challenge. In Advances in Neural Information Processing Systems, volume 17, pp. 545-5555 552, 2004.
David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. In International Conference on Learning Representations, 2016.
Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
Hussein Hazimeh, Natalia Ponomareva, Petros Mol, Zhenyu Tan, and Rahul Mazumder. The tree ensemble layer: Differentiability meets conditional computation. In International Conference Machine Learning, pp. 4138 4148. PMLR, 2020.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open Graph Benchmark: Datasets for machine learning on graphs. In Advances in Neural Information Processing Systems, volume 33, pp. 22118-22133, 2020a.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. International Conference on Learning Representations, 2020b.
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. TabTransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678, 2020.
Francesco lorio, Theo A Knijnenburg, Daniel J Vis, Graham R Bignell, Michael P Menden, Michael Schubert, Nanne Aben, Emanuel Gonçalves, Syd Barthorpe. Howard Lightfoot, al. A landscape of pharnacogenomic interactions in cancer. Cell, 166(3):740-754, 2016.
Gregor Kasieczka, Benjamin Nachman, David Shih, Oz Amram, Anders Andreassen, Kees Benk- endorder, Blaz Bortolato, Gustaaf Broojimans, Florencia Canelli, Jack Collins, et al. The LHC Olympics 2020: a community challenge for anomaly detection in high energy physics. Reports on Progress in Physics, 84:124201, 2021.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. LightGBM: highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems, volume 30, pp. 1146-1154 2017.
Peter Kontschieder, Madalina Fiterau, Antonio Criminisi, and Samuel Rota Bulo. Deep neural decision forests. In Proceedings of the IEEE International Conference Computer Vision, pp. 1467-1475, 2015.
Jannik Kossen, Neil Band, Clare Lyle, Aidan N Gomez, Thomas Rainforth, and Yarin Gal. Self- attention between datapoints: Going beyond individual input-output pairs in deep learning. In Advances in Neural Information Processing Systems, volume 34, pp. 28742-28756. 2021.
Michael Kuhn, Ivica Letunic, Lars Juhl Jensen, and Peer Bork. The SIDER database of drugs and side effects. Nucleic Acids Research, 44(D)))D075-DD007) October 2015. doi: 10. 0093//ma/gkv107755 URL RtRLhttps: http:///o.......0000000/////////

---

Under review as a conference paper at ICLR 2023
Bo Liu, Ying Wei, Yu Zhang, and Qiang Yang. Deep neural networks for high dimension, low sample size data. In International Joint Conference on Artificial Intelligence Organization, pp. 2287-2293, 2017.
Yin Lou and Mikhail Obukhov. BDT: Gradient boosted decision tables for high accuracy and scoring efficiency. In Proceedings of the SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1893-1901, 2017.
Yang Lu, Yingying Fan, Jinchi Lv, and William Stafford Noble. DeepPINK: reproducible feature selection in deep neural networks. In Advances in Neural Information Processing Systems, volume 31, pp. -890-8700, 2018.
Donald W Marquardt and Ronald D Snee. Ridge regression in practice. The American Statistician, 29(1):3-20, 1975.
Lukas Meier, Sara Van De Geer, and Peter Bühlmann. The group LASSO for logistic regression. Journal of the Royal Statistical Society: Series (Statistical Methodology). 70(1):53-71, 2008.
Jörg Menche, Amitabh Sharma, Maksim Kitsak, Susan Dina Ghiassian, Mare Vidal, Joseph Loscalzo, and Albert-László Barabási. Uncovering disease-disease relationships through the incomplete interactome. Science, 1276622), 2015.
Soufiane MC Mourragui, Marco Loog, Daniel Vis, Kat Moore, Anna G Manjon, Mark A van de Wiel, Marcel JT Reinders, and Lodewyk FA Wessels. Predicting patient response with models trained on cell lines and patient-derived xenografts by nonlinear transfer learning. Proceedings of the National Academy of Sciences, 18849):e210668218 2021.
Rose Oughtred, Chris Stark, Bobby-Joe Breitkreutz, Jennifer Rust, Lorrie Boucher, Christie Chang, Nadine Kolas, Lara O' D'onnnll,, Genie Leung, Rochelle McAdam, al. The BioGRID interaction database: 2019 update. Nucleic Acids Research, 47(D1):D529-DSAA 2019.
Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. In International Conference on Learning Representations, 2020.
Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. CatBoost: unbiased boosting with categorical features. In Advances Neural Information Processing Systems, volume 31, pp. 6639-6649, 2018.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv:2204.06125. 2022.
Thomas Rolland, Murat Taşan, Benoit Charlotcaux, Samuel Pevzner, Quan Zhong, Nidhi Sahni, Song Yi, Irma Lemmens, Celia Fontanillo, Roberto Mosca, et al. A protcome-scale map of the human interactome network. Cell, 159(5):212211226 2014.
Adriana Romero, Pierre Lue Carrier, Akram Erraqabi, Tristan Sylvain, Alex Auvolat, Etienne Dejoie, Marc-André Legault, Marie-Pierre Dubé, Julie G Hussin, and Yoshua Bengio. Diet networks: thin parameters for fat genomics. In International Conference on Learning Representations, 2017.
Amélie Dricot, Ning Li, Gabriel Berriz, Francis D Gibbons, Matija Dreze, Nono Ayivi- Suivi-Guedehoussou,, al. Towards a proteome-scale map of the human protein-protein interaction network. Nature, 437 (7062):1173-1177) 2005.
Françors Rual, Kavitha Venkatesan, Tong Hao, Tomoko
Camilo Ruiz, Marinka Zitnik, and Jure Leskovec. Identification discase treatment mechanisms through the multiscale interactome. Nature Communications, 12(1):1-15, 2021.
Lukasz Salwinski, Christopher S Miller, Adam Smith, Frank K Pettit, James U Bowie, and David Eisenberg. The Database of Interacting Proteins: 2004 update. Nucleic Acids Research, 32 (suppL_)):D449 D451, 2004.
Jürgen Schmidhuber. Learning to control weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):111-139, 1992.
12

---

Under review as a conference paper at ICLR 2023
Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion, 81:84-90, 2022.
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. SAINT: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint S..iv:22066013444 2021.
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 1161-1170, 2019.
Damian Szklarezyk, Annika L. Gable, Katerina Nastou, David Lyon, Rebecca Kirsch, Sampo Pyysalo, Nadezhda T Doncheva, Mare Legeay, Tao Fang. Peer Bork, Lars Jensen, and Christian von Mering. The STRING database in 2021: customizable protein-protein networks, and functional characterization of user-uploaded gene/measureemett D612, November 2020. doi: 10. 00.1093/nar/gkaa11074
Nucleic Acids Research, 4965)): URL htthh:ts: /tt.////////////ooooo///////0000
nar/gkaa1074
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288 1996.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In International Conference on Machine Learning, pp. 2071-2080. PMLR, 2016.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE, Journal of Machine Learning Research, 9(1 (111):2579-266) 2008.
Laurens Van Der Maaten, Eric Postma, Jaap Van den Herik, al. Dimensionality reduction: comparative. Journal of Machine Learning Research, 10(66-71):13, 2009.
Kavitha Venkatesan, Jean-Francois Rual, Alexei Vazquez, Uirich Stelzl, Irma Lemmens, Tomoko Sonozane-Kishiawwe Tong Hao, Martina Zenkner, Xiaofeng Xin, Kwang-ll Goh, et al. empirical framework for binary interactome mapping. Nature Methods, 6(1):83-90, 2009.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference Learning Representations, 2019.
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and upplications. IEEE Transactions on Knowledge and Data Engineering, 29(12): 2724-2743, 2017.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating hyperplanes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28, pp. 1112-1119, 2014.
David S Wishart, Yannick D Feunang, An C Guo, Elvis J Lo, Ana Marcu, Jason R Grant, Tanvir Sajed, Daniel Johnson, Carin Li. Zinat Sayeeda, Nazanin Assempour, Ithayavani Iynkkaran, Yifeng Liu, Adam Maciejewski, Nicola Gale, Alex Wilson, Lucy Chin, Ryan Cummings, Diana Le, Allison Pon, Craig Knox, and Michael Wilson. DrugBank 5.0: a major update to the DrugBank database for 2018. Nucleic Acids Research, 46(D1):D1074-D1088 November 2017a. doi: 10. 10.1093/nar/gkx 1037. URL /ttp::pp//.oa....o.......///////
David S Wishart, Yannick D Feunang, An C Guo, Elvis Lo, Ana Marcu, Jason R Grant, Tanvir Sajed, Daniel Johnson, Carin Li, Zinat Sayeeda, et al. DrugBank 5.0: major update to the DrugBank database for 2018. Nucleic Acids Research, 46(D1):D1774-1100 2017b. doi: 10. N0.109//ar/gxxx0033
13

---

Under review as a conference paper at ICLR 2023
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes. Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 23965-23998. PMLR, 2022.
Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, and Yuval Kluger. Feature selection using stochastic gates. In International Conference on Machine Learning, pp. 10648-10659, PMLR, 2020.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Deng. Embedding entities and relations for learning and inference knowledge bases. In International Conference Learning Representations, 2015.
Wanjuan Yang, Jorge Soares, Patricia Greninger, Elena Edelman, Howard Lightfoot, Simon Forbes, Nidhi Bindal, Dave Beare, James Smith, Richard Thompson, al. Genomics of Drug Sensitivity in Cancer (GDSC): resource for therapeutic biomarker discovery in cancer cells. Nucleic Acids Research, 41(D1):D955 D961, 2012.
Yongxin Yang, Irene Garcia Morillo, and Timothy M Hospedales. Deep neural decision trees. In International Conference Machine Learning, Workshop on Human Interpretability in Machine Learning (WHI), 2018.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform badly for graph representation? In Advances in Neural Information Processing Systems, volume 34, pp. 28877-28888, 2021.
Haiyuan Yu, Leah Tardivo, Stanley Tam, Evan Weiner, Fana Gebreab, Changyu Fan, Nenad Svrzikapa, Tomoko Kishhkkww, Edward Rietman, Xinping Yang, al. Next-generation sequencing to generate interactome datasets. Nature Methods, 866):478-480, 2011.
14

---

Under review as a conference paper at ICLR 2023
Appendix
A EVALUATION PROTOCOL AND HYPERPARAMETER RANGES
To ensure fair comparison with baselines, we follow evaluation protocols outlined in tabular benchmarks (Grinsztajn et al. 2022, Gorishniy et al. 2021). We conduct random search with 500 configurations of every model (including PI.TOO) on every dataset across a broad range of hyperparameters. We base the hyperparameter ranges the ranges used in prior tabular learning benchmarks (Grinsztajn et al., 2022 Gorishniy et al. 202T) and the ranges mentioned in the original papers of the methods. Hyperparameter ranges for PLATO given in Supplementary Table Hyperparameter ranges for baseline methods are given in Supplementary Table
Module in PLATO
Hyperparameter
Range
Learning rate
LogUniform(Ie-44 5e-3)
General
Batch size
[16, 32, 64]
L2
0, LogUniform((e-5, le-2)
Embedding dimension
200
KG H
Embedding model
ComplEx
MP layers R
Message Passing (MP) Q
Beta
LogUniform(11-44 le-1)
Hidden dimension in A
Uniformlnt(16,, 512)
Layers R
Uniformint(2, 6)
Param Prediction
Hidden dimension in h
UniformInt( 16, 512)
Supplementary Table Hyperparameter ranges used for PLATO.
15

---

Under review as a conference paper at ICLR 2023
Model
Hyperparameter
Range
LASSO
L1
LogUniformí( (1-44, 10)
Ridge
L2
LogUniform((1E4 10)
n-cstimators
Uniformint( 1, ,,2000)
Max depth
Uniformlnt(3. 10)
Mín weight
LogUniform( (1--,,mmm1E811555)
Subsample
Uniform(O.5, 1)
Learning
LogUniform(1E2: (1-5,11)
Col sample by level
Uniform(0.5, 1)
XGBoost
Col sample by tree
Uniform(0.5, 1)
Gamma
0. LogUniformm 1E-8, 1E2)
Lamhda
0, LogUniformm 1E-8, 1F2)
Alpha
0. LogUniform( IE-8, 1E2) "ghtree" 100
Booster
Early-stopping-roume
Iterations
PCA
Number of PCA Components
Unifomlnn((,10000)
Hidden dimension
Uniformint( 10, 500)
Number of layers Activation
Uniformlat(1. 5)
[Tanh,, Relu, Sigmoid]
STG
Learning rate
LogUniformi( Ie-4, le-1)
Sigma
Uniform(0.001, 2)
Lambda
LogUniformm( (1eoo 10)
Number of layers
Uniformlnt(1, 8)
Hidden dimension
Uniformint(1. I. 512)
Dropout
0, Uniform((0.055)
MLP
Learning rate
LogUniformi (e- 1e-2)
1.2
0, LogUniformm1ees (1e-6, 1c-3)
Decision Steps Layer size
UniformInt(3, 10)
2, 4, 8, 16, 32, 64 Uniform| 1, 2]
Relaxation factor
Sparsity loss weight
LogUniform[Iess (e-66 1e-1]
TabNet
Decay rate
Uniform[0.4, 0.95]
Decay steps
100, 500, 2000
Learning rate
Uniform(le-3, 1e-2) 100
Iterations
Embedding dimension
8,8. 16, 32, 64, 128 UniformInt(1, 10)
Number of heads Number of attention blocks Attention dropout rate
Uniformint( 1, 12)
Uniform(O, 0.5)
Add norm dropout Transformation activation
Uniform(O, 0.5)
[Tanh, Relu, LeakyRel.U] LogUniform (e-foforrm11e6..... le-1)
Tabrrossfomer
L2
Learning rate
LogUniform(1e-4 (1-66, Ie-3) Uniform(O, 0.5)
FF dropout
FF hidden multiplier
1,2, 4. 5, 7, 8, 9, 10 [Tanh, Relu, LeakyReLU) Uniform(O, 0.5)
Out FF activation
Out FF dropout
Learning rate
LogUniformm (l 1)
Number of layers
UniformInt(1. 1, 10)
NODE
Number of trees Depth
Uniformint(2, 2048)
Uniformin(1.. 10)
Embedding choice
X" random Uniformlnt(1. 8)
Number of layers
Hidden dimension
UniformInt 1, 512)
Diet Network
Dropout
0, Uniform((0.0.5))
Learning rate
LogUniform le-5, le-2)
L.2
0, LogUniform( 1e-6, le-3)
Supplementary Table 2: Hyperparameter range for all baselines.
16

---

Under review as a conference paper at ICLR 2023
B GRAPH CLASSIFICATION APPROACHES
Graph classification models are not relevant for PLATO's setting. graph classification models, every input sample is graph with node attributes, and model must make a prediction for that graph. The PLATO problem setting breaks fundamental assumptions made by graph classification models, rendering them not applicable. First, graph classification models assume that different samples correspond to different graphs (Ying et al. [202T] [uu et al. [2020b a). However, in PLATO every sample corresponds to the exact same graph. There single background knowledge graph for all samples, and every sample has input features that correspond to the exact same nodes within the knowledge graph. Second, graph classification approaches typically assume that every node in an input graph has node attribute (Ying et al.) 2021 Hu al.) 2020b a). However, in PLATO only a small subset the nodes in the knowledge graph have measured feature values. Finally, graph classification approaches typically assume small graphs: the largest graph classification task in the Open Graph Benchmark has only 244 nodes (Hu et al.) 2020a). However in PLATO, the knowledge graph contains 108,447 and the smallest dataset has 12,932 features corresponding to nodes.
C RANK ORDERING OF METHODS FOR DATASETS WITH d > n
In Supplementary Table 3 show the rank order performance of all models all >> n datasets. We find that PLATO exhibits consistent and strong performance while the performance of the baselines depends the specific d >> dataset. For example, Tab Transfosfrrmrr the second best performing of all models on the MNSCLC dataset but the worst performing of all models on the PDAC and CH datasets. Similarly, XGBoost is the second best performing of all models seventh best performing of all models on BRCA. The baselines with the most stable performance LASSO and Ridge Regression which rank consistently between the second and fifth best of all models.
PDAC but only the
Supplementary Table For datasets with d n,, PLATO exhibits consistent and strong performance. By contrast, the performance the baselines varies with each dataset. For every dataset, the order performance from Table is shown. The best overall model is bold and the second best model underlined.
Dataset
MNSCLC
CM
PDAC
BRCA
CRC
CH
D/N
52.2
46.1
40.3
28.2
22.6
19.7
Classic Stat ML
Ridge
3
4
2
4
Dim. Reduct.
PCA
9
8
6
LASSO
2
5
Feat. Select.
STG
4
4
5
Decision Tree
XGBoost
11
6
2
7
2
Param. Pred.
Diet
10
10
9
10
MLP
8
NODE
7
9
3
7
Tabular DL
TabTransformer
8
11
10
TabNet
11
10
11
11
10
Ours
PLATO
1
1
17

---

Under review as a conference paper at ICLR 2023
Supplementary Table 4: For datasets with d n,, PLATO competitive with baselines. For XGBoost consistently the strongest baseline. For every dataset, the rank order of performance from Table is shown. The best overall model is in bold and the second best model underlined.
Dataset
ME
BC
SCLC
NSCLC
D/N
2.0
1.8
1.7
1.1
Classic Stat ML
Ridge
7
7
3
Dim. Reduct.
PCA
10
9
10
6
LASSO
4
4
7
Feat. Select.
STG
3
5
5
Decision Tree
XGBoost
2
2
2
Param. Pred.
Diet
11
10
11
MI.P
8
6
8
8
NODE
8
3
10
Tabular DL
TabTransformer
9
11
9
9
TabNet
5
6
4
Ours
PLATO
1
1
2
18

---

Under review as a conference paper at ICLR 2023
D CODE DETAILS
Code to run PLATO will be included as a supplementary file the final version of the manuscript.
E DATASET DETAILS
We compiled 6 datasets with > n and datasets with d ~ nt. In all datasets, a machine learning model must predict the response of cell or mouse drug. In the tabular data, every row corresponds to specific cell or mouse. Every column corresponds to gene name. Every value corresponds the amount of that gene in the tumor of the specific cell mouse. The label the response of the cell or mouse. All genes are nodes in the knowledge graph. In practice, the number of genes large for all tasks and the number of samples is comparatively small making the drug response setting appropriate for d >> n.
Dataset statistics, names, and sources are given below. Dataset pre- pre-proesssing followed standard process described in (Mourragui et al. (02T). Briefly, gene expression values underwent TMM normalization and log transformation (i.e. log(z + 1)). Values were made to have zero mean and unit standard deviation. As labels, used In-ic50 for datasets from (lorio et al.) 2016 Yang et al.) 2012 Garnett et al., 2012) and the minimum average percent tumor growth (i.e. "min-avg tumor-grave momorr-rowth"" > for datasets from (Gao al. 2015). For all datasets,
200-dimensional ComplEx Trouillon et al.) (016) embedding of the drug as the input feature vector. All datasets will be released in the final version of the manuscript.
Dataset
Name
Source
MNSCLC
Non-small cell lung carcinoma
Gao
2015
CM
Cutaneous melanoma
Gao et al.
2015
PDAC
Pancreatic ductal carcinoma
Gao et al.
2015
BRCA
Breast carcinoma
Gao
2013
CRC
Colorectal cancer
Gao
CH
Chondrosarcoma
Iorio et al.
2016
Yang et al.
2012
Garnett et al.
2012
ME
Melanoma
Ior1o el al.
2016
Yang et al.
2012
Garnett et al.
2012
BC
Breast carcinoma
[oroo
2016
Yang
2012
Garnett al.
2012
SCLC
Small lung carcinoma
Iorio al.
2016
Yang al.
2012
Garnett et al.
2012
NSCLC
Non-small cell lung carcinoma
[or10 el al
2016
Yang et al.
2012
Garnett et al.
2012
19

---

Under review as a conference paper at ICLR 2023
F KNOWLEDGE GRAPH DETAILS
As a unified knowledge backbone for the datasets, we compile a general biomedical knowledge graph from prior studies (et al. 2020; 2016; Kuhn al. 2015 Ruiz et al. 2021 Szklarczyk al. 2020
Wishart et al. 201 7 b). A schematic of the KG is in Supplementary Figure 2 A detailed breakdown
of relation types is in Supplementary Table
Our knowledge graph contains 108, 108,447 447 total nodes, including 7.975 drugs, 18,370 diseases, 11,447
phenotypes, 22,319 genes, 11.153 molecular functions, 28,748 biological processes, and 4,184 cellular components. Our knowledge graph contains 3,066,155 156 edges with 99 distinct relation types. All datasets include features which map to subset of nodes the knowledge graph, primarily genes and drugs. The remaining node types and their relationships serve as broader domain knowledge.
Edges between drug nodes and gene/protein nodes were derived from Drugbank (Wishart al. 2017b), Gao (Goo et al. 2015), and the Genomics of Drug Sensitivity in Cancer (Yang al. 2012 lorio et al.) 2016; Garnett et al. 2012). Edges between diseases and genes/proteins were derived from DisGeNet (Bauer-Mehren Mehren et al., 2010). Edges between discases and phenotypes were derived from the Human Phenotype Ontology (ct al., 2016). Edges between drugs and diseases were derived from the Multiscale Interactome (Ruiz et al. 2021). Edges between drugs and side effects were derived from SIDER (Kuhn et al.) 2015). Edges between genes/proteins and other genes/proteins were derived from BioGRID (Oughrre et 2019), (Rual et al. 2005), the Database of Interacting Proteins (Salwinski et al. 2004). (et al. (2020), (Menche et al. 2015). (Rolland et al. 2014). (Yu al. 1). (Venkatesan al. 2009), and STRING (Szklarczyk et al. 2020p. Finally, edges from genes/proteing to molecular functions, biological processes, and cellular components as well as edges between molecular functions, biological processes, and cellular components were derived from the Gene Ontology (Consortium) 2018). The full knowledge graph will be included as a supplementary file the final version the manuscript.
(a)
(c)
43
[1]
Legend
1,070 [4]
7,032
Node Type
Molecular Function
Biological Process 743)
Celluler Component (4:84)
(Nuu Mociee
[11.163
1,017 [4]
05,057 (4]
44,027 [d|
36.281 [4]
wlion typso)
Node Type 2
Gene/Protein #2,51
Ihuum Nodiol
2,308,395 ((]
29,
175
14,850
[1)
(b)
Nondes 108,447
Total
Drug (7.378)
$928
Disease (18,370)
195,102 (1)
Phanotypa/Side Effect (4-251)
Meteo 3,066,156
140.794 I1I
Supplementary Figure 2: Knowledge graph as a unified knowledge backbone. We constructed a knowledge graph as unified knowledge backbone across all datasets. (a) Legend. For each node type, the number of nodes is given in parentheses. Between node types, the number of edges and the number of relation types given, (b) Number of total nodes and edges across entire knowledge graph. (c) Visual schematic of knowledge graph across each node type.
20

---

Under review as a conference paper at ICLR 2023
Head type
Relation
Tail type
edges
Biosos
EndsDuring
Process
BiologicalProeess
HappensDuring
BiologicalProcess
BiologicalProcess
HasPart
BiologicalProcess
229
BiologicalProceen
IsA
BiologicalProcess
53015
BiologicalProeess
NegativelyRegulan
BiologicalProcess
2768
BiologicalProcess
PartOf
BiologicalProcess
5193
BiologicalProeess
PositivelyRegulane
BiooogicalProcess
2756
BiologicalProcess
Regulates
BiologicalProces.
3216
BiologicalProeess
OccursIn
Cellular CellularComponent
149
BiologicalProcess
HasPart
MoleculanFunction
173
BiologicalProeess
SrrativelyRegulaeee
MolecularFunction
269
BiologicalProcess
RositivelyRegulates
MolecularFunction
274
BiologicalProcess
Regulates
MolecularFunction
301
CellularComponent
HasPart
CellularComponent
179
CellularComponent
IsA
CeIlularComponent
4863
CellularComponent
PartOf
CellularComponent
1990
Disease
AlteredExpression
Gene
7157
Disease
Biomarker
Gene
107160
Disease
ChromosomallRarraaeeennt
Gene
162
Disease
GusionGene
Gene
166
Disease
Gene
15076
Disease
GermlineCausalMuaatton
Gene
4677
Disease
Modifying.Mutation
Gene
Disease
SomaticCause MauiiiCaaaaaatttiinn
Gene
130
Disease
BniceptibllityMMuttttii
Gene
441
Disease
Therapeutic
Gene
1793
Disease
Has
Phenotype
195402
Drug
Treats
Disease
5926
Drug
Carries
Gene
866
Drug
Enzymes
Gene
5382
Drug
Targets
Gene
19817
Drug
Transports
Gene
3124
Drug
Has
Phenotype
140764
Gene
Associates
BiologicalProcess
43857
Gene
NotAssociates
BiologicalProcess
470
Gene
Associates
CellularCompooeent
35306
Gene
Colocalizes
CellularComponent
914
Gene
NotAssociates
CellularComponens
160
Gene
NotColocalizes
CellularComponent
Gene
Acetylation
Gene
Gene
Activation
Gene
58502
Gene
AdpRibosylation
Gene
2
Gene
Ampylation
Gene
Gene
Association
Gene
18
Gene
Binary
Gene
56565
Gene
Binding
Gene
287641
Gene
Catalysis
Gene
344801
Gene
Cleavage
Gene
22
Gene
Complexes
Gene
62552
Gene
CovalentBinding
Gene
52
Gene
Deacetylatio
Gene
Gene
Demethylation
Gene
Gene
Dephosphorylatio
Gene
26
21

---

Under review as a conference paper at ICLR 2023
Gene
Deubiquitination
Gene
18
Gene
Directlnteraction
Gene
2904
Gene
DisulfideBond
Gene
Gene
DesageGrrowthDefeett Defect
Gene
Gene
Dosagel Lttaaily
Gene
112
Gene
DosageRescue
Gene
63
Gene
Enzymatic
Gene
Gene
Expression
Gene
188
Gene
GeneticInterference
Gene
32
Gene
Hydroxylation
Gene
26
Gene
Inhibition
Gene
20108
Gene
Kinase
Gene
11960
Gene
Literature
Gene
174162
Gene
Metabolic
Gene
10646
Gene
Methylation
Gene
25
Gene
NegativeGenetic
Gene
3449
Gene
Gene
Gene
PhenodypicEnhaaneement
Gene
209
Gene
PhenotypicSuppression
Gene
214
Gene
Phosphorylation
Gene
166
Gene
Phosphotransfer
Gene
Gene
PhysicalAssociatinn
Gene
824164
Gene
PositiveGenetic
Gene
2331
Gene
PoniTranslaionalModifitttionn
Gene
5306
Gene
ProteinCleavage
Gene
48
Gene
PutativeSelfhnterattio
Gene
Gene
Reaction
Gene
400658
Gene
Regulation
Gene
2650
Gene
Signaling
Gene
65412
Gene
SynthetieGrootthDefect
Gene
407
Gene
SyntheticLethalin
Gene
816
Gene
SyntheticRescue
Gene
91
Gene
Associates
MolecularFunction
35012
Gene
Contributes
MolecularFunction
596
Gene
NotAssociates
MolecularFunction
285
Gene
NotContributes
MolecularFunction
MolecularFunction
PartOf
Biologica/Proces,
1068
AnnccularFunction
Regulates
BiologicalProcess
MolecularFunction
OccursIn
CellularComponent
43
MolecularFunction
HasPart
MolecularFunction
204
MolecularFunction
IsA
MolecularFunction
13631
MolecularFunction
RegauivelyRegulates
MolecularFunctio
42
MolecularFunction
PartOf
MolecularFunction
Funttinn
RositivelyRegulates
MolecularFunction
27
MolecularFunction
Regulates
MolecularFunction
30
Phenotype
IsA
Phenotype
14650
Supplementary Table S: Knowledge graph relations between node types.
22


Reviews:
[{"review_id": "RHRxm1A1Dhi", "reviewer": "Reviewer_AKNa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors define PLATO, a model for underserved scenario of short and fat datasets (d>>n). PLATO works when we have an auxiliary knowledge graph describing the d features and the relations between them. Tabular datasets with d>>n are common in physical sciences and biology where data is collected through expensive experiments. \n\nPlato is an MLP where the first layer is trained by leveraging the knowledge graph as follows:\n1. A c-dimensional embedding M_j is learned for each feature j\n2. A message passing algorithm that uses the knowledge graph is used to learn another embedding Q_j for each feature j based on its neighbors. In each round the embedding of a feature is a weighted combination of its embedding in the previous round and all its neighbors in the previous round.\n3. A neural network outputs P the weights of the first layer.\n\nExperimental results are presented on multiple datasets and compared with multiplied methods (both statistical and deep-learning based). Some experimental results are presented to explain the motivation around using a message passing protocol and large knowledge graphs describing features even if not all are in dataset.", "summary_of_the_review": "Overall, the authors present a novel approach to supervised learning tasks on datasets where d>>n. Given the novelty of the approach, significance of the setting for biological and physical sciences and elegance of the solution, I vote accept. \n\nThe major weaknesses of the paper are largely open questions that can be addressed in a future iteration of the paper. The paper remains of interest to the community  at this iteration.", "strength_and_weaknesses": "Strengths\n1. Solves for an important set of applications that are underserved in deep learning focused research geared towards large datasets.\n2. Elegant way of incorporating prior information via a knowledge graph (KG). While not all applications have associated knowledge graphs, a sizeable set do. KG is a distillation of several studies, and to be able to incorporate effectively is significant. \n\n3. Experimental results are rigorous. On multiple datasets and compared with multiplied methods (both statistical and deep-learning based) Established protocols for tabular dataset are followed.\n4. The weights of the first layer of MLP are learned in 3 steps described above. An ablation study shows the importances of each step showing the added value of a message passing routine and also of adding knowledge about features not present in the datasets.\n5. Clearly describes architecture, datasets, and results.\n\nWeaknesses:\nAblation study is presented only on a single BRCA dataset. It would be valuable for readers to know that findings hold across datasets.\n\nThe intro and related work sections both spend significant ink on related work, including describing works like graph-based prediction that are only tangentially related. Instead, the intro can incorporate a summary of the contributions in the paper. Currently, the reader has to read all through methods and results to understand the significance of the contributions. Examples of items to include earlier are how message passing the graph structure of KG is used effectively to incorporate additional info. This is a significant result not sufficiently highlighted.\n\nThe current paper is not incomplete without these, but some suggestions for future work:\n\nStudy impact of missingness in KG. Is it possible to support datasets where some subset of features are not described in KG. The paper currently assumes every feature in dataset is a node in the KG. \n\nA study on model's sample complexity e.g., we broadly cast this problem as d>>n. e.g., can we afford to drop some rows from the dataset and the models achieve the same performance.. What is the point of diminishing returns. It would be interesting to see how few examples the model can get away with.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HNKFlvaq1N", "reviewer": "Reviewer_yX2X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission contributes a learning method for high-dimensional data that uses an associated knowledge graph between the features to create a lower-dimension representation. The methods pretrains a data-transformation mechanism on the knowledge graph, creates attention weights, and used message passing on the corresponding graph neural network to output an embedding of input data, which consistutes the intermediate layer of an MLP. The method is benchmarked on 6 biomedical datasets, with a dozen thousands features and hundreds samples, where it outperforms classic methods.\n", "summary_of_the_review": "There is an interesting idea, but this seems more like a computational biology paper than a general tabular learning but. The paper does not contribute understandible and insight on top of the graph regularization work.\n\n", "strength_and_weaknesses": "The idea of using a knowledge graph to build a graph neural net on the features is interesting.\n\nHowever, this paper seems more targetted as computational biology applications, and not tabular data in general. Indeed, outside computational biology, it seems quite unlikely to find the configuration of many columns all corresponding to entries of a knowledge base. Can this paper be applied outside computational biology? If so, it would be interesting to demonstrate it.\n\n\nDid the author compare to classic graph-regularized models? There is a literature that is more than 10 years old on the topic, including in computational biologie.\n\nI do not understand where things such as the line 3 of algorithm 1 come from. I just do not understand how was the algorithm designed. By trial and error?\n\nHow are the error bars computed on the tables? Given the small number of n, I would expect much larger error bars.\n\nPCA is not a supervised method. How can it be used in the experimental tables?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "96EG3GFvLq1", "reviewer": "Reviewer_rHdL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To deal with the problem when tabular data has high dimensionality, the paper introduces a knowledge graph to help determine the weights of an MLP. The proposed PLATO method achieves good results on various datasets.\n", "summary_of_the_review": "The main idea of the paper is reasonable. An auxiliary knowledge graph will introduce more information especially since there are many features. The method is clear and the results are good. Please check the weakness part for some questions.\n", "strength_and_weaknesses": "To deal with the case that the dimensionality of the data is very larger than the sample size, the authors introduce an additional knowledge graph as a prior of the feature relationship. Then the weights of the MLP are better determined.\n\nThere are some suggestions for the paper:\n1. Why the weights of the first layer of the MLP is more important than the others? \n2. Do the results depend on the quality of the knowledge graph? How to compare with other methods in a fair manner since an auxiliary knowledge graph is introduced.\n3. Pre-train the node embedding in Eq. 2 is not with a self-supervised node embedding method is not clear enough. How will the pretraining method influences the final results?\n4. The layer-wise predicted embeddings are concatenated in line 9 in the algorithm. It works like an ensemble. How much does the concatenation step improve in the whole method?\n5. Please consider comparing more tabular data methods, such as FT-Transformer based on some dimensionality reduction methods.\n6. It seems the column names are required. Does it limit the field of the method?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6V-MMIM0QGI", "reviewer": "Reviewer_Ddzi", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work attempts to address the curse of high dimensionality problem for the tabular data. The authors build their model, PLATO, on the well formed insight that auxiliary KG about the feature connections can improve performance as the sparsity induced in the model helps with the generalizations in the cases where d>>n. Their method predicts the first layer weights of the MLP and then the remaining layers of the MLP are trained using the loss function. They evaluate their method on different datasets and report improved performance.", "summary_of_the_review": "I have some major concerns about this methodology (pts 2&3 in comments). The rationale given by the authors seems to be hand-wavy and not grounded. I hereby request the authors to develop a mathematically rigorous approach to support their claims. ", "strength_and_weaknesses": "\nPros:\n1. Figure 1 explains their methodology in a good and succinct manner. \n\nComments:\n1. It will be helpful to cover some related graph recovery methods to get auxiliary graphs for more adoption. \n2. (I) In sec 3.2, the rationale given to share the parameters of the first layer does not seem principled to me.  “Typically this intuition is missed since T learns the parameters Θ [1] j and Θ [1] k associated with two features j and k independently by gradient backpropagation.” The features do have common parameters in the subsequent layers through which their commonalities can be modelled. \n3. This methodology seems to learn a projection of the input data to another space, say R^h (here referred to as layer 1 of the MLP). The method comes up with `d’ vectors in R^h space that are put together as a matrix to obtain the $\\Theta[1]$. This is done by running message-passing on the KG which takes individual feature inputs as one-hot vectors to obtain a vector in R^h space.\n4. How does it perform for out-of-distribution samples? \n5. Please share details about any ablation study done, which empirically shows that having the first layer obtained from this method works better than just training a simple MLP. What happens when there are more than 3-4 layers in MLP and what if there are no learnable MLP layer and just this direct projection layer? \n\nI have some major concerns about this methodology (pts 2&3 in comments). The rationale given by the authors seems to be hand-wavy and not grounded. I hereby request the authors to develop a mathematically rigorous approach to support their claims. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject