Title: High dimensional, tabular deep learning with an auxiliary knowledge graph
Venue: NeurIPS 2023
Accepted: yes
Keywords: Tabular Data

Deep Learning

Knowledge Graph

Regularization
Abstract: Machine learning models exhibit strong performance on datasets with abundant labeled samples. However, for tabular datasets with extremely high $d$-dimensional features but limited $n$ samples (i.e. $d \gg n$), machine learning models struggle to achieve strong performance due to the risk of overfitting. Here, our key insight is that there is often abundant, auxiliary domain information describing input features which can be structured as a heterogeneous knowledge graph (KG). We propose PLATO, a method that achieves strong performance on tabular data with $d \gg n$ by using an auxiliary KG describing input features to regularize a multilayer perceptron (MLP). In PLATO, each input feature corresponds to a node in the auxiliary KG. In the MLP’s first layer, each input feature also corresponds to a weight vector. PLATO is based on the inductive bias that two input features corresponding to similar nodes in the auxiliary KG should have similar weight vectors in the MLP's first layer. PLATO captures this inductive bias by inferring the weight vector for each input feature from its corresponding node in the KG via a trainable message-passing function. Across 6 $d \gg n$ datasets, PLATO outperforms 13 state-of-the-art baselines by up to 10.19%.

High dimensional, tabular deep learning with an auxiliary knowledge graph

Abstract

Machine learning models exhibit strong performance on datasets with abundant labeled samples. However, for tabular datasets with extremely high d -dimensional features but limited n samples (i.e. d ≫ n ), machine learning models struggle to achieve strong performance due to the risk of overfitting. Here, our key insight is that there is often abundant, auxiliary domain information describing input features which can be structured as a heterogeneous knowledge graph (KG). We propose PLATO, a method that achieves strong performance on tabular data with d ≫ n by using an auxiliary KG describing input features to regularize a multilayer perceptron (MLP). In PLATO, each input feature corresponds to a node in the auxiliary KG. In the MLP's first layer, each input feature also corresponds to a weight vector. PLATO is based on the inductive bias that two input features corresponding to similar nodes in the auxiliary KG should have similar weight vectors in the MLP's first layer. PLATO captures this inductive bias by inferring the weight vector for each input feature from its corresponding node in the KG via a trainable message-passing function. Across 6 d ≫ n datasets, PLATO outperforms 13 state-of-the-art baselines by up to 10.19%.

1 Introduction

Machine learning models have reached state-of-the-art performance in domains with abundant labeled data like computer vision [76, 10] and natural language processing [70, 12, 52]. However, for tabular datasets in which the number d of features vastly exceeds the number n of samples, machine learning models struggle to achieve strong performance [24, 41]. Crucially, many tabular datasets from scientific domains [21, 30, 79, 17, 16, 33] have high-dimensional features but limited labeled samples due to the high time and labor costs of experiments. For these and other tabular datasets with d ≫ n , the performance of machine learning models is currently limited.

The key challenge for machine learning models when d ≫ n is the risk of overfitting. Indeed, deep models can have a large number of trainable weights, yet training is limited by the comparatively small number of labeled samples. As a result, tabular deep learning approaches so far have focused on data-rich regimes with far more samples than features ( n ≫ d ) [19, 18, 59]. In the low-data regime with far more features than samples ( d ≫ n ), the dominant approaches for single tabular datasets are still statistical methods [24]. These statistical methods reduce the dimensionality of the input space [1, 41, 66, 67], select features [64, 8, 14, 46], impose regularization penalties on parameter magnitudes [45], or use ensembles of weak tree-based models [15, 7, 34, 42, 51].

Here, we present a novel problem setting and framework that enables tabular deep learning when d ≫ n (Figure 1). Our key insight is that there is often abundant, auxiliary domain information

(e)

X ∈ Rnxd

(c)

Sample i

0.1

0.3

3.7

-0.9

1.3

-0.4

0.2

2.4

-1.1

1.0

1.3

0.4-2.10.0

1.1

-0.3

Feature j

Knowledge Graph G

(q)

(d)

B

First layer of a MLP

[rjo

O!l ∈ Rh

!X

F

Feature j

P!X

describing input features which can be structured as a heterogeneous knowledge graph (KG). We propose a novel problem setting in which each input feature of a tabular dataset corresponds to a node in an auxiliary KG (Figure 1a). To represent diverse domain information describing the input features, the KG contains feature and non-feature nodes as well as multiple node and edge types. For example, consider a tabular medical dataset in which each row is a cancer patient, each column is a gene, and each value is the amount of a gene in the patient's tumor. For this tabular dataset, there exists an auxiliary KG with each gene ( i.e. input feature) as a node. Each gene node has edges to other gene nodes ( i.e. other feature nodes) with diverse edge types like 'activates' or 'inhibits.' Each gene node also has edges to other nodes ( i.e. non-feature nodes) representing the gene's function in the body like 'heart rate'. Finally, the function nodes ( i.e. non-feature nodes) have edges to each other representing their anatomical relationships like 'heart rate'-'part of'-'cardiac system'. Note that the KG does not capture the relationships between input data samples but instead captures the relationships between input features and other domain information.

Within our novel problem setting, we propose PLATO, a method that enables deep learning for tabular data with d ≫ n by using an auxiliary KG describing input features (Figure 1). PLATO achieves strong performance by using the auxiliary KG to regularize a multilayer perceptron (MLP). In PLATO, each input feature corresponds to a node in the auxiliary KG (Figure 1a). In the first layer of the MLP, each input feature also corresponds to a weight vector such that the weight vectors of all features collectively compose the weight matrix (Figure 1b). PLATO is based on the inductive bias that two input features which correspond to similar nodes in the KG should have similar weight vectors in the first layer of the MLP. PLATO captures this inductive bias by inferring the weight vector for a feature from its corresponding node in the auxiliary KG with a trainable message-passing function (Figure 1c,d). Inferring the weights in the MLP's first layer also leads to a drastic reduction in the number of trainable weights, since most weights in a MLP are usually in the first layer when d ≫ n .

We exhibit PLATO's performance on 6 d ≫ n tabular datasets with 13 state-of-the-art baselines spanning dimensionality reduction, feature selection, statistical models, graph regularization, weightinference, and tabular deep learning. Following a rigorous evaluation protocol from the tabular deep learning literature [19, 18], PLATO outperforms the prior state-of-the-art on all 6 datasets by up to 10.19%. Ablation studies demonstrate the importance of PLATO's trainable message-passing, the importance of non-feature nodes in the KG, and PLATO's robustness to missing edges in the KG. Ultimately, PLATO enables deep learning for tabular data with d ≫ n by using an auxiliary KG describing the input features.

2 Related Work

Tabular deep learning methods. In contrast to PLATO's setting, tabular deep learning methods have primarily been developed for settings with far more samples than features ( i.e. n ≫ d ). Indeed, recent tabular deep learning benchmarks ignore datasets with a large number of features and a small number of samples [19, 18, 59]. In the n ≫ d setting, various categories of deep tabular models exist. First, decision tree models like NODE [50] make decision trees differentiable to enable gradient-based optimization [25, 36, 80]. Second, multilayer perceptrons (MLPs) apply sequential, non-linear transformations to input features [32, 31]. Third, tabular transformer architectures use an attention mechanism to select and learn interactions among features. Examples include TabNet [3], TabTransformer [29], FT-Transformer [18], TabPFN [26], SAINT [61], Non-Parametric Transformers [37], and AutoInt [62]. Finally, although PLATO focuses on single tabular datasets, transfer learning architectures can learn across multiple tabular datasets [39, 84, 73]. Ultimately, we compare PLATO to several benchmarked, state-of-the-art models for single, tabular datasets [19, 18, 59].

d ≫ n methods. For PLATO's setting in which d ≫ n , various tabular machine learning approaches exist [24]. First, dimensionality reduction techniques like PCA [1] aim to reduce the dimensionality of the input data while preserving as much of the the variance in the data as possible [41, 66, 67]. Second, feature selection approaches select a parsimonious set of features, leading to a smaller feature space. Feature selection approaches include LASSO [64] and its variants [8, 14, 46]. For feature selection with deep learning, Stochastic Gates [77] are among the best performing of many variants [4, 43]. Finally, tree-based models like XGBoost learn ensembles of weak decision trees models to make an overall prediction [15, 7, 34, 51].

Weight inference. Using one network to infer the weights of another has been studied extensively [11, 58, 6]. For example, [22] infers the weights in all layers of a sequential model ( i.e. RNN, LSTM) by using information about the weights' structure. Diet Networks [54] infer weights by hand-crafting prior information about the input features or using random projections. By contrast, PLATO infers the weights in a MLP from prior information describing the input features in an auxiliary KG. PLATO's weight inference uniquely captures the inductive bias that two input features corresponding to similar nodes in a KG should have similar corresponding weight vectors in the first layer of a MLP (Figure 1).

Graph regularization. Graph regularization approaches regularize the weights of a linear model based on a simple graph between input features. The graph is typically constructed from the tabular data based on covariance relationships. Approaches then add a regularization penalty to the loss function which forces the weights of the linear model to vary smoothly over the corresponding feature nodes in the graph. State-of-the-art methods include GraphNet [20] and Network-Constrained LASSO [40] which are based on a Laplacian regularization [60, 2] as well as Network LASSO [23] which generalizes the Group LASSO [83] to a network setting. PLATO differs from graph regularization approaches in two key ways. First, PLATO's KG includes both feature and non-feature nodes and multiple edge types, thereby modeling diverse, prior domain information that is missing in graph regularization approaches. Second, PLATO infers the weights of a deep non-linear model ( i.e. a MLP) rather than adding a regularization penalty to a loss, representing a distinct regularization mechanism.

Knowledge graph methods. Existing KG approaches are designed for tasks directly on the graph like link prediction or node classification [71, 65, 72, 78, 13]. By contrast, PLATO does not make any predictions on the KG. Instead, PLATO makes predictions on a separate, tabular dataset by using the KG as a prior. Graph classification methods also do not apply (Appendix B).

3 PLATO

PLATO is a machine learning method for tabular datasets with d ≫ n and an auxiliary knowledge graph (KG) with input features as nodes (Section 3.1). PLATO's key insight is that there often exists abundant domain information describing input features which can be structured as an auxiliary KG G (Figure 1a). PLATO uses the auxiliary KG to regularize a multilayer perceptron (MLP) and achieve strong performance on tabular data when d ≫ n .

3.1 Problem setting

Consider a tabular dataset X ∈ R n × d with labels y ∈ R n and far more d features than n samples such that d ≫ n . The goal is to train a model F to predict labels ˆ y from the input X . PLATO assumes the existence of an auxiliary knowledge graph G = ( V, E ) with | V | nodes and | E | edges such that each input feature j corresponds to a node in G . Formally, ∀ j ∈ { 1 , . . . , d } , ∃ v ∈ V s.t. j ↦→ v , as shown in Figure 1a. G also contains additional nodes which represent broader knowledge describing the domain. The edges in G are (head node, relation type, tail node) triplets.

3.2 PLATO's inductive bias

In PLATO, each input feature j corresponds to a node in the auxiliary KG (Figure 1a). In the first layer of a MLP with h hidden units, each input feature j also corresponds to a weight vector Θ [1] j ∈ R h such that the weight vectors of all features collectively compose the weight matrix Θ [1] ∈ R d × h (Figure 1b). PLATO is based on the inductive bias that two input features j and k which correspond to similar nodes in the KG should have similar weight vectors Θ [1] j and Θ [1] k in the first layer of the MLP. PLATO captures this inductive bias by inferring the weight vector for a feature from its corresponding node in the auxiliary KG with a trainable message-passing function (Figure 1c,d).

3.3 PLATO overview

PLATO has four key steps. First, PLATO uses a self-supervised objective on the auxiliary KG to pretrain an embedding for each input feature (Section 3.4). Second, PLATO updates each feature embedding with a trainable message-passing function that is trained on the supervised loss objective for the tabular data (Section 3.5, Figure 1c). Third, PLATO infers the weights in the first layer of the MLP from the feature embeddings with a small neural network that is shared across input features (Section 3.6, Figure 1d). Finally, the MLP predicts the label for the input sample.

3.4 Pretraining feature embeddings with self-supervision on the knowledge graph

First, PLATO learns general prior information about each input feature j from the auxiliary KG G . PLATO represents the general prior information about each input feature j as a low-dimensional embedding M j ∈ R c . Since each input feature j corresponds to a node in G , PLATO can learn M j by learning an embedding for the corresponding feature node in G . Any self-supervised node embedding method on G can be used within PLATO's framework.

Formal notation. Formally, PLATO uses self-supervision on G to pretrain an embedding for each input feature according to

M ∈ R d × c is the matrix of all feature embeddings. H is a self-supervised node embedding method. We refer to Eq. (1) as pretraining since only the auxiliary KG G is used but the tabular data X , y is ignored. After pretraining, the feature embeddings M are fixed.

For H , we choose ComplEx as it is a prominent and highly scalable KG node embedding method [65]. ComplEx uses a self-supervised objective which learns an embedding for each node in G by classifying whether a proposed edge exists in G . ComplEx's proposed edges include both feature nodes and other nodes in G , thereby integrating prior information about the input features and the broader domain. We also test KG embedding methods DistMult [78] and TransE [72] in Appendix C.

3.5 Updating feature embeddings with a message-passing function trained on tabular data

PLATO next updates each feature embedding with a trainable message-passing function that is trained on the supervised loss for the tabular data (Figure 1c). During message-passing, PLATO updates the embedding of each input feature to be a weighted aggregation of it's neighbors' embeddings.

Formal notation. Formally, PLATO uses a message-passing function Q on the KG to update each pre-trained feature embedding M j ∈ R c to feature embedding Q j ∈ R c according to

As input, the message-passing function considers the pre-trained feature embeddings M , the knowledge graph G , and the sample value X i . Q uses an attention mechanism which considers the sample value X i . The only trainable weights in Q are in the attention mechanism and are Π .

The message passing network Q . Let Q [ r ] j be the embedding of input feature j after round r ∈ { 1 , ..., R } of message passing. For each input feature j , Q first initializes the updated feature embedding to the pretrained feature embedding.

Q then conducts R rounds of message passing. In each round of message passing, the feature embedding Q [ r ] j is updated from the feature embedding of each neighbor k in the prior round Q [ r -1] k and its own feature embedding in the prior round Q [ r -1] j . The 'message' being passed is the embedding of each feature from the prior round.

σ is an optional nonlinearity. N j are the neighbors of feature node j in G .

During message-passing, Q uses two scalar values β ∈ R and α ijk ∈ R to control the weights of messages. First, Q uses hyperparameter β ∈ R to control the weight of the messages aggregated from the feature node's neighbors vs. from the feature node itself. Second, Q calculates an attention coefficient α ijk ∈ R to allow distinct nodes in the same neighborhood to have distinct weights. The coefficient α ijk specifies the weight of the message between feature j and neighbor k for sample i .

After R rounds of message-passing, the updated feature embeddings Q j are set.

The attention coefficient. PLATO's attention coefficient α ijk is inspired by [68] in which node attributes are used to calculate the weight of a message between neighboring nodes. For a sample i in PLATO, the node attributes for features j and k are their sample values X ij ∈ R and X ik ∈ R . PLATO thus uses the sample values X ij and X ik to calculate the attention coefficient. The attention coefficient e ijk indicates the importance of node j to node k for sample i .

A is a shallow neural network parameterized by Π that is shared across samples and features. The number of trainable weights in Π is small since the input of A is R 2 and the output of A is a scalar R .

To make the attention coefficients comparable across different nodes, PLATO normalizes the attention coefficients with a softmax function across the neighbors N j of node j .

Algorithm 1: The PLATO Algorithm.

Input: A data sample X i ∈ R d , a knowledge graph G containing each input feature in X as a R d × c

Output: A predicted label ˆ y i ∈ R

node, a matrix of input feature embeddings M ∈ pre-trained over G . .

1 Use a trainable message-passing function Q to update the pre-trained feature embeddings: Q = Q ( M , G, X i ; Π ) , Q j ∈ R c , Q ∈ R d × c

2 Infer the weight vector in the first layer of a MLP that corresponds to an input feature j with a neural network B :

3 Repeat to infer the weight vectors corresponding to all input features by sharing the neural network B : [1] d × h

ˆ Θ

∈

R

4 Concatenate the first layer inferred weights with the trainable weights in the rest of the MLP layers:

5 Predict the label with a MLP F that is parameterized by ˆ Θ

Trainable weights: Π , Φ , Θ [2] , . . . , Θ [ L ] .

3.6 Inferring the first layer of weights in F from the updated feature embeddings

Finally, PLATO infers the weights in the first layer of a MLP F from the updated feature embeddings (Figure 1d). In the first layer of a MLP with h hidden units, each input feature j corresponds to a weight vector Θ [1] j ∈ R h (Figure 1b). The weight matrix in the first layer of the MLP, Θ [1] ∈ R d × h , is simply the concatenation of d weight vectors, one corresponding to each input feature. For each input feature j , PLATO infers the weight vector ˆ Θ [1] j ∈ R h from the feature embedding Q j ∈ R c by using a shallow neural network shared across input features. Input features with similar feature embeddings will produce similar weight vectors. Thus, PLATO captures the inductive bias that input features corresponding to similar nodes in the KG should have similar corresponding weight vectors in the MLP's first layer.

Formal notation. PLATO infers the weight vector associated with each input feature j in the first layer of F with

B is a shallow neural network with trainable weights Φ . Q j is the updated feature embedding of j which is conditioned on the specific input sample X i since the input sample is used as an input in its calculation (Section 3.5, Equation 2). Φ are the weights of B . B and its weights Φ are shared across each feature j ∈ { 1 , . . . , d } .

PLATO drastically reduces the number of trainable weights compared to a standard MLP. The sharing of B and Φ across all input features drastically reduces the number of trainable weights compared to a standard MLP. For a high-dimensional tabular dataset ( i.e. d ≫ n ), a standard MLP T with h hidden units has a large number of trainable weights in the first layer since Θ [1] ∈ R d × h . A standard MLP T must learn all dh of these trainable weights by backpropagation. By contrast, B uses a shared set of trainable weights Φ to infer ˆ Θ j from Q j for every j ∈ { 1 , . . . , d } . The number of trainable weights in Φ is small compared to dh since B need only transform every Q j ∈ R c to ˆ Θ [ 1 ] ∈ R h . Thus, | Φ | = ch (assuming B is a single layer neural network). c , the dimensionality of the feature embedding, is much less than d the number of input features. As a result, | Φ | = ch ≪ dh and PLATO drastically reduces the number of trainable weights in the first layer of a MLP.

3.7 The PLATO algorithm

PLATO is outlined in Algorithm 1.

4 Experiments

We evaluate PLATO against 13 baselines on 10 tabular datasets (6 with d ≫ n , 4 with d ∼ n ).

Datasets. We use 6 tabular d ≫ n datasets, 4 d ∼ n datasets [16, 17, 30, 79], and a KG from prior studies [44, 35, 38, 56, 63, 74, 75] (Appendix G, H). The KG contains 108,447 nodes, 3,066,156 edges, and 99 relation types. All datasets include features which map to a subset of knowledge graph nodes. Code, data, and the KG are available at https://github.com/snap-stanford/plato .

Baselines. We compare PLATO to 13 state-of-the art statistical and deep baselines. We consider regularization with Ridge Regression [45], dimensionality reduction with PCA [1] followed by linear regression, feature selection with LASSO [64], deep feature selection with Stochastic Gates [77], and gradient boosted decision trees with XGBoost [7]. We consider tabular deep learning with a standard MLP, self-attention-based methods with TabTransformer [29] and TabNet [3], differentiable decision trees with NODE [50], and weight inference with Diet Networks [54]. We also attempted FTTransformer [18], but it experienced out of memory issues on all datasets due to the large number of features. Finally, we consider graph regularization methods which also have access to the knowledge graph including GraphNet [20], NC LASSO [40], and Network LASSO [23] (Appendix E).

Fair Comparison of PLATO with Baselines. To ensure a fair comparison with baselines, we follow evaluation protocols in recent tabular benchmarks [19, 18]. We conduct a random search with 500 configurations of every model (including PLATO) on every dataset across a broad range of hyperparameters (Appendix A). We split data with a 60/20/20 training, validation, test split. All results are computed across 3 data splits and 3 runs of each model in each data split. We report the mean and standard deviation of the Pearson correlation (PearsonR) between y and ˆ y across runs and splits on the test set. Each model is run on a GeForce RTX 2080 TI GPU.

4.1 Results

PLATO outperforms statistical and deep baselines when d ≫ n . PLATO outperforms all baselines across all 6 datasets with d ≫ n (Table 1). PLATO achieves the largest improvement on the PDAC dataset, improving by 10.19% vs. XGBoost, the best baseline for PDAC (0.400 vs. 0.363). While PLATO achieves the strongest performance across all 6 datasets, the best performing baseline varies across datasets. Ridge Regression is the strongest baseline for BRCA, LASSO for CM and CRC, XGBoost for PDAC and CH, and TabTransformer for MNSCLC. The remaining baselines are not the strongest baseline for any dataset. We also find that the performance of a specific baseline depends largely on the dataset. TabTransformer, for example, is the best baseline for the MNSCLC dataset but the worst baseline for the CH dataset. The rank order of all models on all datasets is Appendix D.

PLATO's performance depends on updating feature embeddings with a trainable messagepassing function. PLATO infers the weights ˆ Θ [ 1 ] in the first layer of a MLP F by using feature embeddings which contain prior information about the input features. PLATO first pretrains general feature embeddings M ∈ R d × c . PLATO then updates the feature embeddings to Q ∈ R d × c with a trainable message-passing function. We test whether updating the feature embeddings based on the trainable message-passing function is necessary by evaluating PLATO's performance on the BRCA dataset in three configurations (Table 2). The default configuration uses the updated feature embeddings Q generated by the message-passing function to infer ˆ Θ [1] according to ˆ Θ [ 1 ] j = B ( Q j | X i ) . The second configuration uses the general feature embeddings M instead of Q to infer ˆ Θ [1] according to ˆ Θ [ 1 ] j = B ( M j ) . The third configuration does not use feature embeddings and thus ablates to a standard MLP. Using general feature embeddings M improves over not using feature embeddings at all (0.522 vs. 0.240). Using feature embeddings Q that are generated by the trainable message-passing function further improves performance (0.583 vs. 0.522). Thus, updating the feature embeddings to Q based on the trainable message-passing function is key to PLATO's performance.

PLATO's performance depends on both feature nodes and broader knowledge nodes in the auxiliary KG. PLATO relies on an auxiliary KG G which contains information describing input features and the broader domain. Information describing input features is represented as feature nodes while information describing the broader domain is represented as other nodes in G (Methods 3.1). To test the relative importance of the feature information in G vs. the broader domain information, we measured the performance of PLATO on the BRCA dataset in two KG configurations: PLATO with the full KG ( i.e. both the feature nodes and the broader domain nodes) and PLATO with a 'feature-only KG' ( i.e. an induced subgraph on only the feature nodes) (Table 3). We also compare to a 'No KG' configuration in which PLATO does not have access to the KG. Without auxiliary information describing the input features or the broader domain, PLATO is ablated to a standard MLP.

We find that both the feature nodes and the broader knowledge nodes are important for PLATO's performance. Using the 'feature-only KG' configuration of PLATO improves performance vs the 'no KG' configuration (0.539 vs 0.240). Using the 'full KG' configuration further improves performance vs the 'feature-only KG' configuration (0.583 vs 0.539). PLATO's performance thus relies on both the feature information and the broader domain information in the KG.

PLATO's performance with an incomplete knowledge graph. All KGs are incomplete since there is undiscovered knowledge. PLATO thus uses low-dimensional embeddings from KG embedding approaches [72, 78, 65] which are designed to account for missing information, enabling predictive performance even with missing edges. We conduct an ablation study to assess PLATO's robustness to missing edges in the KG. We randomly remove edges from the KG and measure PLATO's performance on the BRCA dataset. We observe that with only 50% of the KG's edges, PLATO still has 71% of the performance as PLATO with 100% of the KG's edges ( 0 . 412 vs. 0 . 583 ) (Table 4).

The importance of MLP layers 2 , . . . , L , the layers with trainable weights, for PLATO. PLATO is a MLP in which the weights in the first layer are inferred from the knowledge graph (KG) but the weights in the remaining layers 2 , . . . , L are trained normally. We conduct an ablation study to determine whether MLP layers 2 , ..., L are necessary for PLATO's performance or whether the first layer of inferred weights are sufficient. Note that a single layer of inferred weights in PLATO is equivalent to a linear regression in which the weights are inferred from the KG. We thus compare PLATO to PLATO-LR, a linear regression in which the weights are inferred from the KG (Table 5). PLATO 's standard configuration outperforms PLATO-LR on the BRCA dataset ( 0 . 583 vs. 0 . 550 ). Therefore, layers 2 , . . . , L of the MLP are important for PLATO's performance.

For datasets with d ∼ n , PLATO is competitive with baselines. Finally, we test PLATO's performance for datasets with d ∼ n . We test 4 datasets with d ∼ n ranging from d n = 1 . 1 to 2 . 0 (Table 6). We find that on 4 datasets with d ∼ n , PLATO is competitive with the best baseline, XGBoost, but does not improve performance substantially. PLATO's stronger performance for datasets with d ≫ n than for datasets with d ∼ n is justified. PLATO's key idea is to include auxiliary information describing the input features. Auxiliary information is likely to help performance the most in settings with the least labeled data ( i.e. d ≫ n ). When d ∼ n , auxiliary information is less helpful since the tabular dataset may already have enough information to train a strong predictive model. We further find that XGBoost is the strongest baseline for all datasets with d ∼ n , in contrast to XGBoost's varied performance on the datasets with d ≫ n (Table 1).

5 Discussion

PLATO achieves strong performance on tabular data when d ≫ n by using an auxiliary KG describing input features to regularize a multilayer perceptron (MLP) . Across 6 datasets, PLATO outperforms 13 state-of-the-art baselines by up to 10 . 19% . Ablations demonstrate the importance of PLATO's trainable message-passing function, of including nodes in the KG that don't represent input features but instead represent domain information, and of the layers in the MLP whose weights are trained directly rather than inferred. We also test PLATO's robustness to missing information in the KG. PLATO has several limitations. First, PLATO matches but does not improve the performance of baselines for high-dimensional datasets with more samples ( i.e. d ∼ n ). Second, PLATO depends on the existence of an auxiliary KG of domain information. Overall, PLATO enables tabular deep learning when d ≫ n by using an auxiliary KG of domain information describing input features.

References

Hervé Abdi and Lynne J Williams. Principal Component Analysis. Wiley Interdisciplinary Reviews: Computational Statistics , 2(4):433-459, 2010.

Rie Ando and Tong Zhang. Learning on graph with Laplacian regularization. In Advances in Neural Information Processing Systems , volume 19, 2006.

Sercan Ö Arik and Tomas Pfister. TabNet: Attentive interpretable tabular learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 6679-6687, 2021.

Muhammed Fatih Balın, Abubakar Abid, and James Zou. Concrete autoencoders: Differentiable feature selection and reconstruction. In International Conference on Machine Learning , pages 444-453. Proceedings of Machine Learning Research, 2019.

Anna Bauer-Mehren, Michael Rautschka, Ferran Sanz, and Laura I Furlong. DisGeNET: a cytoscape plugin to visualize, integrate, search and analyze gene-disease networks. Bioinformatics , 26(22):2924-2926, 2010.

Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning a synaptic learning rule. In International Joint Conference on Neural Networks , volume 2, pages 969-975. IEEE, 1991.

Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the SIGKDD Conference on Knowledge Discovery and Data Mining , pages 785-794, 2016.

Héctor Climente-González, Chloé-Agathe Azencott, Samuel Kaski, and Makoto Yamada. Block HSIC lasso: model-free biomarker detection for ultra-high dimensional data. Bioinformatics , 35(14):i427-i435, 2019.

Gene Ontology Consortium. The Gene Ontology resource: 20 years and still GOing strong. Nucleic Acids Research , 47(D1):D330-D338, 2018.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255. IEEE, 2009.

Misha Denil, Babak Shakibi, Laurent Dinh, Marc'Aurelio Ranzato, and Nando De Freitas. Predicting parameters in deep learning. In Advances in Neural Information Processing Systems , volume 26, pages 2148-2156, 2013.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics , 2019.

Claudia d'Amato, Nicola Flavio Quatraro, and Nicola Fanizzi. Injecting background knowledge into embedding models for predictive tasks on knowledge graphs. In European Semantic Web Conference , pages 441-457. Springer, 2021.

Tobias Freidling, Benjamin Poignard, Héctor Climente-González, and Makoto Yamada. Postselection inference with HSIC-Lasso. In International Conference on Machine Learning , pages 3439-3448. PMLR, 2021.

Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of Statistics , pages 1189-1232, 2001.

Hui Gao, Joshua M Korn, Stéphane Ferretti, John E Monahan, Youzhen Wang, Mallika Singh, Chao Zhang, Christian Schnell, Guizhi Yang, Yun Zhang, et al. High-throughput screening using patient-derived tumor xenografts to predict clinical trial drug response. Nature Medicine , 21(11):1318-1325, 2015.

Mathew J Garnett, Elena J Edelman, Sonja J Heidorn, Chris D Greenman, Anahita Dastur, King Wai Lau, Patricia Greninger, I Richard Thompson, Xi Luo, Jorge Soares, et al. Systematic identification of genomic markers of drug sensitivity in cancer cells. Nature , 483(7391):570-575, 2012.

Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. In Advances in Neural Information Processing Systems , volume 34, pages 18932-18943, 2021.

Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning on typical tabular data? In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track , 2022.

Logan Grosenick, Brad Klingenberg, Kiefer Katovich, Brian Knutson, and Jonathan E Taylor. Interpretable whole-brain prediction analysis with GraphNet. NeuroImage , 72:304-321, 2013.

Isabelle Guyon, Steve Gunn, Asa Ben-Hur, and Gideon Dror. Result analysis of the NIPS 2003 feature selection challenge. In Advances in Neural Information Processing Systems , volume 17, pages 545-552, 2004.

David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. In International Conference on Learning Representations , 2016.

David Hallac, Jure Leskovec, and Stephen Boyd. Network LASSO: Clustering and optimization in large graphs. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 387-396, 2015.

Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction , volume 2. Springer, 2009.

Hussein Hazimeh, Natalia Ponomareva, Petros Mol, Zhenyu Tan, and Rahul Mazumder. The tree ensemble layer: Differentiability meets conditional computation. In International Conference on Machine Learning , pages 4138-4148. PMLR, 2020.

Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. In International Conference on Learning Representations , 2023.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open Graph Benchmark: Datasets for machine learning on graphs. In Advances in Neural Information Processing Systems , volume 33, pages 22118-22133, 2020.

Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference on Learning Representations , 2020.

Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar Karnin. TabTransformer: Tabular data modeling using contextual embeddings. In Advances in Neural Information Processing Systems , volume 35, pages 2902-2915, 2022.

Francesco Iorio, Theo A Knijnenburg, Daniel J Vis, Graham R Bignell, Michael P Menden, Michael Schubert, Nanne Aben, Emanuel Gonçalves, Syd Barthorpe, Howard Lightfoot, et al. A landscape of pharmacogenomic interactions in cancer. Cell , 166(3):740-754, 2016.

Alan Jeffares, Tennison Liu, Jonathan Crabbé, Fergus Imrie, and Mihaela van der Schaar. TANGOS: Regularizing tabular neural networks through gradient orthogonalization and specialization. In International Conference on Learning Representations , 2023.

Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned simple nets excel on tabular datasets. In Advances in Neural Information Processing Systems , volume 34, pages 23928-23941, 2021.

Gregor Kasieczka, Benjamin Nachman, David Shih, Oz Amram, Anders Andreassen, Kees Benkendorder, Blaz Bortolato, Gustaaf Broojimans, Florencia Canelli, Jack Collins, et al. The LHC Olympics 2020: a community challenge for anomaly detection in high energy physics. Reports on Progress in Physics , 84:124201, 2021.

Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems , volume 30, pages 3146-3154, 2017.

Sebastian Köhler, Nicole A Vasilevsky, Mark Engelstad, Erin Foster, Julie McMurry, Ségolène Aymé, Gareth Baynam, Susan M Bello, Cornelius F Boerkoel, Kym M Boycott, et al. The Human Phenotype Ontology in 2017. Nucleic Acids Research , 45(D1):D865-D876, November 2016.

Peter Kontschieder, Madalina Fiterau, Antonio Criminisi, and Samuel Rota Bulo. Deep neural decision forests. In Proceedings of the IEEE International Conference on Computer Vision , pages 1467-1475, 2015.

Jannik Kossen, Neil Band, Clare Lyle, Aidan N Gomez, Thomas Rainforth, and Yarin Gal. Self-attention between datapoints: Going beyond individual input-output pairs in deep learning. In Advances in Neural Information Processing Systems , volume 34, pages 28742-28756, 2021.

Michael Kuhn, Ivica Letunic, Lars Juhl Jensen, and Peer Bork. The SIDER database of drugs and side effects. Nucleic Acids Research , 44(D1):D1075-D1079, October 2015.

Roman Levin, Valeriia Cherepanova, Avi Schwarzschild, Arpit Bansal, C Bayan Bruss, Tom Goldstein, Andrew Gordon Wilson, and Micah Goldblum. Transfer learning with deep tabular models. In International Conference on Learning Representations , 2020.

Caiyan Li and Hongzhe Li. Network-constrained regularization and variable selection for analysis of genomic data. Bioinformatics , 24(9):1175-1182, 2008.

Bo Liu, Ying Wei, Yu Zhang, and Qiang Yang. Deep neural networks for high dimension, low sample size data. In International Joint Conference on Artificial Intelligence Organization , pages 2287-2293, 2017.

Yin Lou and Mikhail Obukhov. BDT: Gradient boosted decision tables for high accuracy and scoring efficiency. In Proceedings of the SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1893-1901, 2017.

Yang Lu, Yingying Fan, Jinchi Lv, and William Stafford Noble. DeepPINK: reproducible feature selection in deep neural networks. In Advances in Neural Information Processing Systems , volume 31, pages 8690-8700, 2018.

Katja Luck, Dae-Kyum Kim, Luke Lambourne, Kerstin Spirohn, Bridget E Begg, Wenting Bian, Ruth Brignall, Tiziana Cafarelli, Francisco J Campos-Laborie, Benoit Charloteaux, et al. A reference map of the human binary protein interactome. Nature , 580(7803):402-408, April 2020.

Donald W Marquardt and Ronald D Snee. Ridge regression in practice. The American Statistician , 29(1):3-20, 1975.

Lukas Meier, Sara Van De Geer, and Peter Bühlmann. The group LASSO for logistic regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 70(1):53-71, 2008.

Jörg Menche, Amitabh Sharma, Maksim Kitsak, Susan Dina Ghiassian, Marc Vidal, Joseph Loscalzo, and Albert-László Barabási. Uncovering disease-disease relationships through the incomplete interactome. Science , 347(6224):1257601, 2015.

Soufiane MC Mourragui, Marco Loog, Daniel J Vis, Kat Moore, Anna G Manjon, Mark A van de Wiel, Marcel JT Reinders, and Lodewyk FA Wessels. Predicting patient response with models trained on cell lines and patient-derived xenografts by nonlinear transfer learning. Proceedings of the National Academy of Sciences , 118(49):e2106682118, 2021.

Rose Oughtred, Chris Stark, Bobby-Joe Breitkreutz, Jennifer Rust, Lorrie Boucher, Christie Chang, Nadine Kolas, Lara O'Donnell, Genie Leung, Rochelle McAdam, et al. The BioGRID interaction database: 2019 update. Nucleic Acids Research , 47(D1):D529-D541, 2019.

Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. In International Conference on Learning Representations , 2020.

Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. CatBoost: unbiased boosting with categorical features. In Advances in Neural Information Processing Systems , volume 31, pages 6639-6649, 2018.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv:2204.06125 , 2022.

Thomas Rolland, Murat Ta¸ san, Benoit Charloteaux, Samuel J Pevzner, Quan Zhong, Nidhi Sahni, Song Yi, Irma Lemmens, Celia Fontanillo, Roberto Mosca, et al. A proteome-scale map of the human interactome network. Cell , 159(5):1212-1226, 2014.

Adriana Romero, Pierre Luc Carrier, Akram Erraqabi, Tristan Sylvain, Alex Auvolat, Etienne Dejoie, Marc-André Legault, Marie-Pierre Dubé, Julie G Hussin, and Yoshua Bengio. Diet networks: thin parameters for fat genomics. In International Conference on Learning Representations , 2017.

Jean-François Rual, Kavitha Venkatesan, Tong Hao, Tomoko Hirozane-Kishikawa, Amélie Dricot, Ning Li, Gabriel F Berriz, Francis D Gibbons, Matija Dreze, Nono Ayivi-Guedehoussou, et al. Towards a proteome-scale map of the human protein-protein interaction network. Nature , 437(7062):1173-1178, 2005.

Camilo Ruiz, Marinka Zitnik, and Jure Leskovec. Identification of disease treatment mechanisms through the multiscale interactome. Nature Communications , 12(1):1-15, 2021.

Lukasz Salwinski, Christopher S Miller, Adam J Smith, Frank K Pettit, James U Bowie, and David Eisenberg. The Database of Interacting Proteins: 2004 update. Nucleic Acids Research , 32(suppl_1):D449-D451, 2004.

Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation , 4(1):131-139, 1992.

Ravid Shwartz-Ziv and Amitai Armon. Tabular data: Deep learning is not all you need. Information Fusion , 81:84-90, 2022.

Alexander J Smola and Risi Kondor. Kernels and regularization on graphs. In Learning Theory and Kernel Machines , pages 144-158. Springer, 2003.

Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. SAINT: Improved neural networks for tabular data via row attention and contrastive pre-training. Table Representation Learning Workshop at NeurIPS , 2022.

Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management , pages 1161-1170, 2019.

Damian Szklarczyk, Annika L Gable, Katerina C Nastou, David Lyon, Rebecca Kirsch, Sampo Pyysalo, Nadezhda T Doncheva, Marc Legeay, Tao Fang, Peer Bork, Lars J Jensen, and Christian von Mering. The STRING database in 2021: customizable protein-protein networks, and functional characterization of user-uploaded gene/measurement sets. Nucleic Acids Research , 49(D1):D605-D612, November 2020.

Robert Tibshirani. Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society: Series B (Methodological) , 58(1):267-288, 1996.

Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In International Conference on Machine Learning , pages 2071-2080. PMLR, 2016.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research , 9(11):2579-2605, 2008.

Laurens Van Der Maaten, Eric Postma, Jaap Van den Herik, et al. Dimensionality reduction: a comparative. Journal of Machine Learning Research , 10(66-71):13, 2009.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations , 2018.

Kavitha Venkatesan, Jean-Francois Rual, Alexei Vazquez, Ulrich Stelzl, Irma Lemmens, Tomoko Hirozane-Kishikawa, Tong Hao, Martina Zenkner, Xiaofeng Xin, Kwang-Il Goh, et al. An empirical framework for binary interactome mapping. Nature Methods , 6(1):83-90, 2009.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations , 2019.

Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering , 29(12):2724-2743, 2017.

Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating on hyperplanes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 28, pages 1112-1119, 2014.

Zifeng Wang and Jimeng Sun. TransTab: Learning transferable tabular transformers across tables. In Advances in Neural Information Processing Systems , volume 35, pages 2902-2915, 2022.

David S Wishart, Yannick D Feunang, An C Guo, Elvis J Lo, Ana Marcu, Jason R Grant, Tanvir Sajed, Daniel Johnson, Carin Li, Zinat Sayeeda, Nazanin Assempour, Ithayavani Iynkkaran, Yifeng Liu, Adam Maciejewski, Nicola Gale, Alex Wilson, Lucy Chin, Ryan Cummings, Diana Le, Allison Pon, Craig Knox, and Michael Wilson. DrugBank 5.0: a major update to the DrugBank database for 2018. Nucleic Acids Research , 46(D1):D1074-D1082, November 2017.

David S Wishart, Yannick D Feunang, An C Guo, Elvis J Lo, Ana Marcu, Jason R Grant, Tanvir Sajed, Daniel Johnson, Carin Li, Zinat Sayeeda, et al. DrugBank 5.0: a major update to the DrugBank database for 2018. Nucleic Acids Research , 46(D1):D1074-D1082, 2017.

Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning , pages 23965-23998. PMLR, 2022.

Yutaro Yamada, Ofir Lindenbaum, Sahand Negahban, and Yuval Kluger. Feature selection using stochastic gates. In International Conference on Machine Learning , pages 10648-10659. PMLR, 2020.

Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In International Conference on Learning Representations , 2015.

Wanjuan Yang, Jorge Soares, Patricia Greninger, Elena J Edelman, Howard Lightfoot, Simon Forbes, Nidhi Bindal, Dave Beare, James A Smith, I Richard Thompson, et al. Genomics of Drug Sensitivity in Cancer (GDSC): a resource for therapeutic biomarker discovery in cancer cells. Nucleic Acids Research , 41(D1):D955-D961, 2012.

Yongxin Yang, Irene Garcia Morillo, and Timothy M Hospedales. Deep neural decision trees. In International Conference on Machine Learning, Workshop on Human Interpretability in Machine Learning (WHI) , 2018.

Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? In Advances in Neural Information Processing Systems , volume 34, pages 28877-28888, 2021.

Haiyuan Yu, Leah Tardivo, Stanley Tam, Evan Weiner, Fana Gebreab, Changyu Fan, Nenad Svrzikapa, Tomoko Hirozane-Kishikawa, Edward Rietman, Xinping Yang, et al. Nextgeneration sequencing to generate interactome datasets. Nature Methods , 8(6):478-480, 2011.

Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 68(1):49-67, 2006.

Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. XTab: Cross-table pretraining for tabular transformers. In International Conference on Machine Learning , pages 43181-43204. PMLR, 2023.

A Evaluation protocol and hyperparameter ranges

To ensure a fair comparison with baselines, we follow evaluation protocols outlined in tabular benchmarks [19, 18]. We conduct a random search with 500 configurations of every model (including PLATO) on every dataset across a broad range of hyperparameters. We base the hyperparameter ranges on the ranges used in prior tabular learning benchmarks [19, 18] and the ranges mentioned in the original papers of the methods. Hyperparameter ranges for PLATO are given in Table 7. Hyperparameter ranges for baseline methods are given in Table 8.

B Graph classification approaches

Graph classification models are not appropriate for PLATO's setting. In graph classification models, every input sample is a graph with node attributes, and a model must make a prediction for that graph. The PLATO problem setting breaks key assumptions made by typical graph classification models. First, graph classification models assume that different samples correspond to different graphs [81, 28, 27]. However, in PLATO every sample corresponds to the exact same graph. There is a single background knowledge graph for all samples, and every sample has input features that correspond to the exact same nodes within the knowledge graph. Second, graph classification approaches typically assume that every node in an input graph has a node attribute [81, 28, 27]. However, in PLATO only a small subset of the nodes in the knowledge graph have measured feature values. Finally, graph classification approaches typically assume small graphs: the largest graph classification task in the Open Graph Benchmark has only 244 nodes [27]. However in PLATO, the knowledge graph contains 108,447 and the smallest dataset has 12,932 features corresponding to nodes.

C PLATO's performance across node embedding methods for pre-training the feature embeddings

We conduct an ablation study to assess how PLATO's performance depends on the node embedding method used to pre-train the feature embeddings (Methods 3.4). We test three shallow node embedding methods for knowledge graphs which are scalable and prominent: TransE [72], DistMult [78], and ComplEx [65]. We find that PLATO's performance is similar across TransE, DistMult, and ComplEx (Table 9). More generally, PLATO makes no assumption about what type of self-supervised node embedding method is used to pre-train the feature embeddings. The self-supervised embedding step is simply a module that pre-trains feature embeddings which are then passed to the message passing and weight inference modules of PLATO.

D Rank ordering of methods for datasets with d ≫ n

In Table 10, we show the rank order performance of all models on all d ≫ n datasets. We find that PLATO exhibits consistent and strong performance while the performance of the baselines depends on the specific d ≫ n dataset. For example, TabTransformer is the second best performing of all models on the MNSCLC dataset but the worst performing of all models on the PDAC and CH datasets. Similarly, XGBoost is the second best performing of all models on PDAC but only the tenth best performing of all models on BRCA. The baselines with the most stable performance are LASSO and Ridge Regression which rank consistently between the second and eighth best of all models.

E Graph regularization baselines

We test the state-of-the-art graph regularization baselines GraphNet [20], Network-Constrained LASSO [40], and Network LASSO [23]. The graph regularization baselines can only consider a homogeneous graph with only features as nodes and a single edge type. For the graph regularization baselines, we thus induce a subgraph between feature nodes from the knowledge graph and collapse all edge types between feature nodes into a single edge type. In this context, GraphNet, Networkconstrained LASSO, and Network LASSO correspond to a LASSO model with a mean-squared error loss and a graph regularization penalty. Let λ be the graph regularization coefficient, j and k be two input features, let E be the set of edges in the graph, let Θ ∈ R d be the weights of the linear regression for d input features, and let D j be the degree of feature node j . The graph regularization penalty for GraphNet is λ ∑ j,k ∈ E ( Θ j -Θ k ) 2 , the penalty for Network-constrained LASSO is

F Number of trainable weights in PLATO vs. a multilayer perceptron

G Dataset Details

We use 6 datasets with d ≫ n and 4 datasets with d ∼ n [16, 17, 30, 79]. In all datasets, a machine learning model must predict the response of a cell line or a mouse tumor model to a drug. As input, the model considers a tabular dataset. In the tabular dataset, every row corresponds to a specific cell line or mouse tumor model. Every column corresponds to a gene name. Every value corresponds to the amount of that gene in the cell line or in the mouse tumor ( i.e. gene expression). In practice, the number of genes is large for all tasks and the number of cell lines or mouse tumor models is comparatively small ( i.e. d ≫ n ). For every row, the model also takes as input a fixed feature vector corresponding to the drug ( i.e. a 200-dimensional ComplEx [65] embedding of the drug node in the knowledge graph). The output label is the response of the cell line ( i.e. ln-ic50) [30, 79, 17] or mouse tumor model ( i.e. minimum average percent tumor growth 'min-avg-pct-tumor-growth') to the drug [16]. Data is available at https://github.com/snap-stanford/plato .

Gene expression datasets were pre-processed following a standard process in [48]. Briefly, gene expression values underwent TMM normalization and log transformation ( i.e. log( x +1) ). Values were made to have zero mean and unit standard deviation. Dataset abbreviations are breast carcinoma (BC) [30, 79, 17], breast carcinoma (BRCA) [16], chondrosarcoma (CH) [30, 79, 17], colorectal cancer (CRC) [16], cutaneous melanoma (CM) [16], melanoma (ME) [30, 79, 17], non-small cell lung carcinoma (MNSCLC) [16], non-small cell lung carcinoma (NSCLC) [30, 79, 17], pancreatic ductal carcinoma (PDAC) [16], and small cell lung carcinoma (SCLC) [30, 79, 17].

H Knowledge Graph Details

We compile a general biomedical knowledge graph from prior studies [44, 35, 38, 56, 63, 74, 75] to use across all datasets. A schematic of the KG is in Supplementary Figure 2. A detailed breakdown of relation types is in Supplementary Table 13. The knowledge graph is available at https://github.com/snap-stanford/plato .

The knowledge graph contains 108,447 total nodes, including 7,975 drugs, 18,370 diseases, 11,447 phenotypes, 22,319 genes, 11,153 molecular functions, 28,748 biological processes, and 4,184 cellular components. Every gene and every drug in every dataset is present as a node in the knowledge graph. The knowledge graph also contains 3,066,156 edges with 99 distinct relation types. The remaining node types and their relationships serve as broader domain knowledge.

Edges between drug nodes and gene/protein nodes were derived from Drugbank [75], Gao [16], and the Genomics of Drug Sensitivity in Cancer [79, 30, 17]. Edges between diseases and genes/proteins were derived from DisGeNet [5]. Edges between diseases and phenotypes were derived from the Human Phenotype Ontology [35]. Edges between drugs and diseases were derived from the Multiscale Interactome [56]. Edges between drugs and side effects were derived from SIDER [38]. Edges between genes/proteins and other genes/proteins were derived from BioGRID [49], [55], the Database of Interacting Proteins [57], [44], [47], [53], [82], [69], and STRING [63]. Finally, edges from genes/proteins to molecular functions, biological processes, and cellular components as well as edges between molecular functions, biological processes, and cellular components were derived from the Gene Ontology [9].

(e)

(c)

43

[1]

Legend

13,945

[6]

1,070

[2]

67,186

[8]

149

7,032

[3]

Node Type 1

Molecular

Biological

[1]

Cellular

(SPOn Wnn)

Function

(11,153)

1,017

Process

(28,748)

Component

(4,184)

[4]

Numrelations

35,897

44,327

36,391

[Num relation types]

[4]

[2]

[4]

NodeType 2

Gene/Protein

(SPOn Wnn)

(22,319)

2,336,396

[46]

29,189

[4]

136,772

[10]

14,650

[1]

(q)

Nnodes = 108,447

Total

(7,975)

Drug

5926

[1]

Disease

(18,370)

195,402

[1]

Phenotype/Side

Effect

Nedges = 3,066,156

(4,251)

140,764

[1]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a method to leverage prior knowledge in the form of a knowledge graph for tabular deep learning, illustrating performance on a number of biological prediction problems in which features correspond to gene-level measurements. In this case, the knowledge graph is a biomedical knowledge graph which contains a node for each gene as well as nodes representing related biomedical concepts. A set of initial node embeddings are generated using a standard KG embedding approach. Once these are obtained, a set of node vectors for each sample is generated via message passing on the knowledge graph, making use of the sample values for each feature in the message passing step. These weights are then used as the first layer in a MLP. \n\nThe authors show that their method consistent achieves the best performance across a set of 6 data-sets where the number of features is significantly larger than the number of samples and is consistent with XGBoost when the number of features is comparable to the number of dimensions, suggesting the method could be a good default choice in cases where a relevant knowledge graph is available\n\nThe authors conduct a comprehensive ablation study to demonstrate the necessity of each component of their model in achieving the best prediction performance.\n\nSTRENGTHS:\nThe approach is non-obvious and effective, while also being intuitive and clearly explained.The authors have combined ideas from knowledge graph embedding and graph-based regularisation to provide a state-of-the-art model for tabular problems in which a relevant knowledge graph is available. Given the abundance of prediction problems in the biomedical space and the accessibility of high quality knowledge graphs, I expect this method will see many applications there.\n\nWEAKNESSES:\nThe authors claim PLATO is robust to missing edges in the knowledge graph. However, when 50% of edges have been removed the model is outperformed by multiple approaches on BRCA, the data-set on which this comparison was carried out. To make this claim, I think the authors need to show that the SOTA performance is maintained across all data-sets in the presence of missing edges.\n\nQUESTIONS:\nCan you include data for missing edges in the other 6 data-sets? Looking at Table 4, I am left wondering if the model will indeed outpeform all baselines when using 70% or 75% of the data\n\nAn interesting extension, or an opportunity for future work, could evaluate whether adding high-confidence predictions from ComplEx was sufficient to recover the performance drop at 50%.\n\nLIMITATIONS:\nThe authors have addressed the key technical limitation: the availability of a good quality and comprehensive knowledge graph. The data presented in Table 4 could be reframed as an additional statement of this limitation - in the presence of an incomplete knowledge graph the performance is worse than simple baselines like Ridge.\n\nI do not believe the model has potential negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces PLATO, a novel machine learning approach for high-dimensional tabular data, where the number of features far exceeds the number of samples (d>>n). This scenario is often encountered in fields like physical sciences and biology. PLATO uses a knowledge graph (KG) to train the first layer of a multilayer perceptron (MLP), creating feature embeddings based on the information in the KG and a message-passing algorithm. This approach leverages auxiliary domain information to regularize the MLP, reducing the risk of overfitting. Benchmarking PLATO on six biomedical datasets revealed it outperformed 13 state-of-the-art baselines by up to 10.19%, demonstrating its effectiveness for the d>>n scenario.\n\nSTRENGTHS:\nThe mechanism to induce KG regularization is novel. Studying KG integration into predictions seems understudied, while it has been done in specific contexts (e.g. gene interaction networks https://www.nature.com/articles/nmeth.4627, https://www.nature.com/articles/s42003-021-02622-z). The studies setting matters especially in a scientific context. A general method for knowledge graph integration could be very valuable to numerous scientific fields. The work is clearly written, with minor writing issues (especially in conclusion). It seems to provide most experimental details necessary. \n\nAn ablation of varying the fraction of the KG graph used clearly shows that the method is able to learn from increasing domain knowledge.\n\nWEAKNESSES:\nBaselines may not be as well tuned, e.g. use combinations of PCA + XGB. Hyperparameters might not be appropriate for the high dimensional setting, e.g. for trees.\nA cross-validation procedure, e.g. 5-fold CV to find optimal hyperparameters on train and valid would be more optimal. Baselines with many hyperparameters to tune would profit more from this stronger evaluation. This is also what data-scientists would perform on such data, especially given the low N.\nIts not an apples to apples comparison for all methods since domain knowledge is used for this approach. A simple baseline using domain knowledge + Baselines would be interesting, e.g. keeping only relevant genes in BRCA dataset based on prior knowledge. I believe significance values for genetic locations from prior studies are much more commonly known than graph connectivity. I understand this may still be hard to evaluate.\n\nQUESTIONS:\nMissing / Hard to understand where knowledge graphs come from. Were they created using the full data? Were they created using completely separate experiments? What was the method to create each? the supplement lists only the studies with no further details on how the graphs were obtained.\n\nLIMITATIONS:\nThey adress some limitations. I would add that while PLATO uses domain knowledge, it only models one aspect of the domain knowledge provided by the graph: the similarity of nodes in the graph (Hope I correctly understand that?).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores tabular ML tasks in scenarios where data has more features than samples and auxiliary structured data is available in the form of knowledge graphs. One of the central assumptions is that each feature corresponds to a node in the knowledge graph. A model called PLATO is proposed which combines a tabular MLP with the auxiliary knowledge graph, with the main idea that for features which correspond to similar nodes in the graph weight vectors in the first MLP layer should also be similar. The paper showcases performance gains of PLATO on 6 datasets compared to tabular models trained solely on the tabular data without leveraging the knowledge graph.\n\nSTRENGTHS:\nI would like to thank the authors for this paper. While I believe that it may not be quite ready for publication, I enjoyed reading the paper. The presentation is excellent and the ideas are explained well, it was definitely a fun read. Overall, I believe the paper has potential and I hope my feedback would help make the paper stronger. \n\nStrengths:\n- This paper explores an interesting and useful idea of leveraging auxiliary knowledge graphs to aid in tabular data tasks\n- The presentation is excellent, the ideas are explained well, with great figures\n- It is clear that the authors made a considerable effort to evaluate multiple tabular baselines\n- The work is reproducible with the code included in the submission\n\nWEAKNESSES:\nI believe that a number of weaknesses need to be addressed before the paper can be published.\n\n**Major:**\n\nGraph perspective\n- My main criticism is that it seems that PLATO is more of a method to learn graph embeddings which leverage auxiliary tabular data to solve subgraph classification or regression tasks rather than a tabular architecture regularized by a graph. This is a key distinction because it defines baselines that the model should be compared to. *That is, PLATO is more of a graph model and should be compared to graph models.* \n\n *Explanation for why PLATO is more a graph model than a tabular model*: As PLATO infers weight vectors in the first MLP layer which correspond to the input features from graph embeddings, it essentially means that that first MLP layer becomes simply a linear combination of graph-based feature embeddings with coefficients coming from feature value for a particular sample. This linear combination is then passed through a nonlinearity and into a downstream model (which in this case is all the other layers of MLP but it does not have to be just that). As the central assumption is that each feature has a corresponding graph node, this means that each sample can be represented as a subgraph with different weights on the subgraph nodes which correspond to feature value. So this becomes a subgraph classification/regression problem with the addition of those weights.\n\n- As such, I encourage the authors to compare against suitable graph neural network baselines for the problem of subgraph classification/regression. One possible baseline is GCN-based node embeddings combined into subgraph embeddings as weighted combinations with weights coming from feature values.\n\nTabular perspective\n- If we think of PLATO as a method for regularizing tabular architectures using a graph, then I see two major limitations: (1) PLATO is limited to MLP architectures only, (2) the central assumption is that *each* feature has a corresponding node in the graph is extremely limiting. If any of these assumptions can be relaxed, then it would be great to see corresponding experiments.\n- It seems likely that the performance gains observed in Table 1 come solely from leveraging the knowledge graph and not as much from the strength of PLATO as a tabular architecture. Other models in Table 1 do not leverage the knowledge graph in any way. Table 3 highlights that performance gains likely primarily come from the knowledge graph as Full KG gives significantly better performance than feature-only KG. That is, the performance gains do not come from regularizing and avoiding overfitting in short and fat datasets, but from the additional information source.\n- Ideally, PLATO should be compared against graph baselines, tabular baselines and some simple graph+tabular baselines to showcase the strength and justify the complexity of the PLATO architecture and design choices.\n\n**Less major:**\n- The paper claims to follow evaluation protocols from [1]. However, [1] used Bayesian HPO framework Optuna which is a stronger approach than random hyperparameter search. \n- As mentioned in the paper, it is true that default FT-Transformer fails with out of memory errors on datasets with many features. However, the RTDL package implements Linformer strategy for efficient attention to address cases with many features. To be able to use it as a baseline model, I encourage the authors to explore the kv_compression_ratio hyperparameter from the FT-Transformer documentation https://yura52.github.io/rtdl/stable/api/rtdl.FTTransformer.html?highlight=linformer\n\n**Related work issues:**\n- A significant portion of related work on tabular deep learning is missing. For example, tabular transofmer architectures [2,3,4] could be added to the section Tabular deep learning methods. Additionally, while the authors say that \"tabular deep learning methods have been developed for settings with far more samples than features\" and \"in the low-data regime with far more features than samples, the dominant approaches are still statistical\", there are in fact tabular deep learning works which specifically address low data regimes through transfer learning [5]. There are also other deep tabular transfer learning architectures [6,7].\n\n \nReferences:\n\n[1] Gorishniy, Y., Rubachev, I., Khrulkov, V. and Babenko, A., 2021. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34, pp.18932-18943.\n\n[2] Hollmann, N., Müller, S., Eggensperger, K. and Hutter, F., 2022. Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXiv:2207.01848.\n\n[3] Somepalli, G., Goldblum, M., Schwarzschild, A., Bruss, C.B. and Goldstein, T., 2021. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342.\n\n[4] Kossen, J., Band, N., Lyle, C., Gomez, A.N., Rainforth, T. and Gal, Y., 2021. Self-attention between datapoints: Going beyond individual input-output pairs in deep learning. Advances in Neural Information Processing Systems, 34, pp.28742-28756.\n\n[5] Levin, R., Cherepanova, V., Schwarzschild, A., Bansal, A., Bruss, C.B., Goldstein, T., Wilson, A.G. and Goldblum, M., 2022. Transfer learning with deep tabular models. arXiv preprint arXiv:2206.15306.\n\n[6] Zhu, B., Shi, X., Erickson, N., Li, M., Karypis, G. and Shoaran, M., 2023. XTab: Cross-table Pretraining for Tabular Transformers. arXiv preprint arXiv:2305.06090.\n\n[7] Wang, Z. and Sun, J., 2022. Transtab: Learning transferable tabular transformers across tables. Advances in Neural Information Processing Systems, 35, pp.2902-2915.\n\nQUESTIONS:\n1. I encourage the authors to compare against suitable graph neural network baselines for the problem of subgraph classification/regression. One possible baseline is GCN-based node embeddings combined into subgraph embeddings as weighted combinations with weights coming from feature values.\n2. How can categorical features be handled by PLATO?\n3. If PLATO can be used in cases where only some of the features correspond to nodes in the knowledge graph and other features don't, it would be great to see corresponding experiments.\n4. It is mentioned on page 3, line 101 that PLATO \"represents a new regularization mechanism\". If PLATO can be applied as a regularization technique to architectures other than MLP, it would be great to see corresponding experiments. \n5. Expanding on 1, as it is clear from Table 3 that the knowledge graph is a significant driver of performance gains as an additional information source, is it possible to compare against other tabular baselines which also leverage the KG information?\n\nLIMITATIONS:\nThe authors did not address limitations or societal impact of the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces PLATO, an approach handles tabular deep learning challenges when the number of features (d) significantly outnumbers the available instances (n). The authors propose the integration of an auxiliary knowledge graph into the learning process to help the model better understand the relationships between features and hence make more accurate predictions. The author conducted extensive experiments to validate the effectiveness of their approach, involving scenarios with different data sparsity levels, diverse knowledge graph structures, and various types of tabular data. The results demonstrate the proposed method outperforms conventional deep learning approaches on tabular data when the number of features is much larger than the number of instances.\n\nSTRENGTHS:\nThis paper proposed an original approach to a longstanding issue in machine learning: the 'curse of dimensionality' in situations where the feature dimension (d) greatly exceeds the number of samples (n). Through combining several existing ideas in deep learning and knowledge representation, It creates a potentially transformative technique to address challenge of high-dimensionality in tabular data. \n\nThis paper provided empirical evidence of the method's advancement. The proposed method outperforms 13 state-of-the-art baselines across 6 d >> n datasets by up to 10.19%.\n\nThe paper is generally well written, with clear explanations of the main concepts and methodology.\n\nWEAKNESSES:\nThere has been existing work combining knowledeg graph with deep learning on tabular dataset, and the distinctness of proposed method from previous work is not obviously illustrated.\n\nThe robustness and scalability analysis is lacked, how does performance vary with the size and the complexity of the knowledge graph? Does this method scale to large datasets? It would be useful to include a discussion of these issues.\n\nThe proposed method PLATO doesn't show evident advancement compared to selected baselines on 4 d~n datasets, especially for XGBoost, which performance seems to keep up with PLATO and has been proposed over five years, justifying the weakness of advancement for the selected baselines.\n\nQUESTIONS:\nSince PLATO utilizes auxiliary knowledge graph to pretrain an embedding for each input feature, will other pretrained methods achieve same competitive performance on the selected dataset?\n\nHow diverse or domain-specific is the auxiliary knowledge graph? Can the proposed method adapt to various domains or does it require specific types of knowledge graphs?\n\nCan the auxiliary knowledge graph combine with other advanced models rather than a simple MLP in the experiment's setting.\n\nLIMITATIONS:\nThe reviewer does not see potential negative social impact of the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}